Bug#36066725 Regular mgmd hangs when sending it a stop node for ndbmtd

Ole John Aske · dahlerlend · commit 49ef33f7edad · 2023-12-12T19:05:31.000+01:00
Root cause is that the mutexes 'theMultiTransporterMutex' and
'clusterMgrThreadMutex' are taken in different order in the
two respective call chains:

1) ClusterMgr::threadMain() -&gt; lock() -&gt; NdbMutex_Lock(clusterMgrThreadMutex)
   - ::threadMain(), holding clusterMgrThreadMutex -&gt; TransporterFacade::startConnecting()
     - TF::startConnecting -&gt; lockMultiTransporters()  &lt;&lt;&lt;&lt; HANG while holding clusterMgrThreadMutex

2) TransporterRegistry::report_disconnect() -&gt; lockMultiTransporters()
   - ::report_disconnect(), holding theMultiTransporterMutex, -&gt; TransporterFacade::reportDisconnect()
     - TF::reportDisconnect -&gt; ClusterMgr::reportDisconnected()
       - ClusterMgr::reportDisconnected() -&gt; lock()
         - lock() -&gt; NdbMutex_Lock(clusterMgrThreadMutex) &lt;&lt;&lt;&lt; Held by 1)

Patch change TransporterRegistry::report_disconnect() such that the
theMultiTransporterMutex is released before calling reportDisconnect(NodeId).

It should be sufficient to hold theMultiTransporterMutex while
::report_disconnect check if we are disconnecting a multiTransporter,
and if all its Trps are in DISCONNECTED state. When this
finished we have set up 'ready_to_disconnect' and can release
theMultiTransporterMutex before -&gt; reportDisconnect()

Change-Id: I19be0d9d92184efb8f20a92aa7189b9b85f069bc
diff --git a/storage/ndb/src/common/transporter/TransporterRegistry.cpp b/storage/ndb/src/common/transporter/TransporterRegistry.cpp
@@ -2321,14 +2321,14 @@ void TransporterRegistry::report_disconnect(TransporterReceiveHandle &recvdata,
       remove_allTransporters(this_trp);
     }
   }  // End of multiTransporter DISCONNECT handling
+  unlockMultiTransporters();
 
   if (ready_to_disconnect)  //  5)
   {
     DEBUG_FPRINTF((stderr, "(%u) -> reportDisconnect(node_id=%u)\n",
                    localNodeId, node_id));
     recvdata.reportDisconnect(node_id, errnum);
   }
-  unlockMultiTransporters();
   DBUG_VOID_RETURN;
 }
 

Original file line number	Diff line number	Diff line change
`@@ -2321,14 +2321,14 @@ void TransporterRegistry::report_disconnect(TransporterReceiveHandle &recvdata,`
`2321`	`2321`	`remove_allTransporters(this_trp);`
`2322`	`2322`	`}`
`2323`	`2323`	`} // End of multiTransporter DISCONNECT handling`
	`2324`	`+ unlockMultiTransporters();`
`2324`	`2325`
`2325`	`2326`	`if (ready_to_disconnect) // 5)`
`2326`	`2327`	`{`
`2327`	`2328`	`DEBUG_FPRINTF((stderr, "(%u) -> reportDisconnect(node_id=%u)\n",`
`2328`	`2329`	`localNodeId, node_id));`
`2329`	`2330`	`recvdata.reportDisconnect(node_id, errnum);`
`2330`	`2331`	`}`
`2331`		`- unlockMultiTransporters();`
`2332`	`2332`	`DBUG_VOID_RETURN;`
`2333`	`2333`	`}`
`2334`	`2334`