Tried a dirty patch to icmp.cpp, which improved the situation a lot. Idea is to do a random delay before pinging, otherwise a lot of pings will be fired almost simultaneously.
--- icmp.cpp_bak 2014-11-28 09:23:44.559231668 +0100
+++ icmp.cpp 2014-11-28 09:28:48.212833890 +0100
@@ -227,8 +227,15 @@
// Do ping
nBytes = dwPacketSize - sizeof(IPHDR);
+ UINT32 seed = time(0) * dwAddr; // attempt to create different seeds for each call to each node
+ int iNumRetriesOrig = iNumRetries;
while(iNumRetries--)
- {
+ { // random delay before start pinging
+ int min = 500 * (iNumRetriesOrig - iNumRetries+1); // min = 0 in first run, then wait longer and longer
+ int max = 1000 + min; // increased random window between retries
+ int delay = min + (rand_r(&seed) % (int)(max - min + 1));
+ ThreadSleepMs(delay);
+
dwRTT = 0; // Round-trip time for current request
request.m_icmpHdr.m_wId = ICMP_REQUEST_ID;
request.m_icmpHdr.m_wSeq++;
@@ -364,7 +371,7 @@
#endif
}
- ThreadSleepMs(500); // Wait half a second before sending next packet
+ // ThreadSleepMs(500); // Wait half a second before sending next packet // We do random delay in beginning of loop instead
}
stop_ping:
This feels like a workaround rather than a fix to the original problem, but it can at least help in pinpointing the problem. Looks like it is something outside NetXMS boundaries that doesn't keep up, like linux kernel, wmware hosts, network switches...
Worth to note that this installation contains a lot of nodes with neither netxms or snmp agents, guess this will cause a lot more icmp pinging during status polls than in installations with mostly agent-running nodes?
