--- icmp.cpp_bak        2014-11-28 09:23:44.559231668 +0100
+++ icmp.cpp    2014-11-28 09:28:48.212833890 +0100
@@ -227,8 +227,15 @@

    // Do ping
    nBytes = dwPacketSize - sizeof(IPHDR);
+   UINT32 seed = time(0) * dwAddr; // attempt to create different seeds for each call to each node
+   int iNumRetriesOrig = iNumRetries;
    while(iNumRetries--)
-   {
+   {  // random delay before start pinging
+      int min = 500 * (iNumRetriesOrig - iNumRetries+1); // min = 0 in first run, then wait longer and longer
+      int max = 1000 + min;  // increased random window between retries
+      int delay = min + (rand_r(&seed) % (int)(max - min + 1));
+      ThreadSleepMs(delay);
+
       dwRTT = 0;  // Round-trip time for current request
       request.m_icmpHdr.m_wId = ICMP_REQUEST_ID;
       request.m_icmpHdr.m_wSeq++;
@@ -364,7 +371,7 @@
 #endif
       }

-      ThreadSleepMs(500);     // Wait half a second before sending next packet
+      // ThreadSleepMs(500);     // Wait half a second before sending next packet // We do random delay in beginning of loop instead
    }

 stop_ping:

This feels like a workaround rather than a fix to the original problem, but it can at least help in pinpointing the problem. Looks like it is something outside NetXMS boundaries that doesn't keep up, like linux kernel, wmware hosts, network switches...

Worth to note that this installation contains a lot of nodes with neither netxms or snmp agents, guess this will cause a lot more icmp pinging during status polls than in installations with mostly agent-running nodes?

If you want more of the log i can mail you.

[19-Nov-2014 14:40:22.297] [DEBUG] Node 3404 "NODE-001-UNIT" queued for status poll
[19-Nov-2014 14:40:22.298] [DEBUG] Starting status poll for node NODE-001-UNIT (ID: 3404)
[19-Nov-2014 14:40:32.802] [DEBUG] StatusPoll(NODE-001-UNIT): bAllDown=true, dynFlags=0x00001001
[19-Nov-2014 14:40:32.802] [DEBUG] Node::checkNetworkPath(NODE-001-UNIT [3404]): cannot find interface object for primary IP
[19-Nov-2014 14:40:32.802] [DEBUG] Node::checkNetworkPath(NODE-001-UNIT [3404]): trace available, 0 hops, incomplete
[19-Nov-2014 14:40:32.802] [DEBUG] Node::checkNetworkPath(NODE-001-UNIT [3404]): will do second pass
[19-Nov-2014 14:40:32.802] [DEBUG] Finished status poll for node NODE-001-UNIT (ID: 3404)
[19-Nov-2014 14:40:32.803] [DEBUG] CorrelateEvent: event SYS_NODE_DOWN id 3267062 source NODE-001-UNIT [3404]
[19-Nov-2014 14:40:32.803] [DEBUG] EVENT 28 (ID:3267062 F:0x0001 S:4 TAG:"") FROM NODE-001-UNIT: RSE081: Node down
[19-Nov-2014 14:40:32.803] [DEBUG] CorrelateEvent: event SYS_IF_DOWN id 3267061 source NODE-001-UNIT [3404]
[19-Nov-2014 14:40:32.804] [DEBUG] EVENT 5 (ID:3267061 F:0x0001 S:2 TAG:"" CORRELATED) FROM NODE-001-UNIT: Interface "unknown" changed state to DOWN (IP Addr: 10.120.29.84/255.255.255.192, IfIndex: 1)
[19-Nov-2014 14:40:32.804] [DEBUG] CorrelateEvent: event SYS_NODE_CRITICAL id 3267063 source NODE-001-UNIT [3404]
[19-Nov-2014 14:40:32.804] [DEBUG] EVENT 10 (ID:3267063 F:0x0001 S:4 TAG:"") FROM NODE-001-UNIT: Node status changed to CRITICAL

could you please run server with debug 6 for some time and then send me the log (pointing to fake node down moments)?

Best regards,
Victor

Anyone having same problem?

Yep. Its quite random. Seems like if it misses 1 ping, it calls it down.