Network Discovery

[Egert143]

Hello

Today i noticed that Active discovery haven't found few older switches. NetXms server can ping them fine but for some reason active discovery fails. Other nodes in same /24 subnet are found.

Egert

[Filipp Sudanov]

Please enable system user and login under it. Then try searching in object tree, may be there are nodes with these IP addresses that are, however, not visible to your regular admin account.

[Egert143]

Manually adding did not show error that it already exists. Also is it bad idea to have /16 network in active discovery with interval 900?

Egert

[Filipp Sudanov]

/16 might not be too bad idea. Initial pinging is real fast - it goes in blocks of 1000 addresses and takes a few seconds.
Then there is second stage of discovery - when server tries to connect to nodes to get some information. This and subsequent configuration poll for newly-added nodes occurs in DISCOVERY thread pool.

In Tools -> Server Console you can do

show queues

"Node discovery poller" should show the number of addresses waiting.

show discovery queue

should give a list of addresses.

show threads

DISCOVERY part should tell how busy this pool is, may be it should be increased in size.

On your server node there is a number of internal DCIs that keep historical stats for the above.
Server.QueueSize.Current(NodeDiscoveryPoller) is the actual queue size. Is should get to 0 during these 900 seconds. Would be interesting to see it's graph.

Server.ThreadPool.Load(DISCOVERY) and a few other for thread pools should tell how busy discovery thread pool is and if you need to make it bigger.

And if you want to dig deeper, you can turn on say 6 level of debug for tag poll.discovery. This should produce a lot of information in the server log.

[Egert143]

Server.QueueSize.Current(NodeDiscoveryPoller) for the last 24H has stayed constantly 0, is that normal ?

Server.ThreadPool.Load(DISCOVERY):

Usage (%): 100
Normalized load avg (1min): 0.00
Current size: 16
Current load (%): 0
Avg wait time (ms): 475

Seems its quite idle or not polling at all

[Filipp Sudanov]

It uses 16 out of max 16 threads which means that at some point they were needed and were populated. And avg wait time of about 0.5 second tells that there are moments when it's doing something.
Graph of current load for 24 hours could tell more, but looks like it's not too busy. It definitely had been busy when discovering this network for the first time, but now addresses for which nodes are already created are not polled in this thread pool any more.

You can add another active discovery range that would include just on of these switches that are not getting added, turn on 6 debug level for poll.discovery and do a scan manually only for this specific range. This way log would have just the information we need.

[Egert143]

One example switch that was not auto discovered 192.168.135.93.

i added more precise subnet 192.168.135.0/24 and started manual scan with debug lvl6.

results:

2022.06.02 15:18:39.072 *D* [poll.discovery     ] Starting SNMP check on range 192.168.135.1 - 192.168.135.254

2022.06.02 15:18:44.337 *D* [poll.discovery     ] Active discovery - node 192.168.135.93 responded to SNMP probe
2022.06.02 15:18:44.337 *D* [poll.discovery     ] Checking address 192.168.135.93 in zone 0 (source: Active Discovery)
2022.06.02 15:18:44.337 *D* [poll.discovery     ] New node queued: 192.168.135.93/24
2022.06.02 15:18:44.337 *D* [poll.discovery     ] NodePoller: processing address 192.168.135.93/24 in zone 0 (source type Active Discovery, source node [0])
2022.06.02 15:18:44.337 *D* [poll.discovery     ] FindExistingNodeByMAC: IP=192.168.135.93 MAC=

2022.06.02 15:18:49.598 *D* [poll.discovery     ] Active discovery - node 192.168.135.93 responded to SNMP probe
2022.06.02 15:18:49.598 *D* [poll.discovery     ] Checking address 192.168.135.93 in zone 0 (source: Active Discovery)
2022.06.02 15:18:49.598 *D* [poll.discovery     ] New node queued: 192.168.135.93/24
2022.06.02 15:18:49.598 *D* [poll.discovery     ] NodePoller: processing address 192.168.135.93/24 in zone 0 (source type Active Discovery, source node [0])
2022.06.02 15:18:49.598 *D* [poll.discovery     ] FindExistingNodeByMAC: IP=192.168.135.93 MAC=

2022.06.02 15:18:54.849 *D* [poll.discovery     ] Active discovery - node 192.168.135.93 responded to SNMP probe
2022.06.02 15:18:54.849 *D* [poll.discovery     ] Checking address 192.168.135.93 in zone 0 (source: Active Discovery)
2022.06.02 15:18:54.849 *D* [poll.discovery     ] New node queued: 192.168.135.93/24
2022.06.02 15:18:54.849 *D* [poll.discovery     ] NodePoller: processing address 192.168.135.93/24 in zone 0 (source type Active Discovery, source node [0])
2022.06.02 15:18:54.849 *D* [poll.discovery     ] FindExistingNodeByMAC: IP=192.168.135.93 MAC=

2022.06.02 15:19:01.443 *D* [poll.discovery     ] Active discovery - node 192.168.135.93 responded to SNMP probe
2022.06.02 15:19:01.443 *D* [poll.discovery     ] Checking address 192.168.135.93 in zone 0 (source: Active Discovery)
2022.06.02 15:19:01.443 *D* [poll.discovery     ] New node queued: 192.168.135.93/24
2022.06.02 15:19:02.240 *D* [poll.discovery     ] NodePoller: processing address 192.168.135.93/24 in zone 0 (source type Active Discovery, source node [0])

2022.06.02 15:19:03.037 *D* [poll.discovery     ] FindExistingNodeByMAC: IP=192.168.135.93 MAC=

2022.06.02 15:19:55.062 *D* [poll.discovery     ] Finished active discovery check on range 192.168.139.1 - 192.168.135.254

note the end range "192.168.139.1 - 192.168.135.254" should be 135.1-135.254 instead

[Filipp Sudanov]

range 192.168.139.1 - 192.168.135.254  - this was incorrect debug message, was just fixed will be in next patch release.

What exactly version you currently use?

[Egert143]

Currently running 4.1.333

[Filipp Sudanov]

Descovery debugging was a bit improved. Fix will be included in next patch release. Please try the new version when it will be available and share the log.

[Egert143]

Will do.