Will nodes be rediscovered by auto discovery?

[lweidig]

There is nothing for these nodes when logged in under system or using nxadm -c "show objects".  I have also as the other person tried restarting the server to see if they will get automatically discovered again and no luck with that.

[Tursiops]

Hi,

I wiped a number of discovered nodes from the system last night, ran hkrun and checked show objects before and after.
The nodes were definitely no longer in the list, but have not been rediscovered yet (~10 hours later, with passive discovery meant to run every 15 minutes).

Discovery also doesn't seem to pick up network changes. It really behaves like it only runs once. Is there some flag in the database that might be stuck?

Cheers

[victor]

Hi,

very weird. Please check server queues and try to run server with debug level 6 fr some time and check messages with prefix DiscoveryPoller.

Best regards,
Victor

[Tursiops]

Hi,

The logs show a lot of "potential node x.x.x.x rejected (IP address already queued for polling)".
When I check for existing objects, I either can't find them or they are in a different zone to the node that's used for discovery.

As it mentioned the "queued for polling", I had a look at the queues and Node Poller is at 40k+. Looks like that's our problem.
This value rarely decreases by 1, but otherwise just keeps increasing. Quite possible that 40k is simply the figure of all IPs across our networks which NetXMS discovered and wants to check.
I am not quite sure which poller figure to increase for this in the server config. Status? Discovery? Is there a NumberOfNodePollers configuration item?

Cheers

[lweidig]

For us all of the queues are empty:

netxmsd: show queues
Data collector                   : 0
DCI cache loader                 : 0
Database writer                  : 0
Database writer (IData)          : 0
Database writer (raw DCI values) : 0
Event processor                  : 0
Node poller                      : 0
Syslog processing                : 0
Syslog writer                    : 0

Here is all of the debug output for one of the nodes not being rediscovered:

[25-Jul-2017 07:43:35.010] [DEBUG] DiscoveryPoller(): checking potential node 10.0.140.1 at shf00-pdu-00:1
[25-Jul-2017 07:43:35.011] [DEBUG] DiscoveryPoller(): new node queued: 10.0.140.1/22
[25-Jul-2017 07:43:35.011] [DEBUG] NodePoller: processing node 10.0.140.1/22 in zone 0
[25-Jul-2017 07:43:35.011] [DEBUG] GetOldNodeWithNewIP: ip=10.0.140.1 mac=E4:8D:8C:25:49:78
[25-Jul-2017 07:43:35.012] [DEBUG] AcceptNewNode(10.0.140.1): auto filter, flags=0004
[25-Jul-2017 07:43:35.012] [DEBUG] AcceptNewNode(10.0.140.1): auto filter - checking range
[25-Jul-2017 07:43:35.012] [DEBUG] AcceptNewNode(10.0.140.1): auto filter - range check result is 0
[25-Jul-2017 07:43:35.124] [DEBUG] DiscoveryPoller(): checking potential node 10.0.140.1 at shf00-pdu-00:1
[25-Jul-2017 07:43:35.128] [DEBUG] DiscoveryPoller(): new node queued: 10.0.140.1/22
[25-Jul-2017 07:43:35.128] [DEBUG] NodePoller: processing node 10.0.140.1/22 in zone 0
[25-Jul-2017 07:43:35.128] [DEBUG] GetOldNodeWithNewIP: ip=10.0.140.1 mac=00:00:00:00:00:00
[25-Jul-2017 07:43:35.129] [DEBUG] AcceptNewNode(10.0.140.1): auto filter, flags=0004
[25-Jul-2017 07:43:35.129] [DEBUG] AcceptNewNode(10.0.140.1): auto filter - checking range
[25-Jul-2017 07:43:35.132] [DEBUG] AcceptNewNode(10.0.140.1): auto filter - range check result is 0

But still the node is never getting added.  Also, both the Active Discovery Targets and Address Filters sections of the Network Discovery Configuration contain the address range for this IP.

[victor]

Hi,

from the log it seems that address range filter is not passed. Could you show how address filter is configured?

Best regards,
Victor

[lweidig]

It is an address range of 10.0.140.1 - 10.0.140.30.  Maybe it is not catching the end IP's properly?

Actually, that is exactly it.  Changed my address range to 10.0.104.0 - 10.0.140.30 and it detected the router.  Either it needs fixing or we need to adjust all of our ranges?  Can you also tell if the upper end of the range has the same issue?

[victor]

It's a bug in a server - first address of the range always ignored. I just fixed it in development branch (fix will be included into 2.1.1 patch release).

Best regards,
Victor

[lweidig]

Thank you, appreciate it.

[Tursiops]

Hi,

I believe I found the source of my problem as well.
A switch which was sending syslog data to a proxy node which was incorrectly configured with Zone ID 0 (default).
For every syslog message received, the logs showed that NetXMS was adding the same IP over and over again to the poller queue (I am not sure why it failed to detect that this IP was already in the queue in this particular instance?)
That node also just happened to be generating a syslog message every few seconds due to another monitoring tool setup and controller by a third party trying to connect to the switch using "public" (and failing, thus generating a log entry).

I fixed the Zone ID and now NetXMS can properly link the incoming messages to the node and our queue is in single digits now.

Still leaves the question how the IP could be added to the poller queue over and over?  ???

Cheers

[victor]

Hi,

there was a bug in server - check for already queued address was not performed for addresses discovered from syslog messages or SNMP traps. Just fixed it in development branch, fix will be included in 2.1.1.

Best regards,
Victor