duplicate nodes problem | can't save Nodes

Filipp Sudanov

Ok, let's try to get some debug information with this script: https://github.com/netxms/netxms/blob/master/tools/capture_netxmsd_threads.sh

In order for this to work you should have gdb and all relevant netxms-*-dbg packages installed. Is netxms installed from packages?

Script will produce output file in /tmp. Please attach it here.

Woody

Yes, NetXMS was installed from packages. I used this guide to install NetXMS. Before I started the script, I installed gdb and all relevant NetXMS packages with these commands:

# apt install netxms-*-dbg
# apt install gdb

And after that:

# ./capture_netxmsd_threads.sh

At the moment I executed the script, NetXMS hangs.
In /tmp I found netxmsd-threads.21769.20200101-212952. I attached this file here.

netxmsd-threads.21769.20200101-212952

Filipp Sudanov

I forgot to mention, that you should wait for netxms to hang first and then launch the script. So can you please wait for it to hang on it's own and then run the script.

Woody

In my post before I executed the script while NetXMS did hang. I didn't restart the netxmsd service since the crash I mentioned in my first post.

2019.12.31 12:48:36.346 *E* [                   ] Thread "Poll Manager" does not respond to watchdog thread
2019.12.31 12:48:56.347 *E* [                   ] Thread "Syncer Thread" does not respond to watchdog thread

So NetXMS still hangs.

I don't know if this can help, but I did it:
Now I restarted the netxmsd service and waited for netxms to hang. Than I launched the script. I have attached this file here.
Log file:

2020.01.02 10:16:52.647 *E* [                   ] Thread "Poll Manager" does not respond to watchdog thread
2020.01.02 10:17:12.647 *E* [                   ] Thread "Syncer Thread" does not respond to watchdog thread

I also attached the log file here. As you can see, i lauched the script 3 minutes after NetXMS hangs. Don't be confused about the time in the filename. It's because of the german time shift of one hour.

netxmsd

netxmsd-threads.5109.20200102-112004

Woody

When NetXMS hangs it doesn't save any changes that I make.

Woody

I have found out, that there is a diffrence between a full restart and when I only restart the netxmsd service.
For example when I make some changes and do service netxmsd restart my changes are there.
But when I do reboot my changes are gone.
And some changes are removed after some reboots and that is a very big problem.
When I do # nxdbmgr check after a reboot I get this error: Container 9190 contains non-existing child 79719. Fix it? (Yes/No/All/Skip) yes
Now my saved Nodes are away.

Victor Kirhenshtein

Yesterday we have fixed bug that cause deadlock on object access. It can be root cause for your issue as well. We will publish new patch release for 3.1 today - please check if it will help.

Best regards,
Victor

Woody

Hi,
thanks for fixing the bug. I will try it as soon as I get the new patch.

Woody

Could you please update packages?

Victor Kirhenshtein

deb build is in progress, will be available within hour or so.

Best regards,
Victor

Woody

Hello,
thank you for the update. I think the bug with "Thread "Poll Manager" does not respond to watchdog thread" and "Thread "Syncer Thread" does not respond to watchdog thread" has been fixed, because there was no error for more than 8 hours uptime now. But the other bug, that rainerh also mentioned here, still exists. Here is a copy of his post:

Hello,
I want do delete 2 (old) Template Groups.
After reboot, the 2 groups are still available.
How can I delete theses groups permanent?

Thank you

I have a new problem and I think the reason is the same like above:
When I create a new node, the new node will work fine until I reboot the NetXMS Server.
After reboot the new created node is deletet.

I attached the netxmsd log file here.
Now I checked db writer queue:

# nxadm -i
netxmsd: show queues

I got this output:

netxmsd: show queues
Data collector                   : 459
DCI cache loader                 : 0
Template updates                 : 0
Database writer                  : 0
Database writer (IData)          : 0
Database writer (raw DCI values) : 10321
Event processor                  : 0
Event log writer                 : 0
Poller                           : 0
Node discovery poller            : 0
Syslog processing                : 0
Syslog writer                    : 0
Scheduler                        : 0

When I execute this command multiple times I noticed that this value goes to approximately 12.000 - 20.000 and than quickly goes down to approx. 1000.

netxmsd: show queues
Data collector                   : 0
DCI cache loader                 : 0
Template updates                 : 0
Database writer                  : 0
Database writer (IData)          : 0
Database writer (raw DCI values) : 1263
Event processor                  : 0
Event log writer                 : 0
Poller                           : 0
Node discovery poller            : 0
Syslog processing                : 0
Syslog writer                    : 0
Scheduler                        : 0

netxmsd: show queues
Data collector                   : 0
DCI cache loader                 : 0
Template updates                 : 0
Database writer                  : 0
Database writer (IData)          : 73
Database writer (raw DCI values) : 14532
Event processor                  : 0
Event log writer                 : 0
Poller                           : 0
Node discovery poller            : 0
Syslog processing                : 0
Syslog writer                    : 0
Scheduler                        : 0

netxmsd

rainerh

Hello,

I found one of my problems.
When I configure ImportConfigurationOnStartup with "Only missing elements" (by default) and I delete Template "Windows" or "Generic UNIX" then it will come with next Boot, because these Templates belong to NetXMS and are not made by me.
I can change the value to "Never" and after reboot they will not come again.
Templates, which are made by me can be deleted at any time.

Thank you

But the problem, when I create a new node, the node will be deleted after 30 seconds, when I reboot NetXMS.
After reboot I can see the node in Management Console and after some seconds (about 30) the node is removed automatically.

Woody

This shows how my nodes gets deleted:

I create a new Node
You see the node is there
Now I reboot the server

# reboot

For a short time the node is still there
But after about 1 minute the node disappeares

I attached the log of this test here.
You can click on the images for better quality.

I hope someone knows what the problem is.
Thanks in advance!

netxmsd

rainerh

Hello

I have found the reason, but cannot manage it.
I have 2 identical Networks 192.168.11.0/24
Router 1 at Customer Voelk has on interface X1 192.168.11.1/24
Router 2 at Customer Koelbl has on interface X5192.168.11.1/24
Both have same Adress and I cannot change it.

2020.01.09 17:43:56.175 *D* [poll.conf          ] Primary IP address 192.168.11.1 of node Voelk SonicWALL TZ-500 [1268] found on interface X5 of node Koelbl Service SonicWALL NSA 2650 [80201]
2020.01.09 17:43:56.175 *D* [poll.conf          ] Node Voelk SonicWALL TZ-500 [1268] is a duplicate of node Koelbl Service SonicWALL NSA 2650 [80201]
2020.01.09 17:43:56.175 *D* [poll.conf          ] Removing node Koelbl Service SonicWALL NSA 2650 [80201] as duplicate

How can I resolve this Problem?

Thank you
Rainer

Victor Kirhenshtein

Hi,

one option is to use zones and put those routers into different zones. Another option is to mark internal interfaces as "exclude from topology". Second option is easier if you are not interested in anything behind those interfaces.

Best regards,
Victor

rainerh

Hello Victor,

I have tried to "exclude from topolog" only 1 Router. But these did not work.
Then I built some zones. Now It seems to work pretty fine and no node will be deleted again.

Thank you very much
Rainer

NetXMS Forum

duplicate nodes problem | can't save Nodes