Client Crash after upgrading from 6.2.5 to 7.2.0

[gmonk63]

Client crashes with attached message

Server: "Debian GNU/Linux 11 (bullseye)

Database: timescaledb-2-postgresql-17        2.21.3~debian11

Installed from source: ./configure --with-server --with-client --with-snmp --with-pgsql --with-agent --with-client-proxy --with-vmgr --with-mosquitto

*compiles and installs with no issues

*nxdbmgr upgrade was successful

*netxmsd service shows as active along with postgres

/var/log/messages show several of the following \

Nov  9 11:36:35 NMS kernel: [ 3956.439039] $POLLERS/WRK[95111]: segfault at 7400000074 ip 00007fc4eb9a414f sp 00007fc4e1f380b0 error 4
Nov  9 11:36:35 NMS kernel: [ 3956.439045] Code: 44 31 d9 29 cf 29 c8 89 fa 89 cf c1 e7 10 31 d7 29 f8 29 f9 41 89 c5 89 f8 c1 e8 05 44 31 e8 89 c2 29 c1 29 c7 c1 ea 03 31 d1 <49> 8b 11 41 89 cd 29 cf 29 c8 41 c1 e5 0a 44 31 ef 29

Thanks
 

---

[alk]

Hi,

It's the server crashing, not the client: netxmsd segfaults, the client loses the connection and exits, and systemd restarts the server, which is why the service still shows as active.

The screenshot says 7.0.2. I assume "7.2.0" in the title is a typo?

To see where it crashes I need a full backtrace. If systemd-coredump is installed, take it from the last core:

coredumpctl list netxmsd
coredumpctl gdb netxmsd
(gdb) set logging file /tmp/netxmsd-bt.txt
(gdb) set logging on
(gdb) thread apply all bt full
(gdb) quit

Debian 11 doesn't install it by default (coredumpctl: command not found). In that case either apt install systemd-coredump and wait for the next crash, or run the server under gdb until it crashes:

systemctl stop netxmsd      # gdb runs its own copy, the service must not run in parallel
gdb -ex 'handle SIGPIPE nostop noprint' -ex run --args /usr/local/bin/netxmsd
# wait for "Thread ... received signal SIGSEGV", then:
(gdb) set logging file /tmp/netxmsd-bt.txt
(gdb) set logging on
(gdb) thread apply all bt full
(gdb) quit

Please attach /tmp/netxmsd-bt.txt. A source build keeps debug symbols by default, so there's nothing extra to install.

[alk]

This may be the same crash as NetXMS 7.0.2 crashes with SIGSEGV shortly after startup: also a 6.2.5 → 7.0.2 source upgrade with make install over the existing tree, and a segfault in a hash table lookup shortly after startup. It's tracked in #3889.

To confirm, please add the output of info sharedlibrary from the same gdb session, and ls -lR /usr/local/lib/netxms. make install leaves files from older builds in that directory, and the server loads drivers from it at startup.

[alk]

Possible cause, quick to check: if this server was ever on 6.1 or older, make install has left the old Dell driver in place. In 6.2 it was renamed from dell-pwc to dell, and nothing removes the old file. 7.0.2 opens it at startup, which loads the old NetXMS libraries into the server process and corrupts its memory. Please post the output of:

ls -l /usr/local/lib/netxms/ndd/
ls /usr/local/lib/libnetxms.so*
grep -i 'API version mismatch' <netxmsd log file>

If dell-pwc.ndd is there, delete it and restart netxmsd. The current driver is dell.ndd.

[gmonk63]

I tried to upload netxmsd-bt.txt but it says I do not have permissions.

Here are the results from the following commands

ls -lR /usr/local/lib/netxms

/usr/local/lib/netxms/:
total 45532
-rwxr-xr-x 1 root root 440344 Nov 9 11:37 aifileops.nsm
-rwxr-xr-x 1 root root 11420448 Nov 9 11:37 aitools.nxm
-rwxr-xr-x 1 root root 1539360 Nov 9 11:37 asterisk.nsm
-rwxr-xr-x 1 root root 173832 Nov 9 11:37 bind9.nsm
drwxr-xr-x 2 root root 4096 Nov 9 11:37 dbdrv
-rwxr-xr-x 1 root root 498992 Nov 9 11:37 dbquery.nsm
-rwxr-xr-x 1 root root 917448 Nov 9 11:37 devbackup.nxm
-rwxr-xr-x 1 root root 142368 Nov 9 11:37 devemu.nsm
-rwxr-xr-x 1 root root 563200 Nov 9 11:37 docker.nxm
-rwxr-xr-x 1 root root 107744 Nov 9 11:37 ds18x20.nsm
-rwxr-xr-x 1 root root 95456 Mar 30 2023 ecs.nsm
-rwxr-xr-x 1 root root 582496 Nov 9 11:37 entsoe.nsm
-rwxr-xr-x 1 root root 121296 Nov 9 11:37 fbdev.nsm
-rwxr-xr-x 1 root root 360936 Nov 9 11:37 filemgr.nsm
-rwxr-xr-x 1 root root 214512 Nov 9 11:37 gps.nsm
-rwxr-xr-x 1 root root 291872 Oct 6 11:05 jira.hdlink
-rwxr-xr-x 1 root root 469200 Nov 9 11:37 jira.nxm
-rwxr-xr-x 1 root root 1158840 Nov 9 11:37 kpidirect.nxm
-rwxr-xr-x 1 root root 517104 Nov 9 11:38 leef.nxm
-rwxr-xr-x 1 root root 2049328 Nov 9 11:37 linux.nsm
-rwxr-xr-x 1 root root 506072 Nov 9 11:37 logwatch.nsm
-rwxr-xr-x 1 root root 397208 Nov 9 11:38 mcp.nxm
-rwxr-xr-x 1 root root 503776 Nov 9 11:37 mqtt.nsm
-rwxr-xr-x 1 root root 493928 Nov 9 11:37 mssql.nsm
drwxr-xr-x 2 root root 4096 Nov 10 09:50 ncdrv
drwxr-xr-x 3 root root 4096 Nov 10 09:49 ndd
-rwxr-xr-x 1 root root 728864 Nov 9 11:37 netconf.nsm
-rwxr-xr-x 1 root root 674200 Nov 9 11:37 netsvc.nsm
-rwxr-xr-x 1 root root 884976 Nov 9 11:38 ntcb.nxm
-rwxr-xr-x 1 root root 838440 Nov 9 11:38 ntopng.nxm
-rwxr-xr-x 1 root root 354664 Aug 10 11:25 openmeteo.nsm
-rwxr-xr-x 1 root root 1205848 Nov 9 11:38 oxidized.nxm
drwxr-xr-x 2 root root 4096 Nov 9 11:38 pdsdrv
-rwxr-xr-x 1 root root 509008 Nov 9 11:37 pgsql.nsm
-rwxr-xr-x 1 root root 249136 Nov 9 11:37 ping.nsm
-rwxr-xr-x 1 root root 351056 Mar 30 2023 portcheck.nsm
-rwxr-xr-x 1 root root 162432 Oct 6 11:05 redmine.hdlink
-rwxr-xr-x 1 root root 344144 Nov 9 11:38 redmine.nxm
-rwxr-xr-x 1 root root 166224 Nov 9 11:37 sms.nsm
-rwxr-xr-x 1 root root 433392 Jan 26 2026 spe.nxm
-rwxr-xr-x 1 root root 716480 Nov 9 11:37 ssh.nsm
-rwxr-xr-x 1 root root 428472 Nov 9 11:37 ups.nsm
-rwxr-xr-x 1 root root 533272 Nov 9 11:37 vmgr.nsm
-rwxr-xr-x 1 root root 313296 Nov 9 11:38 wcc.nxm
-rwxr-xr-x 1 root root 544592 Nov 9 11:37 weather.nsm
-rwxr-xr-x 1 root root 11521960 Nov 9 11:38 webapi.nxm
-rwxr-xr-x 1 root root 2005448 Nov 9 11:38 wlcbridge.nxm

/usr/local/lib/netxms/dbdrv:
total 324
-rwxr-xr-x 1 root root 227992 Nov 9 11:37 pgsql.ddr
-rwxr-xr-x 1 root root 102248 Nov 9 11:37 sqlite.ddr

/usr/local/lib/netxms/ncdrv:
total 3696
-rwxr-xr-x 1 root root 108416 Nov 9 11:38 anysms.ncd
-rwxr-xr-x 1 root root 93024 Nov 9 11:38 dbtable.ncd
-rwxr-xr-x 1 root root 85896 Nov 9 11:38 dummy.ncd
-rwxr-xr-x 1 root root 151056 Nov 9 11:38 googlechat.ncd
-rwxr-xr-x 1 root root 126152 Nov 9 11:38 gsm.ncd
-rwxr-xr-x 1 root root 107384 Nov 9 11:38 kannel.ncd
-rwxr-xr-x 1 root root 211696 Nov 9 11:38 matrix.ncd
-rwxr-xr-x 1 root root 256016 Nov 9 11:38 mattermost.ncd
-rwxr-xr-x 1 root root 93736 Nov 9 11:38 mqtt.ncd
-rwxr-xr-x 1 root root 274744 Nov 9 11:38 msteams.ncd
-rwxr-xr-x 1 root root 111408 Nov 9 11:38 mymobile.ncd
-rwxr-xr-x 1 root root 113608 Nov 9 11:38 nexmo.ncd
-rwxr-xr-x 1 root root 174568 Nov 9 11:38 nxagent.ncd
-rwxr-xr-x 1 root root 126224 Nov 9 11:38 portech.ncd
-rwxr-xr-x 1 root root 97528 Nov 9 11:38 shell.ncd
-rwxr-xr-x 1 root root 109832 Nov 9 11:38 slack.ncd
-rwxr-xr-x 1 root root 109224 Nov 9 11:38 smseagle.ncd
-rwxr-xr-x 1 root root 140152 Nov 9 11:38 smtp.ncd
-rwxr-xr-x 1 root root 205728 Nov 9 11:38 snmptrap.ncd
-rwxr-xr-x 1 root root 393544 Nov 9 11:38 telegram.ncd
-rwxr-xr-x 1 root root 110488 Nov 9 11:38 text2reach.ncd
-rwxr-xr-x 1 root root 87528 Nov 9 11:38 textfile.ncd
-rwxr-xr-x 1 root root 119808 Nov 9 11:38 twilio.ncd
-rwxr-xr-x 1 root root 208992 Nov 9 11:38 webhook.ncd
-rwxr-xr-x 1 root root 121680 Nov 9 11:38 websms.ncd

/usr/local/lib/netxms/ndd:
total 12804
-rwxr-xr-x 1 root root 183640 Nov 9 11:38 arista.ndd
-rwxr-xr-x 1 root root 279880 Nov 9 11:38 at.ndd
-rwxr-xr-x 1 root root 682168 Nov 9 11:38 avaya.ndd
-rwxr-xr-x 1 root root 218864 Nov 9 11:38 bdcom.ndd
-rwxr-xr-x 1 root root 135992 Nov 9 11:38 bulat.ndd
-rwxr-xr-x 1 root root 499912 Nov 9 11:38 cambium.ndd
-rwxr-xr-x 1 root root 1632928 Nov 9 11:38 cisco.ndd
-rwxr-xr-x 1 root root 475744 Nov 9 11:38 dell.ndd
-rwxr-xr-x 1 root root 239160 Nov 9 11:38 dlink.ndd
-rwxr-xr-x 1 root root 248824 Nov 9 11:38 edgecore.ndd
-rwxr-xr-x 1 root root 233744 Nov 9 11:38 eltex.ndd
-rwxr-xr-x 1 root root 250600 Nov 9 11:38 etherwan.ndd
-rwxr-xr-x 1 root root 138416 Nov 9 11:38 extreme.ndd
-rwxr-xr-x 1 root root 349152 Nov 9 11:38 fortinet.ndd
-rwxr-xr-x 1 root root 335744 Nov 9 11:38 hirschmann.ndd
-rwxr-xr-x 1 root root 925848 Nov 9 11:38 hpe.ndd
-rwxr-xr-x 1 root root 413392 Nov 9 11:38 huawei.ndd
-rwxr-xr-x 1 root root 133720 Nov 9 11:38 ignitenet.ndd
-rwxr-xr-x 1 root root 460648 Nov 9 11:38 juniper.ndd
-rwxr-xr-x 1 root root 239440 Nov 9 11:38 lenovo.ndd
-rwxr-xr-x 1 root root 352336 Nov 9 11:38 mds.ndd
-rwxr-xr-x 1 root root 179120 Nov 9 11:38 mellanox.ndd
-rwxr-xr-x 1 root root 407896 Nov 9 11:38 mikrotik.ndd
-rwxr-xr-x 1 root root 212736 Nov 9 11:38 moxa.ndd
-rwxr-xr-x 1 root root 134824 Nov 9 11:38 netonix.ndd
-rwxr-xr-x 1 root root 331040 Nov 9 11:38 net-snmp.ndd
drwxr-xr-x 2 root root 4096 Nov 10 09:50 old
-rwxr-xr-x 1 root root 201080 Nov 9 11:38 origo.ndd
-rwxr-xr-x 1 root root 203248 Nov 9 11:38 ping3.ndd
-rwxr-xr-x 1 root root 394912 Nov 9 11:38 planet.ndd
-rwxr-xr-x 1 root root 353248 Nov 9 11:38 qtech.ndd
-rwxr-xr-x 1 root root 308112 Nov 9 11:38 rittal.ndd
-rwxr-xr-x 1 root root 233520 Nov 9 11:38 saf.ndd
-rwxr-xr-x 1 root root 241136 Nov 9 11:38 siemens.ndd
-rwxr-xr-x 1 root root 184168 Nov 9 11:38 symbol-ws.ndd
-rwxr-xr-x 1 root root 178904 Nov 9 11:38 tb.ndd
-rwxr-xr-x 1 root root 242264 Nov 9 11:38 tplink.ndd
-rwxr-xr-x 1 root root 408616 Nov 9 11:38 ubnt.ndd
-rwxr-xr-x 1 root root 207432 Nov 9 11:38 westerstrand.ndd
-rwxr-xr-x 1 root root 176088 Nov 9 11:38 zyxel.ndd

/usr/local/lib/netxms/ndd/old:
total 404
-rwxr-xr-x 1 root root 170880 Aug 10 11:25 dell-pwc.ndd
-rwxr-xr-x 1 root root 136864 Sep 5 2023 qtech-olt.ndd
-rwxr-xr-x 1 root root 99096 Jul 16 2024 xmpp.ncd

/usr/local/lib/netxms/pdsdrv:
total 1492
-rwxr-xr-x 1 root root 985552 Nov 9 11:38 clickhouse.pdsd
-rwxr-xr-x 1 root root 538536 Nov 9 11:38 influxdb.pdsd

Results from ls /usr/local/lib/libnetxms.so*

/usr/local/lib/libnetxms.so
/usr/local/lib/libnetxms.so.42
/usr/local/lib/libnetxms.so.42.0.0
/usr/local/lib/libnetxms.so.43
/usr/local/lib/libnetxms.so.43.0.0
/usr/local/lib/libnetxms.so.44
/usr/local/lib/libnetxms.so.44.0.0
/usr/local/lib/libnetxms.so.45
/usr/local/lib/libnetxms.so.45.0.0
/usr/local/lib/libnetxms.so.50
/usr/local/lib/libnetxms.so.50.0.0
/usr/local/lib/libnetxms.so.51
/usr/local/lib/libnetxms.so.51.0.0
/usr/local/lib/libnetxms.so.52
/usr/local/lib/libnetxms.so.52.0.0
/usr/local/lib/libnetxms.so.60
/usr/local/lib/libnetxms.so.60.0.0
/usr/local/lib/libnetxms.so.62
/usr/local/lib/libnetxms.so.62.0.0
/usr/local/lib/libnetxms.so.70
/usr/local/lib/libnetxms.so.70.0.0

grep -i 'API version mismatch' returned dell-pwc-ndd and xmpp.ncd I move both to an old folder i created and restarted the server and so far so good I have not had any crashes since

[alk]

@gmonk63 said:

I tried to upload netxmsd-bt.txt but it says I do not have permissions.

After forum migration? Thanks, I’ll check permissions

UPDATE: permissions fixed