thank you for the detailed analysis. We checked our current NetXMS 6.2.5 installation.

1. Both Dell drivers are present:

dell.ndd 426K
dell-pwc.ndd 168K

2. Old NetXMS libraries are still present in /usr/local/lib:
   We found both .so.50 and .so.61 versions, including the libraries you mentioned:

libnxsrv.so.61
libnxsl.so.61
libnetxms.so.61
..

3. The old Dell driver is actively loaded under NetXMS 6.2.5.
   Our server log contains several entries such as:

Network device driver DELL-PWC loaded successfully

This confirms that the old driver is still being loaded in our current 6.2.5 installation.
We have not removed or modified any files yet, as the current production server is still running 6.2.5.

Your explanation sounds very plausible and matches what we found.
We will prepare a clean upgrade to 7.0.2, removing the obsolete driver and libraries as suggested.

Thanks again for investigating this issue!

Best regards
Alwin

In 6.2 the Dell driver was renamed from dell-pwc to dell. make install never removes files, so your tree, which went through 5.x and 6.1 (the .so.50 and .so.61 libraries are still in /usr/local/lib), most likely still has the old /usr/local/lib/netxms/ndd/dell-pwc.ndd. At startup 7.0.2 opens every driver in that directory. Opening the old one pulls in the 6.1 libraries it was linked against (libnxsrv.so.61, libnxsl.so.61, libnetxms.so.61), and their initialisation code overwrites part of the 7.0.2 script engine, including the definition of the Table class. The server then rejects the driver because of the API version and unloads it, which leaves the Table class broken. The first script that calls a Table method crashes in exactly the lookup in your backtrace. Your readAgentTable DCI does that within seconds of startup. 6.2.5 isn't affected because the old driver's API version still matched there, so it stayed loaded.

To confirm:

ls -l /usr/local/lib/netxms/ndd/
grep -i 'dell-pwc' /var/log/netxmsd*    # or wherever LogFile points

The 7.0.2 log should have a line Network device driver "dell-pwc.ndd" cannot be loaded because of API version mismatch (driver: 13; server: 14). Ignore my earlier info sharedlibrary request: the old libraries are unloaded right after that check, so they may not appear in the core at all.

For the next upgrade, move /usr/local/lib/netxms away before make install and remove the old libnx*.so.50*, .so.61* and libnetxms.so.50* / .so.61* libraries from /usr/local/lib. Nothing in 6.2.5 or 7.0.2 uses them.

The coredump explains why the offsets didn't match our packages: it's your own build, with full debug info, so the stack trace itself is accurate. The old .so.50/.61/.62 libraries in /usr/local/lib are not a problem - 7.0.2 links only the .so.70 ones, and those are the only ones in the dump. What make install over an existing tree doesn't clean up is /usr/local/lib/netxms. The server loads every network device driver and notification channel driver from there at startup, so a file left over from an older build would put old code into the 7.0.2 process. The dump you sent doesn't show which files were loaded.

Both questions can be answered from the core you already have, with 6.2.5 running. systemd deletes cores after two weeks by default, so copy it somewhere safe first:

coredumpctl dump 107876 -o /root/netxmsd-7.0.2.core
ls -l /usr/local/lib/libnxsl.so.70*     # 7.0.2 libraries must still be in place

Then open it with the 7.0.2 netxmsd from the build tree. If /tmp/netxms-7.0.2 is gone, build the same tarball again with the same configure options (make only, no make install):

gdb /tmp/netxms-7.0.2/src/server/netxmsd/.libs/netxmsd /root/netxmsd-7.0.2.core
(gdb) set logging file /root/netxmsd-bt.txt
(gdb) set logging on
(gdb) info sharedlibrary
(gdb) frame function NXSL_VM::callMethod
(gdb) p name
(gdb) p *object
(gdb) p *object->m_class
(gdb) thread apply all bt full
(gdb) quit

Some values may show as <optimized out>, that's fine. Please attach /root/netxmsd-bt.txt, plus the output of ls -lR /usr/local/lib/netxms and any Module or driver lines from netxmsd.conf (without passwords).

If gdb lists libraries from an older version, the fix is a clean install - move /usr/local/lib/netxms away before make install. If everything loaded is 7.0.2, the backtrace goes into the tracking issue #3889, and the test with script DCIs disabled stays as the plan for the next window.

Thank you for your detailed analysis on github.

I would like to provide some additional information about our installation procedure and attach two files for your investigation:

coredump_7.0.2_first_try.txt – the complete core dump information from our first attempt with NetXMS 7.0.2.
NetXMS_packages_6.2.5_before_Update_to_7.0.2.txt – package and library information collected from our currently running NetXMS 6.2.5 installation, before upgrading to 7.0.2.
Our NetXMS installations are not based on Ubuntu/Debian packages. We always download the official source tarball and compile NetXMS locally.

Our upgrade procedure is as follows:

cd /tmp
wget https://netxms.org/download/releases/7.0/netxms-7.0.2.tar.gz
tar zxvf netxms-7.0.2.tar.gz
cd netxms-7.0.2
./configure --enable-release-build --with-server --with-odbc --with-agent --with-internal-libargon2
make
make install
nxdbmgr upgrade
systemctl start netxmsd

This might also explain why the package-related commands (dpkg and apt-cache) do not provide any useful information in our environment.

Would the attached information and our installation procedure be helpful for your further investigation? Perhaps this already provides some additional clues before we schedule another attempt with NetXMS 7.0.2.

As a next step, we are planning to repeat the test with script DCIs disabled to determine whether they are involved in triggering the crash.

Since another test would require a maintenance window and a subsequent rollback to 6.2.5, we would appreciate your feedback on whether the information provided so far might already help narrow down the issue.

Best regards
Alwin

coredump_7.0.2_first_try.txt

NetXMS_packages_6.2.5_before_Update_to_7.0.2.txt

I can't reproduce this locally. Your script runs fine on 7.0.2 here, and nobody else has reported this crash. I've opened an issue for it: https://github.com/netxms/netxms/issues/3889

The stack shows a crash while a script DCI calls a method on an object (node, table, etc.). Every script DCI goes through that same path, though, so it doesn't prove this particular script is the one. The Ubuntu updates you listed are very unlikely to be related.

If you still have the core dump, please send a full backtrace and the list of installed NetXMS packages:

coredumpctl gdb netxmsd
(gdb) thread apply all bt full
(gdb) quit

dpkg -l | grep netxms

If you try the upgrade again, disable the script DCIs on 6.2.5 first. If 7.0.2 then stays up, we know it's script-related and can narrow it down from there.