NetXMS Forum

Victor Kirhenshtein

We just published patch release 3 for version 7.0.

Notable changes

New server configuration variable Agent.TLS.ReverseTunnels (disabled/allowed/preferred) for controlling use of TLS tunels to agents
Modbus: unit ID from DCI metric and from Modbus sensor is honored; DCIs on sensors are collected with sensor's unit ID
File transfer over encrypted connection is aborted on encryption or send failure instead of being reported as successful with missing data
New agent and server configuration parameter CurlSSLBackend
libcurl uses Windows certificate store when no CA bundle is configured
Docker connector no longer disables TLS verification on Windows
New AI assistant tool get-node-health (one-call OS health snapshot of a node with pre-evaluated findings)
AI security fixes
DCI data export includes delta method, transformation flag, time zone and SHA-256 digests
Network map link color calculation script can access link via global variable $link
FDB entries with status other(1) (e.g. MAC addresses authorized by port security or 802.1X) are no longer discarded
JUNIPER driver recognizes CLI prompt on devices without configured host name
SYS_NODE_UNREACHABLE events caused by agent tunnel being down are correlated to SYS_NETWORK_CONN_LOST
Hardware inventory: storage device and network adapter index changes no longer cause spurious component added/removed events
Windows: mysql.ddr is treated as alias for mariadb.ddr (MySQL driver removed from Windows build)
Agent web service requests: fixed bearer authentication; improved workarounds for memory leaks in old libcurl versions
Added Chinese (Simplified) translation of management console

Fixed issues

#3868 (Hardware inventory: storage device index changes cause spurious component added/removed events)
#3869 (Bundled libcurl on Windows has no trusted CA roots, https verification fails with curl error 60)
#3871 (Docker connector disables TLS verification on Windows and depends on OpenSSL-only curl callback)
#3872 (Empty user variable values cannot be stored on Oracle (ORA-01400 on user_profiles.var_value))
#3873 (Decommission with "Clear IP addresses" leaves interfaces in zone address index: former IP "already in use" until server restart when zoning is enabled)
#3874 (Windows server installer: upgrade from <7.0 with DBDriver = mysql.ddr leaves server unable to start)
#3875 (Create public SSH key from private automatically)
#3876 (SSH terminal via agent proxy disconnects after login (libssh 0.11+))
#3877 (Modbus data collection ignores unit ID from DCI metric and from Modbus sensors)
#3879 (Web console: inactivity timeout never fires while node Overview is open)
#3881 (HPE Comware FDB entries with status other(1) are discarded — configurable acceptance policy requested)
#3883 (AI tool get-node-health: one-call OS health snapshot of a node with pre-evaluated findings)
#3885 (MinGW subagent build: embedded resource (parser XML) is not rebuilt when the embedded file changes)
#3892 (MCP endpoint must work standalone without AI provider: core tools, NXSL ai_tool scripts and disabled-items list are skipped when InitAIAssistant() bails out on missing provider)
#3893 (Device driver data is not reset when node's network device driver changes)
#3895 (Data export download fails in web console)
#3897 (aifileops log_tail/log_grep only see the first 10000 lines of a file)
#3898 (AI log/incident search tools: no row limit on Oracle, negative limit accepted, hard-coded LIMIT breaks MSSQL/Oracle/DB2)
#3899 (execute-agent-tool discards agent error details and returns only numeric error code)
#3900 (Add server configuration variable Agent.TLS.ReverseTunnels (disabled/allowed/preferred) to control server-initiated (reverse) agent TLS tunnels globally)
#3901 (Node::connectToAgent() establishes reverse TLS tunnel while reusing alive legacy NXCP connection, leaving tunnel unused)
#3903 (Allow map link color scripts to distinguish parallel links)
#3904 (JUNIPER driver: CLI prompt without host name not recognized, {master:0} line left in configuration read via interactive SSH)
#3905 (File transfer over encrypted connection reported success with missing data)
#3906 (AI operator owner can be set to any user, allowing privilege escalation via MANAGE_AI_OPERATORS)
#3907 (Race condition in server object index reader/writer handshake)
#3908 (NetXMS 7.0.2: recurring agent tunnel timeouts, proxy polling failures and unexpected agent process termination)

Alex Kirhenshtein

@gmonk63 said:

I tried to upload netxmsd-bt.txt but it says I do not have permissions.

After forum migration? Thanks, I’ll check permissions

UPDATE: permissions fixed

Victor Kirhenshtein

Thank you! Just added to the source tree. Note that before nxmc had only Chinese Traditional translation, so I have added yours as separate language entry "Chinese (Simplified)".

Could you please also translate client error messages in src/client/java/netxms-client/src/main/resources/netxms-client-messages.properties?

Best regards,
Victor

Alex Kirhenshtein

Possible cause, quick to check: if this server was ever on 6.1 or older, make install has left the old Dell driver in place. In 6.2 it was renamed from dell-pwc to dell, and nothing removes the old file. 7.0.2 opens it at startup, which loads the old NetXMS libraries into the server process and corrupts its memory. Please post the output of:

ls -l /usr/local/lib/netxms/ndd/
ls /usr/local/lib/libnetxms.so*
grep -i 'API version mismatch' <netxmsd log file>

If dell-pwc.ndd is there, delete it and restart netxmsd. The current driver is dell.ndd.

Alex Kirhenshtein

I think I found it, and it can be checked without a maintenance window.

In 6.2 the Dell driver was renamed from dell-pwc to dell. make install never removes files, so your tree, which went through 5.x and 6.1 (the .so.50 and .so.61 libraries are still in /usr/local/lib), most likely still has the old /usr/local/lib/netxms/ndd/dell-pwc.ndd. At startup 7.0.2 opens every driver in that directory. Opening the old one pulls in the 6.1 libraries it was linked against (libnxsrv.so.61, libnxsl.so.61, libnetxms.so.61), and their initialisation code overwrites part of the 7.0.2 script engine, including the definition of the Table class. The server then rejects the driver because of the API version and unloads it, which leaves the Table class broken. The first script that calls a Table method crashes in exactly the lookup in your backtrace. Your readAgentTable DCI does that within seconds of startup. 6.2.5 isn't affected because the old driver's API version still matched there, so it stayed loaded.

To confirm:

ls -l /usr/local/lib/netxms/ndd/
grep -i 'dell-pwc' /var/log/netxmsd*    # or wherever LogFile points

The 7.0.2 log should have a line Network device driver "dell-pwc.ndd" cannot be loaded because of API version mismatch (driver: 13; server: 14). Ignore my earlier info sharedlibrary request: the old libraries are unloaded right after that check, so they may not appear in the core at all.

For the next upgrade, move /usr/local/lib/netxms away before make install and remove the old libnx*.so.50*, .so.61* and libnetxms.so.50* / .so.61* libraries from /usr/local/lib. Nothing in 6.2.5 or 7.0.2 uses them.

Alex Kirhenshtein

This may be the same crash as NetXMS 7.0.2 crashes with SIGSEGV shortly after startup: also a 6.2.5 → 7.0.2 source upgrade with make install over the existing tree, and a segfault in a hash table lookup shortly after startup. It's tracked in #3889.

To confirm, please add the output of info sharedlibrary from the same gdb session, and ls -lR /usr/local/lib/netxms. make install leaves files from older builds in that directory, and the server loads drivers from it at startup.

Alex Kirhenshtein

Yes, it helps, and you don't need another maintenance window for the next step.

The coredump explains why the offsets didn't match our packages: it's your own build, with full debug info, so the stack trace itself is accurate. The old .so.50/.61/.62 libraries in /usr/local/lib are not a problem - 7.0.2 links only the .so.70 ones, and those are the only ones in the dump. What make install over an existing tree doesn't clean up is /usr/local/lib/netxms. The server loads every network device driver and notification channel driver from there at startup, so a file left over from an older build would put old code into the 7.0.2 process. The dump you sent doesn't show which files were loaded.

Both questions can be answered from the core you already have, with 6.2.5 running. systemd deletes cores after two weeks by default, so copy it somewhere safe first:

coredumpctl dump 107876 -o /root/netxmsd-7.0.2.core
ls -l /usr/local/lib/libnxsl.so.70*     # 7.0.2 libraries must still be in place

Then open it with the 7.0.2 netxmsd from the build tree. If /tmp/netxms-7.0.2 is gone, build the same tarball again with the same configure options (make only, no make install):

gdb /tmp/netxms-7.0.2/src/server/netxmsd/.libs/netxmsd /root/netxmsd-7.0.2.core
(gdb) set logging file /root/netxmsd-bt.txt
(gdb) set logging on
(gdb) info sharedlibrary
(gdb) frame function NXSL_VM::callMethod
(gdb) p name
(gdb) p *object
(gdb) p *object->m_class
(gdb) thread apply all bt full
(gdb) quit

Some values may show as <optimized out>, that's fine. Please attach /root/netxmsd-bt.txt, plus the output of ls -lR /usr/local/lib/netxms and any Module or driver lines from netxmsd.conf (without passwords).

If gdb lists libraries from an older version, the fix is a clean install - move /usr/local/lib/netxms away before make install. If everything loaded is 7.0.2, the backtrace goes into the tracking issue #3889, and the test with script DCIs disabled stays as the plan for the next window.

Alex Kirhenshtein

Hi,

It's the server crashing, not the client: netxmsd segfaults, the client loses the connection and exits, and systemd restarts the server, which is why the service still shows as active.

The screenshot says 7.0.2. I assume "7.2.0" in the title is a typo?

To see where it crashes I need a full backtrace. If systemd-coredump is installed, take it from the last core:

coredumpctl list netxmsd
coredumpctl gdb netxmsd
(gdb) set logging file /tmp/netxmsd-bt.txt
(gdb) set logging on
(gdb) thread apply all bt full
(gdb) quit

Debian 11 doesn't install it by default (coredumpctl: command not found). In that case either apt install systemd-coredump and wait for the next crash, or run the server under gdb until it crashes:

systemctl stop netxmsd      # gdb runs its own copy, the service must not run in parallel
gdb -ex 'handle SIGPIPE nostop noprint' -ex run --args /usr/local/bin/netxmsd
# wait for "Thread ... received signal SIGSEGV", then:
(gdb) set logging file /tmp/netxmsd-bt.txt
(gdb) set logging on
(gdb) thread apply all bt full
(gdb) quit

Please attach /tmp/netxmsd-bt.txt. A source build keeps debug symbols by default, so there's nothing extra to install.

Alex Kirhenshtein

Hi Alwin,

I can't reproduce this locally. Your script runs fine on 7.0.2 here, and nobody else has reported this crash. I've opened an issue for it: https://github.com/netxms/netxms/issues/3889

The stack shows a crash while a script DCI calls a method on an object (node, table, etc.). Every script DCI goes through that same path, though, so it doesn't prove this particular script is the one. The Ubuntu updates you listed are very unlikely to be related.

If you still have the core dump, please send a full backtrace and the list of installed NetXMS packages:

coredumpctl gdb netxmsd
(gdb) thread apply all bt full
(gdb) quit

dpkg -l | grep netxms

If you try the upgrade again, disable the script DCIs on 6.2.5 first. If 7.0.2 then stays up, we know it's script-related and can narrow it down from there.

Alex Kirhenshtein

Hi,

Yes, this works with a self-hosted UniFi OS Server, and SNMP isn't needed. The bridge doesn't use the node at all. The NetXMS server calls the UniFi Network API directly, and everything is configured as custom attributes on the wireless domain object. You can remove the VM node from the domain, because it does nothing in this mode.

Add Module = wlcbridge to netxmsd.conf and restart the server.
On the wireless domain, add these custom attributes:

wlcbridge        UniFi
unifi.base-url   https://<vm-ip>:<port>    # same URL you open the UniFi web UI on
unifi.login      <local admin account>
unifi.password   <password>                # plain text, or encrypted with: nxencpasswd <login> <password>
unifi.site       <site id>                 # only if not "default"

Instead of login/password you can set unifi.token to an API key created in the Network application. Login/password is the safer option, though. The account has to be a local one, not a UI.com cloud account.

Run a configuration poll on the wireless domain. The output should show "Reading access point list via bridge interface UniFi" and the number of access points reported. The access points are created under the domain.

Important detail: leave unifi.isAppliance unset. The name is misleading: true is for the old standalone Network Application (the one on port 8443), and UniFi OS Server needs it unset.

If the poll reports "Cannot retrieve access point list", set debug level 7 for tag wlcbridge.unifi. The server log will then show the exact URL being requested and the HTTP code that came back.

NetXMS Forum

administrators

Posts

Notable changes

Fixed issues

Member List