NetXMS Forum

@gmonk63 said:

I tried to upload netxmsd-bt.txt but it says I do not have permissions.

After forum migration? Thanks, I’ll check permissions

UPDATE: permissions fixed

Possible cause, quick to check: if this server was ever on 6.1 or older, make install has left the old Dell driver in place. In 6.2 it was renamed from dell-pwc to dell, and nothing removes the old file. 7.0.2 opens it at startup, which loads the old NetXMS libraries into the server process and corrupts its memory. Please post the output of:

ls -l /usr/local/lib/netxms/ndd/
ls /usr/local/lib/libnetxms.so*
grep -i 'API version mismatch' <netxmsd log file>

If dell-pwc.ndd is there, delete it and restart netxmsd. The current driver is dell.ndd.

I think I found it, and it can be checked without a maintenance window.

In 6.2 the Dell driver was renamed from dell-pwc to dell. make install never removes files, so your tree, which went through 5.x and 6.1 (the .so.50 and .so.61 libraries are still in /usr/local/lib), most likely still has the old /usr/local/lib/netxms/ndd/dell-pwc.ndd. At startup 7.0.2 opens every driver in that directory. Opening the old one pulls in the 6.1 libraries it was linked against (libnxsrv.so.61, libnxsl.so.61, libnetxms.so.61), and their initialisation code overwrites part of the 7.0.2 script engine, including the definition of the Table class. The server then rejects the driver because of the API version and unloads it, which leaves the Table class broken. The first script that calls a Table method crashes in exactly the lookup in your backtrace. Your readAgentTable DCI does that within seconds of startup. 6.2.5 isn't affected because the old driver's API version still matched there, so it stayed loaded.

To confirm:

ls -l /usr/local/lib/netxms/ndd/
grep -i 'dell-pwc' /var/log/netxmsd*    # or wherever LogFile points

The 7.0.2 log should have a line Network device driver "dell-pwc.ndd" cannot be loaded because of API version mismatch (driver: 13; server: 14). Ignore my earlier info sharedlibrary request: the old libraries are unloaded right after that check, so they may not appear in the core at all.

For the next upgrade, move /usr/local/lib/netxms away before make install and remove the old libnx*.so.50*, .so.61* and libnetxms.so.50* / .so.61* libraries from /usr/local/lib. Nothing in 6.2.5 or 7.0.2 uses them.

This may be the same crash as NetXMS 7.0.2 crashes with SIGSEGV shortly after startup: also a 6.2.5 → 7.0.2 source upgrade with make install over the existing tree, and a segfault in a hash table lookup shortly after startup. It's tracked in #3889.

To confirm, please add the output of info sharedlibrary from the same gdb session, and ls -lR /usr/local/lib/netxms. make install leaves files from older builds in that directory, and the server loads drivers from it at startup.

Yes, it helps, and you don't need another maintenance window for the next step.

The coredump explains why the offsets didn't match our packages: it's your own build, with full debug info, so the stack trace itself is accurate. The old .so.50/.61/.62 libraries in /usr/local/lib are not a problem - 7.0.2 links only the .so.70 ones, and those are the only ones in the dump. What make install over an existing tree doesn't clean up is /usr/local/lib/netxms. The server loads every network device driver and notification channel driver from there at startup, so a file left over from an older build would put old code into the 7.0.2 process. The dump you sent doesn't show which files were loaded.

Both questions can be answered from the core you already have, with 6.2.5 running. systemd deletes cores after two weeks by default, so copy it somewhere safe first:

coredumpctl dump 107876 -o /root/netxmsd-7.0.2.core
ls -l /usr/local/lib/libnxsl.so.70*     # 7.0.2 libraries must still be in place

Then open it with the 7.0.2 netxmsd from the build tree. If /tmp/netxms-7.0.2 is gone, build the same tarball again with the same configure options (make only, no make install):

gdb /tmp/netxms-7.0.2/src/server/netxmsd/.libs/netxmsd /root/netxmsd-7.0.2.core
(gdb) set logging file /root/netxmsd-bt.txt
(gdb) set logging on
(gdb) info sharedlibrary
(gdb) frame function NXSL_VM::callMethod
(gdb) p name
(gdb) p *object
(gdb) p *object->m_class
(gdb) thread apply all bt full
(gdb) quit

Some values may show as <optimized out>, that's fine. Please attach /root/netxmsd-bt.txt, plus the output of ls -lR /usr/local/lib/netxms and any Module or driver lines from netxmsd.conf (without passwords).

If gdb lists libraries from an older version, the fix is a clean install - move /usr/local/lib/netxms away before make install. If everything loaded is 7.0.2, the backtrace goes into the tracking issue #3889, and the test with script DCIs disabled stays as the plan for the next window.

Hi,

It's the server crashing, not the client: netxmsd segfaults, the client loses the connection and exits, and systemd restarts the server, which is why the service still shows as active.

The screenshot says 7.0.2. I assume "7.2.0" in the title is a typo?

To see where it crashes I need a full backtrace. If systemd-coredump is installed, take it from the last core:

coredumpctl list netxmsd
coredumpctl gdb netxmsd
(gdb) set logging file /tmp/netxmsd-bt.txt
(gdb) set logging on
(gdb) thread apply all bt full
(gdb) quit

Debian 11 doesn't install it by default (coredumpctl: command not found). In that case either apt install systemd-coredump and wait for the next crash, or run the server under gdb until it crashes:

systemctl stop netxmsd      # gdb runs its own copy, the service must not run in parallel
gdb -ex 'handle SIGPIPE nostop noprint' -ex run --args /usr/local/bin/netxmsd
# wait for "Thread ... received signal SIGSEGV", then:
(gdb) set logging file /tmp/netxmsd-bt.txt
(gdb) set logging on
(gdb) thread apply all bt full
(gdb) quit

Please attach /tmp/netxmsd-bt.txt. A source build keeps debug symbols by default, so there's nothing extra to install.

Hi Alwin,

I can't reproduce this locally. Your script runs fine on 7.0.2 here, and nobody else has reported this crash. I've opened an issue for it: https://github.com/netxms/netxms/issues/3889

The stack shows a crash while a script DCI calls a method on an object (node, table, etc.). Every script DCI goes through that same path, though, so it doesn't prove this particular script is the one. The Ubuntu updates you listed are very unlikely to be related.

If you still have the core dump, please send a full backtrace and the list of installed NetXMS packages:

coredumpctl gdb netxmsd
(gdb) thread apply all bt full
(gdb) quit

dpkg -l | grep netxms

If you try the upgrade again, disable the script DCIs on 6.2.5 first. If 7.0.2 then stays up, we know it's script-related and can narrow it down from there.

Hi,

Yes, this works with a self-hosted UniFi OS Server, and SNMP isn't needed. The bridge doesn't use the node at all. The NetXMS server calls the UniFi Network API directly, and everything is configured as custom attributes on the wireless domain object. You can remove the VM node from the domain, because it does nothing in this mode.

1. Add Module = wlcbridge to netxmsd.conf and restart the server.

2. On the wireless domain, add these custom attributes:

wlcbridge        UniFi
unifi.base-url   https://<vm-ip>:<port>    # same URL you open the UniFi web UI on
unifi.login      <local admin account>
unifi.password   <password>                # plain text, or encrypted with: nxencpasswd <login> <password>
unifi.site       <site id>                 # only if not "default"

Instead of login/password you can set unifi.token to an API key created in the Network application. Login/password is the safer option, though. The account has to be a local one, not a UI.com cloud account.

3. Run a configuration poll on the wireless domain. The output should show "Reading access point list via bridge interface UniFi" and the number of access points reported. The access points are created under the domain.

Important detail: leave unifi.isAppliance unset. The name is misleading: true is for the old standalone Network Application (the one on port 8443), and UniFi OS Server needs it unset.

If the poll reports "Cannot retrieve access point list", set debug level 7 for tag wlcbridge.unifi. The server log will then show the exact URL being requested and the HTTP code that came back.

Это новый веб, он пока еще в разработке. В ближайшее время откроем репозиторий

@moba said in NetXMS 7.0 patch release 2:

Клиент не зависает. спасибо!

Вопрос а что за интерфейс такой симпотичный https://netxms.com/blog/how-companies-choose-a-monitoring-platform-and-where-netxms-clearly-wins/ в этой статье? Какая-то неизданная версия ещё?

I've removed the database password from your post. It was public for a while, so please change it anyway.

"Connection refused" means the core service still isn't running, so the client has nothing to connect to. The most likely reason is that the database schema wasn't upgraded. The installer runs the upgrade hidden, so if it fails you see nothing, and netxmsd refuses to start on an old schema. Please run this from an elevated command prompt:

cd C:\NetXMS\bin
nxdbmgr upgrade
netxmsd -D 4

The first command shows any upgrade error directly. The second runs the server in the foreground with debug output, so you can see where it stops. If either one fails, post the output here, or the end of C:\NetXMS\log\netxmsd.log.

Hi,

CACertificates is an agent parameter. netxmsd doesn't read it, the config dump just prints every key in the file. The server has no equivalent: notification drivers rely on libcurl defaults, and in the 7.0.0-7.0.2 Windows builds libcurl has no trusted roots at all.

It's fixed in 7.0.3 (https://github.com/netxms/netxms/issues/3869). libcurl is now built with Schannel as the default TLS backend, so it uses the Windows certificate store, same as your -CAstore org.openssl.winstore: test. You don't have to wait for the release: libcurl.dll from that build works as a drop-in replacement on 7.0.2, you can get it here: https://cloud.radensolutions.com/s/x3j62JbWPqsNtXJ

Stop the NetXMS server and agent services, keep a copy of the old C:\NetXMS\bin\libcurl.dll, copy the new one over it and start the services again. After that the cURL initialized (version: ...) line in the log should include Schannel, and you can remove VerifyPeer=false from the SMTP channel and CACertificates from both configs.

The next upgrade overwrites the DLL, which is fine because 7.0.3 ships the same one. The same file works for 64-bit Windows agents too: put it in the agent's bin directory instead of setting CACertificates.

Hi,

The crash in prunsrv.exe is the Reporting Server service ("NetXMS Reporting Server"), not the core. The NetXMS Core service is netxmsd.exe and gets its own service entry. The reporting server needs a 64-bit Java 11 or later, where 5.0 ran on Java 8. If you don't use reports, set that service to Disabled for now.

For the core itself, what is in DBDriver in C:\NetXMS\etc\netxmsd.conf? I suspect it's mysql.ddr. In 7.0 the separate MySQL driver was removed and MySQL is served by the MariaDB driver. The upgrade keeps your old config and leaves the old 5.0 mysql.ddr in place, and that driver can't be loaded by 7.0. The server stops at startup, and the database upgrade step in the installer fails silently for the same reason. If that's the case, change the line in netxmsd.conf to DBDriver = mariadb, then run:

cd C:\NetXMS\bin
nxdbmgr -c C:\NetXMS\etc\netxmsd.conf upgrade
net start netxmsd

A direct upgrade from 5.0.2 to 7.0.2 is fine. nxdbmgr goes through all the intermediate versions itself.

If your database isn't MySQL, or it still won't start, please attach the last 50-100 lines of C:\NetXMS\log\netxmsd.log.

Сообщение об успешном резолве в логе, скорее всего, вида "Node name resolved to ..." - это синхронизация имени ноды, а не резолв primary host name. Она работает в обратную сторону: имя объекта берётся из reverse DNS адресов интерфейсов или из SNMP sysName, поэтому срабатывает, даже если прямой записи для имени нет ни в одном DNS. Успешный резолв primary host name сразу заполняет primary IP, в логе это "IP address for node ... changed from ... to ...".

SNMP при этом на самом деле не опрашивается. Все запросы идут на primary IP, и без него сервер проверки пропускает: configuration poll пишет "SNMP polling is not possible", status poll проверку SNMP не делает, проверка доступности ноды для нод без IP отключена. "SNMP status: Responding" и статус Normal - это состояние на момент, когда IP ещё был. Фактически нода не мониторится.

С включённым Objects.Nodes.ClearIPAddressOnDNSFailure так же выпадут все ноды, чьё имя не резолвится с сервера NetXMS. Найти их можно по SYS_DNS_RESOLUTION_FAILED в event log или по Primary IP address "none". Вернуть - вписать в Primary host name резолвящееся имя или IP.

По интерфейсу: в 6.x и 7.0 тут ничего не менялось. Primary host name всегда был на странице Communication в Properties. В 4.4 (2023) на Overview блок Communications, где показываются primary host name и IP, вынесли из General - видимо, его вы и помнили.

Yes, this comes from a change in the Windows build. The libcurl bundled with the 7.0 agent uses OpenSSL for TLS and doesn't read the Windows certificate store, so unless a CA bundle is configured the agent has no trusted roots at all. Error 19 means your server sends the DigiCert root along with the chain and the agent doesn't trust it. A server that doesn't send the root fails with error 20 instead. The browser checks the same chain against the Windows store, which is why it looks valid there.

7.0.2, released today, doesn't change this.

Workaround is to give the agent a CA bundle. Mozilla's bundle from https://curl.se/docs/caextract.html includes the DigiCert roots:

# nxagentd.conf, top level - applies to netsvc and other https requests made by the agent
CACertificates = C:\NetXMS\etc\cacert.pem

Restart the agent after the change.

Can you please post the "cURL initialized (version: ...)" line from the agent log? It's logged on startup with DebugLevel 3 or higher, and it shows the TLS backend, which will confirm this.

Добрый день,

Возможно, причина в размере war-файла: nxmc-7.0.1.war весит около 53 МБ, а Tomcat Manager по умолчанию принимает upload не больше 50 МБ (52428800 байт). 6.2.6 (около 51 МБ) в этот лимит укладывался, 7.0.x уже нет. В таком случае Manager отклоняет файл ещё до деплоя и показывает пустую страницу, а в логе Tomcat (catalina.out или localhost.*.log) должно быть исключение о превышении размера - стоит проверить.

Если подтвердится, есть два варианта:

1. Скопировать war напрямую в каталог webapps. Tomcat задеплоит его сам, лимит Manager при этом не действует.
2. Если нужен именно Manager, увеличить лимиты в webapps/manager/WEB-INF/web.xml и перезапустить Tomcat:

   <multipart-config>
     <max-file-size>104857600</max-file-size>
     <max-request-size>104857600</max-request-size>
     <file-size-threshold>0</file-size-threshold>
   </multipart-config>
   

Hi,

This is a bug in the 7.0.1 Windows agent build, fixed in 7.0.2, which was released today. Upgrading the agent brings back the correct version, build and product name.

Here's what's actually happening: Windows reports itself as Windows 8 (6.2.9200) to any program that doesn't declare support for newer Windows versions in its application manifest. The 7.0 Windows build moved to a new toolchain, and the 7.0.1 agent shipped without that manifest. It's back in 7.0.2.

Note: on 7.0.1 the agent also skipped Windows Store apps in software inventory, because it thought it was running on Windows 8. If you have inventory enabled, the first configuration poll after the upgrade will show those packages as newly installed. That's expected, nothing was actually installed.

Причина в обоих thread dump'ах одна: deadlock в клиенте между обновлением списка алармов и таймером мигания. Возникает только при включённой опции "Blinking outstanding alarms".

Обход: отключить эту опцию в настройках клиента, раздел Alarms.

Тикет: https://github.com/netxms/netxms/issues/3863

Hi,

5 is not specifically a certificate error. For http/https checks the agent returns it for any libcurl failure except DNS and connection errors, so a timeout looks the same as a TLS problem. Service.Check and NetworkService.Status run the same code, so both returning 5 is one failure, not two.

There are no changes in the check code between 6.2.6 and 7.0.1. What did change is the Windows agent build - new toolchain and new bundled libraries, libcurl included. Is this a Windows agent, or deb/rpm? And which version did you upgrade from?

To see the real error, set DebugLevel = 6 in nxagentd.conf, restart the agent and post these two lines from the agent log:

cURL initialized (version: ...)
NetworkServiceStatus_HTTP(...): call to curl_easy_perform failed (NN: ...)

Quick test for certificate trust:

NetworkService.Status(https://your.site, verify-peer=false)

If this returns 0, then it's trust. As a workaround you can point the agent to a CA bundle in PEM format:

[netsvc]
CA = /path/to/ca-bundle.pem

Добрый день,

Primary host name находится в Properties ноды на странице Communication - это верхнее поле на вашем скрине Communication.png. В 7.0 оно не переносилось, в 6.2.4 оно было там же. Отдельного поля для primary IP нет: сервер получает адрес из primary host name, а это поле принимает и DNS имя, и IP адрес. Страница EtherNet/IP на primary IP не влияет, там задаётся только адрес для EtherNet/IP.

Чтобы вернуть IP на этой ноде, впишите в Primary host name адрес 192.168.x.x и нажмите Apply. Сервер принудительно запустит configuration poll и обновит primary IP.

Если нужно оставить DNS имя, оно должно резолвиться на сервере NetXMS. Тогда IP вернётся сам на следующем configuration poll, а повторный ввод того же имени ничего не изменит.

Primary IP "none" при заполненном имени похоже на сбой DNS резолва при включённом Objects.Nodes.ClearIPAddressOnDNSFailure (по умолчанию выключен), этот параметр включён на сервере? Если да, в event log будет SYS_DNS_RESOLUTION_FAILED, и другие ноды, заданные по DNS имени, могут потерять IP таким же образом.

Hi Andrea,

Yes, on both. For monitoring-only use, the setting that matters most is the access level you give the server, covered at the end.

Linux. The packaged agent runs as root. Create a dedicated user and either change the systemd unit with "systemctl edit netxms-agent":

[Service]
User=nxagent
Group=nxagent

or let the agent drop privileges at startup, with this in nxagentd.conf:

UserId = nxagent
GroupId = nxagent

Use the systemd route if the user needs extra groups, such as adm for reading logs, because the agent applies only the primary group. Either way, this user needs write access to the agent's log file and data directory. CPU, memory, disk, filesystem, network and process metrics keep working. You lose SMBIOS hardware data (serial numbers, memory modules, etc.) and ICMP from the ping subagent, and the agent can only read files and logs that this user can read.

Windows. The installer always creates the service under LocalSystem, but the agent works under a limited account if you change it in the service properties afterwards. The account needs rights on the agent's etc, log and var folders. Anything that needs administrator rights stops working, such as System.Restart, and per-process metrics such as CPU time and I/O will quietly return 0 for processes of other users.

Both. Give the server read-only access and use an outbound tunnel, so the agent has no open port:

Servers = netxms.example.com             # read-only, remove the MasterServers line
ServerConnection = netxms.example.com    # outbound tunnel to the server
ListenPort = 0                           # no inbound listener

With this the server can read metrics, but it can't run actions, edit the agent's config, deploy policies or write files. The tunnel has to be bound to the node on the server side.

Note: the installer writes MasterServers, which is full access, so that line has to be removed. All config changes are then made locally on the host.