Do you have any non-Ethernet interfaces there with MAC address longer than 6 bytes?

not that i can see

I have tried running with debug 6 and see if i can see anything in the logs but no errors there but i see a trend.

Line before "Log file opened" is always "NetworkDeviceDriver::getInterfaces"

example:

2023.09.20 09:45:19.570 *D* [ndd.common         ] NetworkDeviceDriver::getInterfaces(0x7f0dae84c740): completed, ifList=0x7f0db5c81300

2023.09.20 09:45:21.466 *I* [logger             ] Log file opened (rotation policy 2, max size 16777216)
2023.09.20 09:45:21.466 *I* [startup            ] Starting NetXMS server version 4.4.2 build tag 4.4-568-g3a9a8aa557

if i search for  0x7f0dae84c740 in the log i found witch node it was

2023.09.20 09:45:18.722 *D* [node.iface         ] Node::getInterfaceList(node=TPFIBSW02 [10402]): calling driver (useIfXTable=true)
2023.09.20 09:45:18.722 *D* [ndd.common         ] NetworkDeviceDriver::getInterfaces(0x7f0dae84c740,true)

I started server and quickly unmanaged  TPFIBSW02 and now server has been running for a while without crashing.
Since nothing is output to the log even with level 6 i guess this needs core dump to get why it crashes on this node on poll for interfaces but there i am at a loss.

Noticed that there is another thread that also restarted server on "SQL query failed" is this intended behavior now or bug?

exec sp_prepexec @p1 output,N'@P1 varchar(15),@P2 varchar(15),@P3 int,@P4 int,@P5 int,@P6 varchar(7),@P7 int,@P8 varchar(1),@P9 varchar(567)

@P9 varchar(567), P9 is snmp_oid and it is max 255

this is has been strange i removed it and will see if the issue is gone.

@marcush said in 4.4.2 Crash:

Any way to figure out what creates this query?

It's executed by object syncer thread, which saves node changes back into the database.

From the error message it's unclear which field is not accepted by the SQL server, this link might help with tracing it: https://stackoverflow.com/a/62905763

I though all the =? values was for log obfuscation but even the trace on the SQL server only shows =? i guess that would cause this issue since a lot of the columns are int and it tries to update ? into that.

Any idea on what could generate this type of node update?

@marcush said in 4.4.2 Crash:

I have a core dump but it gives a lot of reference error and shows no stack trace.

Have you installed netxms-dbg package? It contains all debug symbols for the product.

I have not i will have look at this on the test VM

Any way to figure out what creates this query?

It's executed by object syncer thread, which saves node changes back into the database.

From the error message it's unclear which field is not accepted by the SQL server, this link might help with tracing it: https://stackoverflow.com/a/62905763

I have a core dump but it gives a lot of reference error and shows no stack trace.

Have you installed netxms-dbg package? It contains all debug symbols for the product.

@marcush said in 4.4.2 Crash:

 Running server on docker making core dump a challange.

it's rather straighforward if you can control host's core pattern

sysctl -w kernel.core_pattern='/core/core.%e.%p.%t'
docker volume create core_vol
docker run --ulimit core=-1 --mount source=core_vol,target=/core container
sysctl -w kernel.core_pattern=core # reset it back to the default

I have a core dump but it gives a lot of reference error and shows no stack trace.

 Running server on docker making core dump a challange.

it's rather straighforward if you can control host's core pattern

sysctl -w kernel.core_pattern='/core/core.%e.%p.%t'
docker volume create core_vol
docker run --ulimit core=-1 --mount source=core_vol,target=/core container
sysctl -w kernel.core_pattern=core # reset it back to the default

On the issue i reverted back to 4.2.395 making at least server stable and this strange query is outputed to the server console and only this i have nothing else outputed to server console. Any way to figure out what creates this query?

Why does netxms still depend on libssl1.1 and not libssl3?

Because Debian 11 ships with OpenSSL 1.1:

root@da539131bae5:/# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye
root@da539131bae5:/# apt-cache search libssl
libssl-ocaml - OCaml bindings for OpenSSL (runtime)
libssl-ocaml-dev - OCaml bindings for OpenSSL
libssl-dev - Secure Sockets Layer toolkit - development files
libssl-doc - Secure Sockets Layer toolkit - development documentation
libssl1.1 - Secure Sockets Layer toolkit - shared libraries
libssl-utils-clojure - library for SSL certificate management on the JVM
root@da539131bae5:/# apt-cache show libssl-dev|grep Version
Version: 1.1.1n-0+deb11u5
Version: 1.1.1n-0+deb11u4

There are no reference to libssl1 in the official packages for debian 12:

root@d483efbfc136:~# ldd /usr/bin/netxmsd | grep ssl
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007ffb9ae88000)
root@d483efbfc136:~# dpkg -l netxms-server
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version      Architecture Description
+++-===================-============-============-=================================
ii  netxms-server:amd64 4.4.2-1      amd64        meta package

I was about to install server on a clean temp VM and test it without docker and then i noticed an issue.

Why does netxms still depend on libssl1.1 and not libssl3?