nxAgent, crash, libcrypto (4.x.x-4.5.6,linux,x86_64,package)
-
Hello,
Problem description:
- tuneled agent crashes after 0s..45min after start (repeatedly)
Environment description:
- LXC container running (bookworm) on Proxmox running (bookworm) all up2date @04.16.2024
- kernel 6.5.11-7-pve
- DEB Package: netxms-agent, Version: 4.5.6-1
-------------------------------- :~# gdb /usr/bin/nxagentd GNU gdb (Debian 13.1-3) 13.1 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/nxagentd... Reading symbols from /usr/lib/debug/.build-id/95/2e3f9104195351fe4deab7db75a2be62a439f2.debug... (gdb) set args -f -D 4 ---------------------------------------------------------------- (gdb) run Starting program: /usr/bin/nxagentd -f -D 4 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Error reading additional configuration files from "/etc/nxagentd.conf.d" 2024.04.16 07:20:04.009 *I* [logger ] Log rotation policy set to 2 (size=16777216, count=4) 2024.04.16 07:20:04.010 *I* [startup ] Core agent version 4.5.6 (build tag 4.5-213-g58d2aa16a8) 2024.04.16 07:20:04.010 *I* [config ] Additional configuration files was loaded from /etc/nxagentd.conf.d 2024.04.16 07:20:04.010 *I* [logger ] Debug level set to 4 2024.04.16 07:20:04.010 *I* [config ] Main configuration file: /etc/nxagentd.conf 2024.04.16 07:20:04.010 *I* [config ] Configuration tree: 2024.04.16 07:20:04.010 *I* [config ] config 2024.04.16 07:20:04.010 *I* [config ] +- CORE 2024.04.16 07:20:04.010 *I* [config ] +- ServerConnection 2024.04.16 07:20:04.010 *I* [config ] value: 172.16.xx.xx 2024.04.16 07:20:04.010 *I* [startup ] Data directory: /var/lib/netxms 2024.04.16 07:20:04.010 *I* [startup ] File store: /tmp 2024.04.16 07:20:04.010 *D* [startup ] Effective user ID 0 2024.04.16 07:20:04.010 *D* [startup ] Effective group ID 0 2024.04.16 07:20:04.010 *D* [startup ] Configuration policy directory: /var/lib/netxms/config_ap/ 2024.04.16 07:20:04.010 *D* [startup ] Log parser policy directory: /var/lib/netxms/logparser_ap/ 2024.04.16 07:20:04.010 *D* [startup ] User agent policy directory: /var/lib/netxms/useragent_ap/ 2024.04.16 07:20:04.011 *D* [startup ] Certificate directory: /var/lib/netxms/certificates/ 2024.04.16 07:20:04.011 *D* [startup ] CRL directory: /var/lib/netxms/crl/ 2024.04.16 07:20:04.014 *D* [crypto ] Validating ciphers 2024.04.16 07:20:04.014 *D* [crypto ] AES-256 enabled 2024.04.16 07:20:04.014 *D* [crypto ] Blowfish-256 disabled (validation failed) 2024.04.16 07:20:04.014 *D* [crypto ] IDEA disabled (config) 2024.04.16 07:20:04.014 *D* [crypto ] 3DES enabled 2024.04.16 07:20:04.014 *D* [crypto ] AES-128 enabled 2024.04.16 07:20:04.014 *D* [crypto ] Blowfish-128 disabled (validation failed) 2024.04.16 07:20:04.014 *I* [crypto ] Crypto library initialized (OpenSSL 3.0.11 19 Sep 2023) [New Thread 0x7ffff649b6c0 (LWP 33983)] [New Thread 0x7ffff5bff6c0 (LWP 33984)] [New Thread 0x7ffff5afe6c0 (LWP 33985)] 2024.04.16 07:20:04.016 *D* [threads.pool ] Thread pool TIMER initialized (min=2, max=16) 2024.04.16 07:20:04.016 *D* [startup ] Subagent API initialized 2024.04.16 07:20:04.035 *D* [db.drv.sqlite ] SQLite version 3.40.1 2024.04.16 07:20:04.035 *I* [db.drv ] Database driver "sqlite.ddr" loaded and initialized successfully 2024.04.16 07:20:04.036 *D* [db.conn ] New DB connection opened: handle=0x7ffff61641c0 2024.04.16 07:20:04.038 *D* [db.query ] DB Library: global long running query threshold set to 250 2024.04.16 07:20:04.045 *I* [db.agent ] Local database opened successfully 2024.04.16 07:20:04.046 *I* [startup ] Agent ID is 6334d94b-6ab3-46ba-8515-891f843f5907 2024.04.16 07:20:04.047 *I* [startup ] Local host name is "d001-spooler01.local" 2024.04.16 07:20:04.047 *I* [startup ] Using system name "d001-spooler01" 2024.04.16 07:20:04.047 *D* [comm ] Maximum number of sessions set to 32 [New Thread 0x7ffff645a6c0 (LWP 33986)] [New Thread 0x7ffff561e6c0 (LWP 33987)] [New Thread 0x7ffff551d6c0 (LWP 33988)] [New Thread 0x7ffff541c6c0 (LWP 33989)] [New Thread 0x7ffff531b6c0 (LWP 33990)] [New Thread 0x7ffff521a6c0 (LWP 33991)] [New Thread 0x7ffff51196c0 (LWP 33992)] [New Thread 0x7ffff50186c0 (LWP 33993)] [New Thread 0x7ffff4f176c0 (LWP 33994)] 2024.04.16 07:20:04.049 *D* [threads.pool ] Thread pool COMM initialized (min=8, max=512) [New Thread 0x7ffff4e166c0 (LWP 33995)] [New Thread 0x7ffff4dd56c0 (LWP 33996)] [New Thread 0x7ffff4cd46c0 (LWP 33997)] [New Thread 0x7ffff4bd36c0 (LWP 33998)] [New Thread 0x7ffff4ad26c0 (LWP 33999)] 2024.04.16 07:20:04.049 *D* [threads.pool ] Thread pool WEBSVC initialized (min=4, max=64) 2024.04.16 07:20:04.050 *D* [tunnel ] Added server tunnel 172.16.xx.xx 2024.04.16 07:20:04.050 *I* [startup ] Arbitrary command execution disabled 2024.04.16 07:20:04.064 *D* [smbios ] BIOS read failed [New Thread 0x7ffff49256c0 (LWP 34001)] 2024.04.16 07:20:05.084 *D* [linux ] Using /sys/block to distinguish devices from partitions 2024.04.16 07:20:05.084 *D* [linux ] CPU usage collector thread started [New Thread 0x7ffff48246c0 (LWP 34002)] [New Thread 0x7ffff47236c0 (LWP 34003)] 2024.04.16 07:20:05.089 *I* [subagents ] Subagent "Linux" (/usr/lib/x86_64-linux-gnu/netxms/linux.nsm) loaded successfully (version 4.5.6) 2024.04.16 07:20:05.089 *I* [startup ] System hardware ID is unknown 2024.04.16 07:20:05.090 *I* [linux ] Unable to parse /proc/drbd, DRBD data collector will not start 2024.04.16 07:20:05.090 *D* [linux ] ParseIoStat(): new device added (name=loop0 isRealDevice=1) 2024.04.16 07:20:05.090 *D* [linux ] ParseIoStat(): new device added (name=loop1 isRealDevice=1) 2024.04.16 07:20:05.090 *D* [linux ] ParseIoStat(): new device added (name=loop2 isRealDevice=1) 2024.04.16 07:20:05.091 *D* [linux ] ParseIoStat(): new device added (name=loop3 isRealDevice=1) 2024.04.16 07:20:05.091 *D* [linux ] ParseIoStat(): new device added (name=loop4 isRealDevice=1) 2024.04.16 07:20:05.091 *D* [linux ] ParseIoStat(): new device added (name=loop5 isRealDevice=1) 2024.04.16 07:20:05.091 *D* [linux ] ParseIoStat(): new device added (name=loop6 isRealDevice=1) 2024.04.16 07:20:05.092 *D* [linux ] ParseIoStat(): new device added (name=loop7 isRealDevice=1) 2024.04.16 07:20:05.092 *D* [linux ] ParseIoStat(): new device added (name=nvme0n1 isRealDevice=1) 2024.04.16 07:20:05.092 *D* [linux ] ParseIoStat(): new device added (name=nvme0n1p1 isRealDevice=0) 2024.04.16 07:20:05.092 *D* [linux ] ParseIoStat(): new device added (name=nvme0n1p2 isRealDevice=0) 2024.04.16 07:20:05.092 *D* [linux ] ParseIoStat(): new device added (name=nvme0n1p3 isRealDevice=0) 2024.04.16 07:20:05.093 *D* [linux ] ParseIoStat(): new device added (name=nvme0n1p4 isRealDevice=0) 2024.04.16 07:20:05.093 *D* [linux ] ParseIoStat(): new device added (name=nvme0n1p128 isRealDevice=0) 2024.04.16 07:20:05.093 *D* [linux ] ParseIoStat(): new device added (name=zd0 isRealDevice=1) 2024.04.16 07:20:05.093 *D* [linux ] ParseIoStat(): new device added (name=zd16 isRealDevice=1) 2024.04.16 07:20:05.093 *D* [linux ] ParseIoStat(): new device added (name=zd16p1 isRealDevice=0) 2024.04.16 07:20:05.094 *D* [linux ] ParseIoStat(): new device added (name=zd16p14 isRealDevice=0) 2024.04.16 07:20:05.094 *D* [linux ] ParseIoStat(): new device added (name=zd16p15 isRealDevice=0) 2024.04.16 07:20:05.094 *D* [linux ] ParseIoStat(): new device added (name=zd32 isRealDevice=1) 2024.04.16 07:20:05.094 *D* [linux ] ParseIoStat(): new device added (name=zd48 isRealDevice=1) 2024.04.16 07:20:05.095 *D* [linux ] ParseIoStat(): new device added (name=zd64 isRealDevice=1) 2024.04.16 07:20:05.095 *D* [linux ] ParseIoStat(): new device added (name=zd80 isRealDevice=1) 2024.04.16 07:20:05.095 *D* [linux ] ParseIoStat(): new device added (name=zd96 isRealDevice=1) 2024.04.16 07:20:05.095 *D* [linux ] ParseIoStat(): new device added (name=zd112 isRealDevice=1) 2024.04.16 07:20:05.096 *D* [linux ] ParseIoStat(): new device added (name=zd112p1 isRealDevice=0) 2024.04.16 07:20:05.096 *D* [linux ] ParseIoStat(): new device added (name=zd112p14 isRealDevice=0) 2024.04.16 07:20:05.096 *D* [linux ] ParseIoStat(): new device added (name=zd112p15 isRealDevice=0) 2024.04.16 07:20:05.096 *D* [linux ] ParseIoStat(): new device added (name=zd128 isRealDevice=1) 2024.04.16 07:20:05.096 *D* [linux ] ParseIoStat(): new device added (name=zd144 isRealDevice=1) 2024.04.16 07:20:05.097 *D* [linux ] ParseIoStat(): new device added (name=zd144p1 isRealDevice=0) 2024.04.16 07:20:05.097 *D* [linux ] ParseIoStat(): new device added (name=zd144p14 isRealDevice=0) 2024.04.16 07:20:05.097 *D* [linux ] ParseIoStat(): new device added (name=zd144p15 isRealDevice=0) 2024.04.16 07:20:05.097 *D* [linux ] ParseIoStat(): new device added (name=zd160 isRealDevice=1) 2024.04.16 07:20:05.097 *D* [linux ] ParseIoStat(): new device added (name=zd160p1 isRealDevice=0) 2024.04.16 07:20:05.097 *D* [linux ] ParseIoStat(): new device added (name=zd160p2 isRealDevice=0) 2024.04.16 07:20:05.098 *D* [linux ] ParseIoStat(): new device added (name=zd160p3 isRealDevice=0) 2024.04.16 07:20:05.098 *D* [linux ] ParseIoStat(): new device added (name=zd160p4 isRealDevice=0) 2024.04.16 07:20:05.098 *D* [linux ] ParseIoStat(): new device added (name=zd160p5 isRealDevice=0) [Thread 0x7ffff47236c0 (LWP 34003) exited] 2024.04.16 07:20:05.099 *D* [linux ] ParseIoStat(): new device added (name=zd160p6 isRealDevice=0) 2024.04.16 07:20:05.099 *D* [linux ] ParseIoStat(): new device added (name=zd160p7 isRealDevice=0) 2024.04.16 07:20:05.099 *D* [linux ] ParseIoStat(): new device added (name=zd160p8 isRealDevice=0) [New Thread 0x7ffff39ff6c0 (LWP 34004)] [New Thread 0x7ffff39be6c0 (LWP 34005)] 2024.04.16 07:20:06.091 *D* [threads.pool ] Thread pool PROCEXEC initialized (min=1, max=16) 2024.04.16 07:20:06.091 *D* [notifications ] StartNotificationProcessor: Loaded 1 servers [New Thread 0x7ffff38bd6c0 (LWP 34006)] 2024.04.16 07:20:06.092 *D* [notifications ] Notification processor started 2024.04.16 07:20:06.092 *D* [notifications ] Notification housekeeper started 2024.04.16 07:20:06.092 *D* [notifications ] Notification housekeeper execution completed [New Thread 0x7ffff37bc6c0 (LWP 34007)] [New Thread 0x7ffff377b6c0 (LWP 34008)] [New Thread 0x7ffff31ff6c0 (LWP 34009)] [New Thread 0x7ffff30fe6c0 (LWP 34010)] [New Thread 0x7ffff2ffd6c0 (LWP 34011)] 2024.04.16 07:20:06.093 *D* [threads.pool ] Thread pool DATACOLL initialized (min=4, max=64) [New Thread 0x7ffff2efc6c0 (LWP 34012)] 2024.04.16 07:20:06.094 *D* [dc ] Data collection scheduler thread started [New Thread 0x7ffff2dfb6c0 (LWP 34013)] [New Thread 0x7ffff27ff6c0 (LWP 34014)] [New Thread 0x7ffff26fe6c0 (LWP 34015)] 2024.04.16 07:20:06.094 *D* [dc ] Data sender thread started [New Thread 0x7ffff25fd6c0 (LWP 34016)] [New Thread 0x7ffff1dff6c0 (LWP 34017)] [New Thread 0x7ffff1cfe6c0 (LWP 34018)] 2024.04.16 07:20:06.095 *D* [ ] Session agent connector disabled 2024.04.16 07:20:06.096 *D* [comm.listener ] SocketListener/ProxyHeartbeat: Trying to bind on 0.0.0.0:4700/udp 2024.04.16 07:20:06.096 *D* [comm.listener ] SocketListener/ProxyHeartbeat: Trying to bind on [::]:4700/udp 2024.04.16 07:20:06.096 *D* [comm ] Trying to bind on 0.0.0.0:4700 2024.04.16 07:20:06.096 *D* [ ] Trying to bind on [::]:4700 2024.04.16 07:20:06.096 *I* [comm ] Listening on socket 0.0.0.0:4700 2024.04.16 07:20:06.096 *I* [comm ] Listening on socket :::4700 2024.04.16 07:20:06.096 *D* [dc ] Database writer thread started 2024.04.16 07:20:06.096 *D* [dc ] Data reconciliation thread started (block size 1024, timeout 60000 ms) [New Thread 0x7ffff19fd6c0 (LWP 34019)] [New Thread 0x7ffff18fc6c0 (LWP 34020)] 2024.04.16 07:20:06.107 *D* [ ] NamedPipeListener(nxagentd.push): waiting for connection 2024.04.16 07:20:06.107 *D* [ ] NamedPipeListener(nxagentd.events): waiting for connection 2024.04.16 07:20:06.107 *I* [comm ] Control connector is disabled [New Thread 0x7ffff17fb6c0 (LWP 34021)] 2024.04.16 07:20:06.108 *D* [tunnel ] Tunnel manager started 2024.04.16 07:20:06.114 *D* [tunnel ] 172.16.xx.xx: Certificate and private key loaded 2024.04.16 07:20:06.126 *D* [tunnel ] 172.16.xx.xx: Server certificate subject is CN=d001-mgmt01 2024.04.16 07:20:06.127 *D* [tunnel ] 172.16.xx.xx: Server certificate issuer is CN=d001 2024.04.16 07:20:06.127 *D* [tunnel ] 172.16.xx.xx: Server certificate verification is disabled 2024.04.16 07:20:06.127 *D* [tunnel ] 172.16.xx.xx: Server certificate pinning is disabled [New Thread 0x7ffff16fa6c0 (LWP 34022)] 2024.04.16 07:20:06.171 *I* [ ] Tunnel with 172.16.xx.xx established 2024.04.16 07:20:06.171 *D* [tunnel ] 172.16.xx.xx: Tunnel is active 2024.04.16 07:20:07.109 *I* [filemon ] Path list for file monitor is empty 2024.04.16 07:20:07.109 *I* [ ] NetXMS Agent started Agent running. Press Ctrl+C to shutdown. 2024.04.16 07:20:13.988 *D* [comm.cs.2 ] Session registered (control=false, master=false) [New Thread 0x7ffff15f96c0 (LWP 34023)] 2024.04.16 07:20:13.989 *D* [comm.cs.2 ] Using protocol version 5 2024.04.16 07:20:13.994 *D* [comm.cs.2 ] Server capabilities: IPv6: yes; bulk reconciliation: yes; compression: yes 2024.04.16 07:20:13.994 *D* [comm.cs.2 ] Server ID set to 319612E4EC7036B6 2024.04.16 07:20:13.995 *D* [comm.cs.2 ] Server ID set to 319612E4EC7036B6 2024.04.16 07:20:14.005 *D* [dc ] 0 SNMP targets received from server 319612E4EC7036B6 2024.04.16 07:20:14.005 *D* [dc ] 0 proxies received from server 319612E4EC7036B6 2024.04.16 07:20:14.005 *D* [dc ] 0 data collection elements received from server 319612E4EC7036B6 (extended data: YES) 2024.04.16 07:20:14.005 *D* [dc ] Data collection for server 319612E4EC7036B6 reconfigured 2024.04.16 07:20:14.009 *D* [notifications ] RegisterSessionForNotifications: starting background sync for server 319612E4EC7036B6 2024.04.16 07:20:14.009 *D* [notifications ] Notification synchronization started 2024.04.16 07:20:14.009 *D* [notifications ] Vacuum local database 2024.04.16 07:20:14.016 *D* [notifications ] Notification synchronization finished 2024.04.16 07:48:09.800 *D* [comm.cs.3 ] Session registered (control=false, master=false) [New Thread 0x7ffff11f86c0 (LWP 34029)] 2024.04.16 07:48:09.808 *D* [comm.cs.3 ] Using protocol version 5 Thread 42 "nxagentd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff11f86c0 (LWP 34029)] 0x00007ffff7a330a2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3 -------------------------------------------------------------------------------------------------- (gdb) bt #0 0x00007ffff7a330a2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3 #1 0x00007ffff7a315af in OPENSSL_LH_doall_arg () from /lib/x86_64-linux-gnu/libcrypto.so.3 #2 0x00007ffff7a332ab in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3 #3 0x00007ffff7a24af6 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3 #4 0x00007ffff7ec8fe0 in RSAPublicEncrypt (outlen=4096, padding=4, rsa=0x7ffff321e0e0, out=0x7ffff11ed990 "0\202\001\n\002\202\001\001", inlen=32, in=0x7ffff320b0a0) at ../../include/nxcrypto.h:211 #5 SetupEncryptionContext (msg=msg@entry=0x7ffff328b000, ppCtx=ppCtx@entry=0x7ffff11eea20, ppResponse=ppResponse@entry=0x7ffff11eea30, privateKey=privateKey@entry=0x0, nxcpVersion=<optimized out>) at ./src/libnetxms/crypto.cpp:474 #6 0x00005555555a8dfe in CommSession::readThread (this=0x7ffff1e61310) at ./src/agent/core/session.cpp:340 #7 0x00005555555aa9a2 in ThreadCreate_SharedPtr_Wrapper_0<CommSession> (arg=0x7ffff1e22080) at ../../../include/nms_threads.h:795 #8 0x00007ffff7357134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #9 0x00007ffff73d77dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 (gdb) -------------------------------------------------thank you,
Br, Aleš
-
Agent crashes within OpenSSL library (specifically within EVP_PKEY_CTX_new) and I do not see any obvious problems with calling code. This part was not changed for quite some time. Was it working on older versions or this is new setup? What exact version of openssl package you are using?
Best regards,
Victor -
Hello,
Q: Was it working on older versions or this is new setup?
A: Unknown. This is a new/clean setup. Specific is tunneled agent setup. And agent uptime monitoring (which pointed out issue).Q: What exact version of openssl package you are using?
A: Package: libssl3, Version: 3.0.11-1~deb12u2, APT-Sources: http://deb.debian.org/debian bookworm/main amd64 PackagesBr, Ales
-
We will try to reproduce that, but it likely will take some time.
I registered it a s a bug: https://track.radensolutions.com/issue/NX-2527
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login