Client timeouts and Alarms not processing in a timely manner
-
I am currently using NetXMS 6.1 and I've noticed that I am not always getting alarms when nodes become unreachable at least not immediately. I have also observed timeouts when using the client for simple activities like creating a new node that I have previously not encountered before.
I took a look at the log for my server and noticed these entries -
2026.04.17 09:16:50.436 *I* [watchdog ] Thread "Recurrent scheduler" r eturned to running state
2026.04.17 09:17:48.752 *I* [watchdog ] Thread "Syncer Thread" returne d to running state
2026.04.17 09:19:36.335 *E* [watchdog ] Thread "Syncer Thread" does no t respond to watchdog thread
2026.04.17 09:21:56.337 *E* [watchdog ] Thread "Recurrent scheduler" d oes not respond to watchdog thread
2026.04.17 09:24:24.322 *I* [watchdog ] Thread "Recurrent scheduler" r eturned to running state
2026.04.17 09:26:36.342 *E* [watchdog ] Thread "Recurrent scheduler" d oes not respond to watchdog thread
2026.04.17 09:32:38.323 *I* [watchdog ] Thread "Recurrent scheduler" r eturned to running state
2026.04.17 09:33:06.359 *I* [watchdog ] Thread "Syncer Thread" returne d to running state
2026.04.17 09:34:56.347 *E* [watchdog ] Thread "Syncer Thread" does no t respond to watchdog thread
2026.04.17 09:35:56.348 *E* [watchdog ] Thread "Recurrent scheduler" d oes not respond to watchdog threadI'd appreciate any suggestions on how to fix this.
-
Pls show output of the following from debug console:
sh st
sh th
sh quWhat are the overall specs of the server? What are CPU and memory for a few days?
-
NetXMS Server Remote Console V6.1.0 Ready
sh st
Objects............: 25145
Nodes...........: 3177
Interfaces......: 19762
Access points...: 651
Sensors.........: 0
Collectible DCIs...: 44734
Active alarms......: 3924
Uptime.............: 2 days, 19:42:54sh th
MAIN
Threads.............. 8 (8/600)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 1%
Active requests...... 0
Scheduled requests... 3
Total requests....... 4624
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0AGENT
Threads.............. 32 (32/256)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 12%
Active requests...... 0
Scheduled requests... 0
Total requests....... 9455
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0POLLERS
Threads.............. 484 (10/600)
Load average......... 195.17 240.33 250.33
Current load......... 58%
Usage................ 80%
Active requests...... 284
Scheduled requests... 0
Total requests....... 5389134
Thread starts........ 749
Thread stops......... 275
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0FILE-TRANSFER
Threads.............. 2 (2/16)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 12%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0DATACOLL
Threads.............. 10 (10/600)
Load average......... 2.51 1.65 1.38
Current load......... 0%
Usage................ 1%
Active requests...... 0
Scheduled requests... 0
Total requests....... 17275228
Thread starts........ 66
Thread stops......... 66
Wait time EMA........ 29 ms
Wait time SMA........ 28 ms
Wait time SD......... 37 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0THREVT
Threads.............. 2 (2/4)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 50%
Active requests...... 0
Scheduled requests... 1
Total requests....... 75
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0CLIENT
Threads.............. 16 (16/2048)
Load average......... 0.03 0.01 0.00
Current load......... 12%
Usage................ 0%
Active requests...... 2
Scheduled requests... 0
Total requests....... 25173
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0DISCOVERY
Threads.............. 8 (8/64)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 12%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0SCHEDULER
Threads.............. 6 (1/64)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 9%
Active requests...... 0
Scheduled requests... 1
Total requests....... 1026
Thread starts........ 5
Thread stops......... 0
Wait time EMA........ 643 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0MOBILE
Threads.............. 4 (4/256)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 1%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0PACKAGE-MANAGER
Threads.............. 2 (2/25)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 8%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0sh qu
Data collector : 0
DCI cache loader : 0
Template updater : 0
Database writer : 0
Database writer (alarms) : 0
Database writer (IData) : 0
Database writer (raw DCI values) : 1934
Event processor : 0
Event log writer : 0
Poller : 0
Node discovery poller : 0
SNMP trap processor : 0
SNMP trap writer : 0
Syslog processor : 0
Syslog writer : 0
Scheduler : 0
Windows event processor : 0
Windows event writer : 0Ubuntu VM on 22.04 Intel Xeon Skylake Processor 16 Cores, 32 Gbs RAM, 500 GB Virtual drive running on SSD
Using 15 GBs of RAM currently and CPU usage stays under 10%
-
Possible case could be a locking of some server thread. You can capture server threads when you see delays in working with the server from the GUI, here's the script: https://github.com/netxms/netxms/blob/master/tools/capture_netxmsd_threads.sh
Script should be executed 3 times with 20-30 second interval. It needs gdb and netxms-dbg packages installed on the system.Also to check - on netxms server node there should be Server: Number of Processed Events (based on internal Server.TotalEventsProcessed metric) - just to verify that there we no sudden spikes in event quantity.
In overall there's quite a few metrics there, e.g. Server QueueSize ones, it's worth checking them for any big spikes.
If you don't have Server.ImportConfigurationOnStartup set to Always, you might not have the most recent template with above mentioned DCIs, you may want to import them from /usr/share/netxms/templates/netxms_server.xml
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login