NetXMS V5.2.3 - Data loss and requests not loading
-
Hi,
We are investigating an issue where objects disappeared after a NetXMS crash and we are seeing unusual behaviour with the database connection pool and parts of the UI. We would appreciate guidance on possible causes.
Environment
NetXMS: v5.2.3
Database: MariaDB 10.11
NetXMS server and DB run on separate VMs with separate disks
NetXMS server specs:
16 CPU cores
64 GB RAMSystem size:
Objects: 23022
Nodes: 1129
Interfaces: 21081
Access Points: 0
Sensors: 0
Collectible DCIs: 14890MaxTransactionSize = 1000
Incident
The NetXMS server crashed after the root partition on the NetXMS server VM became full. The database is hosted on a different VM with its own storage and did not run out of space.When bringing NetXMS back online:
Some objects that had been created several days earlier had disappeared.
Other newer objects and data were still present (inc alarms for dci that had disappered), suggesting that only some metadata was lost or never committed.Shutdown behaviour
When attempting to restart the NetXMS service:
The server did not shut down gracefully.
It had to be killed manually.Database connection pool observation
During investigation we ran show dbcp and noticed that for extended periods (10–30 seconds) it reports:
0 database connections in useThis occurs even though the system is actively monitoring ~23k objects and ~14.8k collectible DCIs.
show dbstat
SQL query counters:
Total .......... 1100783
SELECT ......... 306641
Non-SELECT ..... 794106
Long running ... 0
Failed ......... 0
Background writer requests:
DCI data ....... 246179
DCI raw data ... 246179
Others ......... 234Additional anomalies
We are also seeing unusual behaviour in both the web client and the desktop client:
Some log views fail to load or take an extremely long time to load pages or do not load at all.
Pagination for logs sometimes does not work or stalls.These behaviours appear in both the web UI and the native NetXMS client.
This makes us suspect there may be an issue with database access, internal queues, or blocked threads.Questions
Could a full root partition on the NetXMS server prevent metadata commits to the database even if the DB is on a separate system?
Is it possible that object metadata was only held in server memory and never committed, resulting in the objects disappearing after the crash?
What could cause the DB connection pool to show 0 connections in use for long periods on a system of this size?
Could this indicate blocked writer threads, transaction batching issues, or internal locks?
Could the log loading and pagination issues in both clients be related to the same underlying database or thread issue?Any guidance on what diagnostics we should run (thread state, DB writer queues, etc.) would be greatly appreciated.
Thanks
Darren -
Additional info that I missed from the main post
show thread
MAIN
Threads.............. 8 (8/512)
Load average......... 0.00 0.01 0.00
Current load......... 0%
Usage................ 1%
Active requests...... 0
Scheduled requests... 1
Total requests....... 104912
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0AGENT
Threads.............. 32 (32/256)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 12%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0POLLERS
Threads.............. 80 (10/500)
Load average......... 15.51 28.01 35.31
Current load......... 2%
Usage................ 16%
Active requests...... 2
Scheduled requests... 0
Total requests....... 367629
Thread starts........ 427
Thread stops......... 357
Wait time EMA........ 274 ms
Wait time SMA........ 152 ms
Wait time SD......... 272 ms
Queue size EMA....... 7
Queue size SMA....... 0
Queue size SD........ 0FILE-TRANSFER
Threads.............. 2 (2/16)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 12%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0NPE
Threads.............. 1 (1/1024)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 0%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0DATACOLL
Threads.............. 17 (10/300)
Load average......... 0.00 0.07 0.42
Current load......... 0%
Usage................ 5%
Active requests...... 0
Scheduled requests... 0
Total requests....... 275896
Thread starts........ 17
Thread stops......... 10
Wait time EMA........ 80 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0CLIENT
Threads.............. 16 (16/2048)
Load average......... 0.00 0.00 0.00
Current load......... 12%
Usage................ 0%
Active requests...... 2
Scheduled requests... 0
Total requests....... 84567
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0DISCOVERY
Threads.............. 8 (8/64)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 12%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0SCHEDULER
Threads.............. 1 (1/64)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 1%
Active requests...... 0
Scheduled requests... 1
Total requests....... 40
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0MOBILE
Threads.............. 4 (4/256)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 1%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0PACKAGE-MANAGER
Threads.............. 2 (2/25)
Load average......... 0.00 0.00 0.00
Current load......... 0%
Usage................ 8%
Active requests...... 0
Scheduled requests... 0
Total requests....... 0
Thread starts........ 0
Thread stops......... 0
Wait time EMA........ 0 ms
Wait time SMA........ 0 ms
Wait time SD......... 0 ms
Queue size EMA....... 0
Queue size SMA....... 0
Queue size SD........ 0show queue
Data collector : 105
DCI cache loader : 0
Template updater : 0
Database writer : 0
Database writer (IData) : 2
Database writer (raw DCI values) : 6119
Event processor : 0
Event log writer : 0
Poller : 0
Node discovery poller : 0
SNMP trap processor : 0
SNMP trap writer : 0
Syslog processor : 0
Syslog writer : 0
Scheduler : 0
Windows event processor : 0
Windows event writer : 0 -
Sorry, this was overlooked for a while. First guess is that there are some issues with syncer thread. It's not visible in show threads, as there's just one syncer threads. In debug console there's
netxmsd: sh sy
Last run at .........: 16.Apr.2026 15:22:58
Last run time .......: 1 ms
Average run time ....: 119 ms
Max run time ........: 132 ms
Min run time ........: 0 msAnd System -> NetXMS Server -> Server Performance has metrics for it, specifically Server.SyncerRunTime.Last. Pls check if this metric looks reasonable.
Slow log opening might indicate problems with DB performance, DB might need tuning. What are the specs on DB VM, is it dedicated to NetXMS or a shared database? How's CPU consumption and IO stats? How big is the database on disk?
You can also use mysql subagent to collect stats about DB operation - https://netxms.org/documentation/adminguide/database-monitoring.html#mysql
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login