NetXMS Forum

Additional info that I missed from the main post

show thread
MAIN
  Threads.............. 8 (8/512)
  Load average......... 0.00 0.01 0.00
  Current load......... 0%
  Usage................ 1%
  Active requests...... 0
  Scheduled requests... 1
  Total requests....... 104912
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

AGENT
  Threads.............. 32 (32/256)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 12%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

POLLERS
  Threads.............. 80 (10/500)
  Load average......... 15.51 28.01 35.31
  Current load......... 2%
  Usage................ 16%
  Active requests...... 2
  Scheduled requests... 0
  Total requests....... 367629
  Thread starts........ 427
  Thread stops......... 357
  Wait time EMA........ 274 ms
  Wait time SMA........ 152 ms
  Wait time SD......... 272 ms
  Queue size EMA....... 7
  Queue size SMA....... 0
  Queue size SD........ 0

FILE-TRANSFER
  Threads.............. 2 (2/16)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 12%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

NPE
  Threads.............. 1 (1/1024)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 0%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

DATACOLL
  Threads.............. 17 (10/300)
  Load average......... 0.00 0.07 0.42
  Current load......... 0%
  Usage................ 5%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 275896
  Thread starts........ 17
  Thread stops......... 10
  Wait time EMA........ 80 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

CLIENT
  Threads.............. 16 (16/2048)
  Load average......... 0.00 0.00 0.00
  Current load......... 12%
  Usage................ 0%
  Active requests...... 2
  Scheduled requests... 0
  Total requests....... 84567
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

DISCOVERY
  Threads.............. 8 (8/64)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 12%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

SCHEDULER
  Threads.............. 1 (1/64)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 1%
  Active requests...... 0
  Scheduled requests... 1
  Total requests....... 40
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

MOBILE
  Threads.............. 4 (4/256)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 1%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

PACKAGE-MANAGER
  Threads.............. 2 (2/25)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 8%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

 show queue
Data collector                  : 105
DCI cache loader                : 0
Template updater                : 0
Database writer                  : 0
Database writer (IData)          : 2
Database writer (raw DCI values) : 6119
Event processor                  : 0
Event log writer                : 0
Poller                          : 0
Node discovery poller            : 0
SNMP trap processor              : 0
SNMP trap writer                : 0
Syslog processor                : 0
Syslog writer                    : 0
Scheduler                        : 0
Windows event processor          : 0
Windows event writer            : 0

Hi,

We are investigating an issue where objects disappeared after a NetXMS crash and we are seeing unusual behaviour with the database connection pool and parts of the UI. We would appreciate guidance on possible causes.

Environment

NetXMS: v5.2.3

Database: MariaDB 10.11

NetXMS server and DB run on separate VMs with separate disks

NetXMS server specs:
16 CPU cores
64 GB RAM

System size:
Objects: 23022
Nodes: 1129
Interfaces: 21081
Access Points: 0
Sensors: 0
Collectible DCIs: 14890

MaxTransactionSize = 1000

Incident
The NetXMS server crashed after the root partition on the NetXMS server VM became full. The database is hosted on a different VM with its own storage and did not run out of space.

When bringing NetXMS back online:
Some objects that had been created several days earlier had disappeared.
Other newer objects and data were still present (inc alarms for dci that had disappered), suggesting that only some metadata was lost or never committed.

Shutdown behaviour
When attempting to restart the NetXMS service:
The server did not shut down gracefully.
It had to be killed manually.

Database connection pool observation
During investigation we ran show dbcp and noticed that for extended periods (10–30 seconds) it reports:
0 database connections in use

This occurs even though the system is actively monitoring ~23k objects and ~14.8k collectible DCIs.

 show dbstat
SQL query counters:
  Total .......... 1100783
  SELECT ......... 306641
  Non-SELECT ..... 794106
  Long running ... 0
  Failed ......... 0
Background writer requests:
  DCI data ....... 246179
  DCI raw data ... 246179
  Others ......... 234

Additional anomalies
We are also seeing unusual behaviour in both the web client and the desktop client:
Some log views fail to load or take an extremely long time to load pages or do not load at all.
Pagination for logs sometimes does not work or stalls.

These behaviours appear in both the web UI and the native NetXMS client.
This makes us suspect there may be an issue with database access, internal queues, or blocked threads.

Questions
Could a full root partition on the NetXMS server prevent metadata commits to the database even if the DB is on a separate system?
Is it possible that object metadata was only held in server memory and never committed, resulting in the objects disappearing after the crash?
What could cause the DB connection pool to show 0 connections in use for long periods on a system of this size?
Could this indicate blocked writer threads, transaction batching issues, or internal locks?
Could the log loading and pagination issues in both clients be related to the same underlying database or thread issue?

Any guidance on what diagnostics we should run (thread state, DB writer queues, etc.) would be greatly appreciated.

Thanks
Darren

Hi all,

Hoping someone might know the answer, we are looking to retain the audits, events, DCI's and Syslogs etc for node's that are being decommissioned, we are needing to keep this information for over a rolling year.
As I understand you can change the IP and Node name etc without loosing this information, but when decommissioning and deleting the node, the logs when searching in the client cannot found (this makes sense as the node is no longer there).
But is the information still within the database or are these removed too?

I'm not a sysadmin and have very little experience with databases so do not want to break anything should I go looking.

I see that NetXMS V6 has a inbuilt decommissioning function (which is great), but we cannot roll this out quickly as it would need to be signed of and a whole new project started for this to get off the ground.

Thanks
Darren

Hi all, we have been using NetXMS 2.2.12 since 2019 and have got to the point that due to old hardware and multiple failures that we have now rebuilt NetXMS 5.2.2 on new hardware and are in the mist of getting it ready to deploy. 

One thing (that I have probably read and cannot now find) is that when you filter for an interface in the Infrastructure tab the interface is not found. The only way to get this to work is by finding the interface manually (finding the interface by expanding through the containers ect till you get to the interface). After selecting the interface and then shrinking containers, the interface is then findable using the filter, but after short while or closing client the interface cannot find the interface. 
We have 21151 interfaces across 1087 nodes, NetXMS version 2.2.12 would find using the filter. 

Using the Find Object tool it finds the interface and show where it lives. 

Am I missing something or slight configuration?

Thanks
Darren