NetXMS Forum

NetXMS Server Remote Console V6.1.0 Ready

 sh st
Objects............: 25145
  Nodes...........: 3177
  Interfaces......: 19762
  Access points...: 651
  Sensors.........: 0
Collectible DCIs...: 44734
Active alarms......: 3924
Uptime.............: 2 days, 19:42:54

 sh th
MAIN
  Threads.............. 8 (8/600)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 1%
  Active requests...... 0
  Scheduled requests... 3
  Total requests....... 4624
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

AGENT
  Threads.............. 32 (32/256)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 12%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 9455
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

POLLERS
  Threads.............. 484 (10/600)
  Load average......... 195.17 240.33 250.33
  Current load......... 58%
  Usage................ 80%
  Active requests...... 284
  Scheduled requests... 0
  Total requests....... 5389134
  Thread starts........ 749
  Thread stops......... 275
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

FILE-TRANSFER
  Threads.............. 2 (2/16)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 12%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

DATACOLL
  Threads.............. 10 (10/600)
  Load average......... 2.51 1.65 1.38
  Current load......... 0%
  Usage................ 1%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 17275228
  Thread starts........ 66
  Thread stops......... 66
  Wait time EMA........ 29 ms
  Wait time SMA........ 28 ms
  Wait time SD......... 37 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

THREVT
  Threads.............. 2 (2/4)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 50%
  Active requests...... 0
  Scheduled requests... 1
  Total requests....... 75
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

CLIENT
  Threads.............. 16 (16/2048)
  Load average......... 0.03 0.01 0.00
  Current load......... 12%
  Usage................ 0%
  Active requests...... 2
  Scheduled requests... 0
  Total requests....... 25173
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

DISCOVERY
  Threads.............. 8 (8/64)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 12%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

SCHEDULER
  Threads.............. 6 (1/64)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 9%
  Active requests...... 0
  Scheduled requests... 1
  Total requests....... 1026
  Thread starts........ 5
  Thread stops......... 0
  Wait time EMA........ 643 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

MOBILE
  Threads.............. 4 (4/256)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 1%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

PACKAGE-MANAGER
  Threads.............. 2 (2/25)
  Load average......... 0.00 0.00 0.00
  Current load......... 0%
  Usage................ 8%
  Active requests...... 0
  Scheduled requests... 0
  Total requests....... 0
  Thread starts........ 0
  Thread stops......... 0
  Wait time EMA........ 0 ms
  Wait time SMA........ 0 ms
  Wait time SD......... 0 ms
  Queue size EMA....... 0
  Queue size SMA....... 0
  Queue size SD........ 0

 sh qu
Data collector                  : 0
DCI cache loader                : 0
Template updater                : 0
Database writer                  : 0
Database writer (alarms)        : 0
Database writer (IData)          : 0
Database writer (raw DCI values) : 1934
Event processor                  : 0
Event log writer                : 0
Poller                          : 0
Node discovery poller            : 0
SNMP trap processor              : 0
SNMP trap writer                : 0
Syslog processor                : 0
Syslog writer                    : 0
Scheduler                        : 0
Windows event processor          : 0
Windows event writer            : 0

Ubuntu VM on 22.04 Intel Xeon Skylake Processor 16 Cores, 32 Gbs RAM, 500 GB Virtual drive running on SSD

Using 15 GBs of RAM currently and CPU usage stays under 10%

I am currently using NetXMS 6.1 and I've noticed that I am not always getting alarms when nodes become unreachable at least not immediately. I have also observed timeouts when using the client for simple activities like creating a new node that I have previously not encountered before.

I took a look at the log for my server and noticed these entries -

2026.04.17 09:16:50.436 *I* [watchdog          ] Thread "Recurrent scheduler" r                          eturned to running state
2026.04.17 09:17:48.752 *I* [watchdog          ] Thread "Syncer Thread" returne                          d to running state
2026.04.17 09:19:36.335 *E* [watchdog          ] Thread "Syncer Thread" does no                          t respond to watchdog thread
2026.04.17 09:21:56.337 *E* [watchdog          ] Thread "Recurrent scheduler" d                          oes not respond to watchdog thread
2026.04.17 09:24:24.322 *I* [watchdog          ] Thread "Recurrent scheduler" r                          eturned to running state
2026.04.17 09:26:36.342 *E* [watchdog          ] Thread "Recurrent scheduler" d                          oes not respond to watchdog thread
2026.04.17 09:32:38.323 *I* [watchdog          ] Thread "Recurrent scheduler" r                          eturned to running state
2026.04.17 09:33:06.359 *I* [watchdog          ] Thread "Syncer Thread" returne                          d to running state
2026.04.17 09:34:56.347 *E* [watchdog          ] Thread "Syncer Thread" does no                          t respond to watchdog thread
2026.04.17 09:35:56.348 *E* [watchdog          ] Thread "Recurrent scheduler" d                          oes not respond to watchdog thread

I'd appreciate any suggestions on how to fix this.

I had to increase my shared memory and increase max transaction locks, but I am also back in business. 

This appears more relevant

>>> RELEASE SAVEPOINT nxdbmgr_1771000399839
>>> SELECT var_value FROM metadata WHERE var_name='IDataTableCreationCommand'
>>> CREATE TABLE idata_46970 (item_id integer not null,idata_timestamp bigint not null,idata_value varchar(255) null,raw_value varchar(255) null, PRIMARY KEY(item_id,idata_timestamp))
SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
CREATE TABLE idata_46970 (item_id integer not null,idata_timestamp bigint not null,idata_value varchar(255) null,raw_value varchar(255) null, PRIMARY KEY(item_id,idata_timestamp))
Rolling back last stage due to upgrade errors...
Database upgrade failed
>>> SELECT var_value FROM metadata WHERE var_name='PendingOnlineUpgrades'

Getting a lot of text - here's a snippet

DROP TABLE tdata_8505
>>> RELEASE SAVEPOINT nxdbmgr_1771000353984
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_0'
>>> CREATE TABLE tdata_8505 (item_id integer not null,tdata_timestamp bigint not null,tdata_value text null, PRIMARY KEY(item_id,tdata_timestamp))
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_1'
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_2'
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_3'
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_4'
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_5'
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_6'
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_7'
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_8'
>>> SELECT var_value FROM metadata WHERE var_name='TDataTableCreationCommand_9'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_0'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_1'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_2'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_3'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_4'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_5'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_6'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_7'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_8'
>>> SELECT var_value FROM metadata WHERE var_name='TDataIndexCreationCommand_9'
>>> SELECT COUNT(*) FROM idata_74790

I am receiving similar errors, but I believe I caused my own problems by running the commands that Victor posted earlier.

DROP TABLE idata;
CREATE TABLE idata
(
  item_id integer not null,
  idata_timestamp integer not null,
  idata_value varchar(255) null,
  raw_value varchar(255) null
);
DROP TABLE tdata;
CREATE TABLE tdata
(
  item_id integer not null,
  tdata_timestamp integer not null,
  tdata_value varchar(max) null
);

Is my database irrevocably damaged?

I saw the database upgrade start when I used apt get upgrade. I thought that the server could then be started while that was running, but I suppose not.

I can see that the database upgrade is happening in the background, but the netxms-server service won't start.

"Your database has format version 60.7, but server is compiled for version 60.32"

Hello,

I am running into a similar issue on a Ubuntu installation where the database has taken multiple days to upgrade showing that it is converting idata and tdata tables, but never completing.

Thanks in advance!

Thanks tolimanjo. That limited it, but it seems that I can't call the variable interface.name for the display name.

Hello,

I'm trying to setup a template that collects port negotiation rates on Mikrotik devices. I get the info that I want on physical ports, but I get 0 results on virtual interfaces. I would like to filter out virtual interfaces so DCIs are not being created.

transport = CreateSNMPTransport($node);
if (transport == null)
 return false;
// get the interface name to return as instance-name
vb = SNMPGet(transport, ".1.3.6.1.2.1.31.1.1.1.1." .. $1);
if (vb == null) or (vb == 0)
 return false;
// filter out loopback
if (vb.value == "lo")
 return false;
return %(true, $1, vb.value);

I've tried using contains(vb.value, "vlan") on the bottom statement, but that just breaks the script.

Quite a few of these cameras were added to NetXMS when they were already offline. They then all polled down as they were checking for SNMP. I terminated the first alarms as I knew they didn't have SNMP functionality, but I have been unable to get them to change status based on ICMP either. They have also been offline since addition so I figure that would be why their status is not changing.

It is indeed.

 

ID	Time	Origin time	Origin	Source	Zone	DCI	Event	Severity	Message	Event tags	Root ID
48056062	11.05.2025 06:03:47	11.05.2025 06:03:47	System	4th Fl Stair 2	Default	0	SYS_ICMP_UNREACHABLE	Major	Node is unreachable by ICMP	48056060
48056061	11.05.2025 06:03:46	11.05.2025 06:03:46	System	4th Fl Stair 2	Default	0	SYS_NODE_CRITICAL	Critical	Node status changed to CRITICAL	NodeStatus	0
48056060	11.05.2025 06:03:46	11.05.2025 06:03:46	System	4th Fl Stair 2	Default	0	SYS_NODE_DOWN	Critical	Site:Vicina Main 4th Fl Stair 2 is unresponsive - 10.1.102.171	0

So it looks like ICMP status check updates for cameras that were added when they were online.

Offline Camera:
[12.05.2025 09:24:38] **** Poll request sent to server ****
[12.05.2025 09:24:38] Poll request accepted, waiting for outstanding polling requests to complete...
[12.05.2025 09:24:38] Starting status poll of node Tara Kitchen - 3
[12.05.2025 09:24:38]    Starting status poll on interface unknown
[12.05.2025 09:24:38]      Current interface status is NORMAL
[12.05.2025 09:24:38]      Starting ICMP ping
[12.05.2025 09:24:41]      Interface is NORMAL for 3659 polls (3 polls required for status change)
[12.05.2025 09:24:41]      Interface status after poll is NORMAL
[12.05.2025 09:24:41]    Finished status poll on interface unknown
[12.05.2025 09:24:41] Checking primary IP address with ICMP ping
[12.05.2025 09:24:45]   Primary IP address is not responding to ICMP ping
[12.05.2025 09:24:45] Node is still unreachable
[12.05.2025 09:24:45] Finished status poll of node Tara Kitchen - 3
[12.05.2025 09:24:45] Node status after poll is NORMAL
[12.05.2025 09:24:45] **** Poll completed successfully ****

Online Camera:
[12.05.2025 09:30:20] **** Poll request sent to server ****
[12.05.2025 09:30:20] Poll request accepted, waiting for outstanding polling requests to complete...
[12.05.2025 09:30:20] Starting status poll of node Tara Kitchen - 4
[12.05.2025 09:30:20]    Starting status poll on interface unknown
[12.05.2025 09:30:20]      Current interface status is NORMAL
[12.05.2025 09:30:20]      Starting ICMP ping
[12.05.2025 09:30:20]      Interface is NORMAL for 3813 polls (3 polls required for status change)
[12.05.2025 09:30:20]      Interface status after poll is NORMAL
[12.05.2025 09:30:20]    Finished status poll on interface unknown
[12.05.2025 09:30:20] Node is connected
[12.05.2025 09:30:24] Finished status poll of node Tara Kitchen - 4
[12.05.2025 09:30:24] Node status after poll is NORMAL
[12.05.2025 09:30:24] **** Poll completed successfully ****

I suspect that this is due to certain cameras being down when they were first added, but I would like for them to show down even if they never polled up when first added.

Hello,

I am trying to monitor network security cameras that have no SNMP functionality. I set the objects to use ICMP to determine node status, but the objects don't generate alarms even though they are offline. I tried resetting the alarms using the link below:

Cameras showing 100% ICMP packet loss show as normal and don't have any alarms even after alarm status reset. Any suggestions?

It works beautifully now. Thank you so much, Filipp!

Thanks, Filipp. That got me farther.

Traceback (most recent call last):
  File "create_nodes.py", line 17, in <module>
    node_name = data[i].strip()
AttributeError: 'list' object has no attribute 'strip'

  Hey Filipp,

I didn't notice your response to this nearly 2 years ago. I don't know if this table structure is still useful, but it is what is exported from the Dude device list.

Flag	Name	Addresses	MAC	Type	Maps	Services Down	Notes

I was revisiting this thread as I was trying to bulk import older devices we weren't previously monitoring into our NetXMS deployment. I figured that I could re-use this script, but I am running into errors now when I try to use it.

SyntaxError: no viable alternative at input 'admin'

Thanks Victor - that did it!