NetXMS Forum

Thanks Filipp.

The grep could afford to be a bit more detailed, for example, 'bin/netxmsd', as any open/tailed netxms log files will pass the grep and cause script failure later on. netxmsd-threads-8416.zip

Thanks, Victor. [jaustin@jaustin ~]$ nxdbmgr background-upgrade NetXMS Database Manager Version 3.3.285 Build 3.3-285-gfe2e9b646f (UNICODE) No pending background upgrades

Is there a way to speed up deletion of objects? syncer.cpp appears to be deleting objects one at a time. In this particular case, we're removing many individual interfaces prior to deletion of the parent node. The time to delete each interface is approximately 1 second, and then the next interface is queued for deletion. Is there no way to batch the deletions so they happen faster?

Thanks again, Filipp. That's quite interesting; I'm seeing an average of 10x more objects per node than in your examples. This could be due to device types, of course. I will see if I can determine relative counts for different types of objects.

Here's an example of such script, that will change community string for all nodes, that have SNMP polling enabled newAuthName = 'public2' for node in [o for o in session.getAllObjects() if isinstance(o, objects.Node)]: # filter all objects for objects.Node if (node.getFlags() & Node.NF_DISABLE_SNMP) = 0: # SNMP polling is not disabled if (node.getSnmpAuthName() != newAuthName): print('Processing node %s: changing community string from %s to %s' % (node.objectName, node.getSnmpAuthName(), newAuthName)) md = NXCObjectModificationData(node.objectId) md.setSnmpAuthName(newAuthName) session.modifyObject(md) The script is launched like this: java -jar nxshell-3.1.241.jar script_name.py There are some examples of Jython scripts in netxms sources in src/client/nxshell/samples/ Some examples are also here: https://wiki.netxms.org/wiki/Using_nxshell_to_automate_bulk_operations Documentation of API calls is here: https://www.netxms.org/documentation/javadoc/latest/overview-summary.html

I've tried on my system with Postgres 10 and NetXMS 3.1.261 on Ubuntu 18.04 - the reconnect actually occurs. Can you increase debugging for DB operations, e.g. by running this from command line: nxadm -c "debug db.* 8" and show the log during the moment when database stops/restarts.

I believe we were simply running out of memory overall. In this dev environment it's sharing space with postgresql. In the latest release, my best guess is that netxms behavior changed, resulting in more concurrent database queries. I dramatically lowered the number of database pools in use, and so far netxms has been stable with no reduction in performance. (I am monitoring threads via DCIs.)

Problem fixed will be included in next patch release

I believe I've tracked the pool exhaustion down. It's DCIs that are querying idata, which we've already shown is slow. I can disable them temporarily, but they are ultimately essential. netxms@netxmsts STATEMENT:  SELECT max(idata_value::double precision) FROM idata WHERE item_id=$1 AND idata_timestamp BETWEEN $2 AND $3 AND idata_value~E'^\\d+(\\.\\d+)*$' 2019-10-02 13:13:24.901 AKDT [28860] netxms@netxmsts ERROR:  out of shared memory 2019-10-02 13:13:24.901 AKDT [28860] netxms@netxmsts HINT:  You might need to increase max_locks_per_transaction. 2019-10-02 13:13:25.043 AKDT [28099] netxms@netxmsts LOG:  duration: 10728.421 ms  bind <unnamed>: SELECT max(idata_value::double precision) FROM idata WHERE item_id=$1 AND idata_timestamp BETWEEN $2 AND $3 AND idata_value~E'^\\d+(\\.\\d+)*$' 2019-10-02 13:13:25.043 AKDT [28099] netxms@netxmsts DETAIL:  parameters: $1 = '219502', $2 = '1570049993', $3 = '1570050293' 2019-10-02 13:13:26.104 AKDT [28108] netxms@netxmsts LOG:  duration: 10927.128 ms  bind <unnamed>: SELECT min(idata_value::double precision) FROM idata WHERE item_id=$1 AND idata_timestamp BETWEEN $2 AND $3 AND idata_value~E'^\\d+(\\.\\d+)*$' 2019-10-02 13:13:26.104 AKDT [28108] netxms@netxmsts DETAIL:  parameters: $1 = '219506', $2 = '1570049993', $3 = '1570050293' 2019-10-02 13:14:31.489 AKDT [28107] netxms@netxmsts LOG:  duration: 144850.468 ms  statement: SELECT idata_value,idata_timestamp FROM idata WHERE item_id=222029 ORDER BY idata_timestamp DESC LIMIT 1

Yes, NetXMS uses explicit cutoff points for dropping chunks. But please note that TimescaleDB support in 2.2.16 is quite immature - we have improved it significantly in 3.0 branch. Best regards, Victor

Does anyone have experience querying DC table data directly using Postgres? I'm working in Grafana, and having good success with non-table DCIs. For complicated reasons I need to be able to create custom queries rather than relying on NetXMS' built-in data export features. By parsing dctable.cpp, I believe I get the gist of how tables are constructed using dc_table_columns, with the cells themselves stored in tdata, but I'm not understanding how the tdata_value field actually works, or whether it's even possible to parse it outside of NetXMS itself. Another option would be if the Grafana API plugin for NetXMS supported tables, but I'm not sure anyone's working on that.

I narrowed the error down to the included json-20160212.jar (maven dependency?) I replaced this with json-20180813.jar json-20180813.jar works for non-table DCIs.

Thanks, Victor. I can confirm that this works for both base stations and clients: [jaustin@jaustin applications]$ snmpwalk -v1 -c xxxxx <mgmt address> 1.3.6.1.4.1.4458.1000.1.1.6 iso.3.6.1.4.1.4458.1000.1.1.6.0 = IpAddress: <mgmt IPv4 address> [jaustin@jaustin applications]$ snmpwalk -v1 -c xxxxx <mgmt address> 1.3.6.1.4.1.4458.1000.1.1.7 iso.3.6.1.4.1.4458.1000.1.1.7.0 = IpAddress: <mgmt IPv4 mask> Would a Radwin driver fix the problem of the response to the ping3 driver? Currently I'm having to disable the ping3 driver, as I don't have any ping3 devices, and the Radwins reject the entire SNMP discovery query (currently three OIDs, two in mib2 and one from ping3 private, IIRC).

Thanks, Victor; duplicate detection makes sense. For some reason I had thought that it was enabled by default. Will let you know if we still have issues.

Thank you, Victor. I may build from source... but is there a timetable or release schedule for 2.2.15?

I'm testing Hook::CreateSubnet functionality in 2.2.14 HEAD. Is there documentation of the NXSL 'Subnet' class? I note that the local server node is automatically discovered and added to the node database. Multiple instances (!) of one of the IPv4 subnets get created. (In this case, there are multiple IPv4 addresses on the server in that subnet.) Is the subnet creation logic not checking for a pre-existing subnet when multiple addresses exist on the same interface?

I have been seeing some strange behavior for a few days. Database performance seems fine, running on flash. Attempting to handle/terminate/resolve more than even a few alarm entries at once results in a pegged CPU core with netxmsd. netxmsd: show dbstats SQL query counters: Total .......... 2061537 SELECT ......... 861140 Non-SELECT ..... 1200397 Long running ... 0 Failed ......... 0 Background writer requests: DCI data ....... 20263 DCI raw data ... 20262 Others ......... 49 netxmsd: show msgwq 0 active queues Housekeeper thread state is RUNNING Show pollers shows about half and half in cleanup and awaiting execution. netxmsd: show queues Data collector : 0 DCI cache loader : 0 Template updates : 0 Database writer : 0 Database writer (IData) : 0 Database writer (raw DCI values) : 0 Event processor : 0 Event log writer : 0 Poller : 0 Node discovery poller : 0 Syslog processing : 0 Syslog writer : 0 Scheduler : 0 Show stats will time out while the CPU core is pegged. netxmsd: show watchdog Thread Interval Status ---------------------------------------------------------------------------- Item Poller 10 Running Syncer Thread 30 Sleeping Poll Manager 5 Sleeping Ad hoc scheduler 5 Sleeping Recurrent scheduler 5 Sleeping Stopping the netxmsd process and repairing the DB will resolve the stuck CPU temporarily. Viewing the logs, I get a lot of "Poll Manager" does not respond to watchdog thread. Anything further to check?

The check did not fix anything and when looking at Data Collection Configuration, the DCI items were still showing template as unknown. Now, thanks to the generosity of Tursiops, I have a Linux snmp template - so thought a clean install in case of the unknown templates. The new template is in and already live - a fantastic day for an absolutely fantastic product.  

@victor said in Radwin discovery failing: Just added two new configuration options for version 3.3: driver blacklisting and switching SNMP probing into one OID per request mode. Either one should fix this issue. Best regards, Victor This configuration option for SNMP probing resolved a tough issue I had been trying to resolve for a year.  I agree with jermudgeon... you rock, Victor!