<![CDATA[Server Performance]]>I know from the many NMS systems that I have implemented / tested that something is up.  Just not sure how to start digging to find the issue and hoping to get some pointers.  We are running on a quad core Xeon / 4GB RAM machine running Ubuntu 12.04, NetXMS 1.2.1 (from .deb files).  Currently have about 200 nodes and 1500 DCI's setup.  The machine is running NOTHING other than NetXMS and services to support the installation.  Here is a screen grab from top:

# top
top - 11:15:55 up 18:36,  1 user,  load average: 14.94, 14.19, 14.36
Tasks:  30 total,  1 running,  29 sleeping,  0 stopped,  0 zombie
Cpu(s): 23.2%us, 48.7%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 28.0%st
Mem:  4194304k total,  436432k used,  3757872k free,        0k buffers
Swap:  2097152k total,        0k used,  2097152k free,  176968k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                             
  284 root      20  0 2282m  24m 3592 S  400  0.6  3679:06 netxmsd                                             
  236 mysql    20  0 1654m 211m 8000 S  24  5.2 617:58.69 mysqld                                             
  291 root      20  0  632m 4884 1460 S  10  0.1 194:04.51 nxagentd                                           
    1 root      20  0 24024 2024 1340 S    0  0.0  0:00.15 init                                               
    2 root      20  0    0    0    0 S    0  0.0  0:00.00 kthreadd/105
...

Have considered recompiling from source, but prefer to use packages if available for ease of installation and hopefully optimal settings in build.  Can anybody else confirm they are running the .deb files under the latest Ubuntu LTS? 

I do have a question though about the software itself.  Are containers inside the "Infrastructure Services" node just considered logical groupings?  That is what I have assumed so we do have nodes that can appear under 3+ containers.  We have setup an "all node" container, grouped by device type, grouped by location and a few other ways we like to analyze the nodes in our network.    I am assuming software is polling node once, no matter how many times it appears under various containers.  If that is NOT the case then I probably have too much activity going on. 

Thanks!

]]>https://forum.netxms.org/topic/1820/server-performanceRSS for NodeFri, 13 Nov 2026 15:17:01 GMTWed, 11 Jul 2012 16:29:33 GMT60<![CDATA[Reply to Server Performance on Thu, 12 Jul 2012 14:37:36 GMT]]>Ok, this has been resolved.  While I had checked for the leap second bug on the machine, I had not checked the host it was virtualized on top of.  This was being affected by the issue and after fixing the host the NetXMS server machine returned to normal, in fact crazy LOW CPU usage (0.06) even though it is running perfectly from what I can tell.  Odd, because other machines on this host did not seem to be having this issue.   

Sorry about this and do appreciate all of the assistance!

]]>https://forum.netxms.org/post/7876https://forum.netxms.org/post/7876Thu, 12 Jul 2012 14:37:36 GMT<![CDATA[Reply to Server Performance on Thu, 12 Jul 2012 12:12:01 GMT]]>Yeah, I had already run in to the July 1st post stuff and have insured this is not the issue we are running into.

I have attached the gdb output, thanks!

netxmsd-gdb.txt

]]>https://forum.netxms.org/post/7873https://forum.netxms.org/post/7873Thu, 12 Jul 2012 12:12:01 GMT<![CDATA[Reply to Server Performance on Thu, 12 Jul 2012 10:16:48 GMT]]>Some time passed already since July 1st, but just in case - could it be this problem: https://www.netxms.org/forum/announcements/high-cpu-usage-after-172012/?

Could you also please attach debugger to netxmsd process and dump stack trace for all threads:

# gdb /usr/bin/netxmsd
(gdb) attach <netxmsd_pid>
(gdb) thread apply all bt
(gdb) detach

Best regards,
Victor

]]>https://forum.netxms.org/post/7872https://forum.netxms.org/post/7872Thu, 12 Jul 2012 10:16:48 GMT<![CDATA[Reply to Server Performance on Thu, 12 Jul 2012 03:07:15 GMT]]>Ok, so on the core router I checked:
  - Disable routing table polling
  - Disable topology polling
  - Disable network discovery polling

Stopped netxmsd and verified that load did indeed drop to very near zero which it did almost right away.  Restarted the server and then waited a few hours in case it had to calm down.  Had no effect, load averages were at the same level.

Downloaded the source code to the machine and compiled using:
  ./configure --prefix=/usr --with-server --with-mysql --with-agent
All went well and then installed/restarted again.  Had no positive / negative effect other than what appears to be a minimal amount more RAM being used - so you apparently have better compiler settings than I do 🙂 

Open to any further suggestions.  Thanks!

]]>https://forum.netxms.org/post/7870https://forum.netxms.org/post/7870Thu, 12 Jul 2012 03:07:15 GMT<![CDATA[Reply to Server Performance on Wed, 11 Jul 2012 19:39:42 GMT]]>Turn off routing and discovery polls on those BGP routers (you did that already if I understand correctly). Restart netxmsd after that if possible.

Routing table polls used for discovery and also can be used for routing change monitoring via Net.IP.NextHop internal parameter.

Best regards,
Victor

]]>https://forum.netxms.org/post/7867https://forum.netxms.org/post/7867Wed, 11 Jul 2012 19:39:42 GMT<![CDATA[Reply to Server Performance on Wed, 11 Jul 2012 18:56:02 GMT]]>Yes, I have multiple full bgp tables on the core router.  Also, depending what you consider large some of the internal routers have a lot of routes as well. 

What at this point is the routing table and topology polling being used for?  Since you are not auto creating dependencies, I am not sure other than  potentially discovery.

Potentially we can just turn this off on all devices.  I did turn those two off on the core router and have not seen any change, but maybe it has a lot queued up it needs to process.  Will give it some time.

]]>https://forum.netxms.org/post/7866https://forum.netxms.org/post/7866Wed, 11 Jul 2012 18:56:02 GMT<![CDATA[Reply to Server Performance on Wed, 11 Jul 2012 17:44:54 GMT]]>Could it be that you have router(s) with very large routing tables, like full BGP table?

Best regards,
Victor

]]>https://forum.netxms.org/post/7865https://forum.netxms.org/post/7865Wed, 11 Jul 2012 17:44:54 GMT<![CDATA[Reply to Server Performance on Wed, 11 Jul 2012 17:01:09 GMT]]>The two clients are my Android phone and Windows machine.  The android app is very nice by the way, but is definitely battery intensive. 

netxmsd: show sessions
ID  STATE                    CIPHER  USER [CLIENT]
0  idle                    AES-256  admin@xxx.xxx.xxx.249 [nxjclient/1.2.1 (Linux 2.6.32.9-00005-g2440aba; libnxcl 1.2.1)]
1  idle                    AES-128  admin@xxx.xxx.xxx.131 [nxjclient/1.2.1 (Windows 7 6.1; libnxcl 1.2.1)]

2 active sessions

netxmsd: show pollers
PT  TIME                  STATE
S  11/Jul/2012 11:55:57  wait
S  11/Jul/2012 11:55:59  wait
S  11/Jul/2012 11:56:01  wait
S  11/Jul/2012 11:56:01  wait
S  11/Jul/2012 11:56:01  wait
S  11/Jul/2012 11:55:54  wait
S  11/Jul/2012 11:55:57  wait
S  11/Jul/2012 11:56:01  wait
S  11/Jul/2012 11:55:54  wait
S  11/Jul/2012 11:55:58  wait
S  11/Jul/2012 11:56:02  wait
S  11/Jul/2012 11:55:57  wait
S  11/Jul/2012 11:55:58  wait
S  11/Jul/2012 11:56:02  wait
S  11/Jul/2012 11:55:53  wait
S  11/Jul/2012 11:56:01  wait
S  11/Jul/2012 11:55:54  wait
S  11/Jul/2012 11:56:01  wait
S  11/Jul/2012 11:56:01  wait
S  11/Jul/2012 11:56:02  wait
S  11/Jul/2012 11:56:01  wait
S  11/Jul/2012 11:55:59  wait
S  11/Jul/2012 11:55:58  wait
S  11/Jul/2012 11:56:01  wait
S  11/Jul/2012 11:55:58  wait
C  11/Jul/2012 11:54:11  wait
C  11/Jul/2012 11:51:45  wait
C  11/Jul/2012 11:55:59  wait
C  11/Jul/2012 11:55:57  poll: xxx00-au-nn.excel.net [2268] - capability check
C  11/Jul/2012 11:54:04  wait
C  11/Jul/2012 11:56:01  poll: xxx00-au-nn.excel.net [2275] - capability check
C  11/Jul/2012 11:51:51  wait
C  11/Jul/2012 11:52:04  wait
C  11/Jul/2012 11:55:18  wait
C  11/Jul/2012 11:55:18  wait
C  11/Jul/2012 11:51:56  wait
C  11/Jul/2012 11:55:35  wait
C  11/Jul/2012 11:56:03  wait
C  11/Jul/2012 11:55:25  poll: xxx03-au-xx.excel.net [2210] - capability check
C  11/Jul/2012 11:56:03  wait
R  11/Jul/2012 11:51:33  wait
R  11/Jul/2012 11:50:57  wait
R  11/Jul/2012 11:51:41  wait
R  11/Jul/2012 11:52:23  wait
R  11/Jul/2012 11:51:29  wait
R  11/Jul/2012 11:51:29  wait
R  11/Jul/2012 11:51:29  wait
R  11/Jul/2012 11:56:00  wait
R  11/Jul/2012 11:52:04  wait
R  11/Jul/2012 11:55:49  poll: xxx05-rtr-00.excel.net [2043]
D  11/Jul/2012 11:53:08  wait
N  10/Jul/2012 16:39:09  wait
N  10/Jul/2012 16:39:09  wait
N  10/Jul/2012 16:39:09  wait
N  10/Jul/2012 16:39:09  wait
N  10/Jul/2012 16:39:09  wait
N  10/Jul/2012 16:39:09  wait
N  10/Jul/2012 16:39:09  wait
N  10/Jul/2012 16:39:09  wait
N  10/Jul/2012 16:39:09  wait
N  10/Jul/2012 16:39:09  wait
T  11/Jul/2012 11:54:14  wait
T  11/Jul/2012 11:55:19  wait
T  11/Jul/2012 11:55:13  wait
T  11/Jul/2012 11:55:19  wait
T  11/Jul/2012 11:55:35  wait
T  11/Jul/2012 11:54:41  wait
T  11/Jul/2012 11:56:03  wait
T  11/Jul/2012 11:56:01  poll: xxx00-au-nn.excel.net [2268]
T  11/Jul/2012 11:55:08  wait
T  11/Jul/2012 11:54:13  wait
B  10/Jul/2012 16:41:09  wait
B  10/Jul/2012 16:41:09  wait
B  10/Jul/2012 16:41:09  wait
B  10/Jul/2012 16:41:09  wait
B  10/Jul/2012 16:41:09  wait
B  10/Jul/2012 16:41:09  wait
B  10/Jul/2012 16:41:09  wait
B  10/Jul/2012 16:41:09  wait
B  10/Jul/2012 16:41:09  wait
B  10/Jul/2012 16:41:09  wait
A  11/Jul/2012 05:11:51  wait

netxmsd: show queue
Condition poller                : 0
Configuration poller            : 0
Topology poller                  : 0
Data collector                  : 0
Database writer                  : 0
Database writer (IData)          : 0
Event processor                  : 0
Network discovery poller        : 0
Node poller                      : 1587
Routing table poller            : 0
Status poller                    : 0

netxmsd: show stats
Total number of objects:    1580
Number of monitored nodes:  172
Number of collectable DCIs:  1396

netxmsd: show flags
Flags: 0x4310067D
  AF_DAEMON                        = 1
  AF_USE_SYSLOG                    = 0
  AF_ENABLE_NETWORK_DISCOVERY      = 1
  AF_ACTIVE_NETWORK_DISCOVERY      = 1
  AF_LOG_SQL_ERRORS                = 1
  AF_DELETE_EMPTY_SUBNETS          = 1
  AF_ENABLE_SNMP_TRAPD            = 1
  AF_ENABLE_ZONING                = 0
  AF_SYNC_NODE_NAMES_WITH_DNS      = 0
  AF_CHECK_TRUSTED_NODES          = 1
  AF_WRITE_FULL_DUMP              = 0
  AF_RESOLVE_NODE_NAMES            = 1
  AF_CATCH_EXCEPTIONS              = 0
  AF_INTERNAL_CA                  = 0
  AF_DB_LOCKED                    = 1
  AF_ENABLE_MULTIPLE_DB_CONN      = 1
  AF_DB_CONNECTION_LOST            = 0
  AF_NO_NETWORK_CONNECTIVITY      = 0
  AF_EVENT_STORM_DETECTED          = 0
  AF_SERVER_INITIALIZED            = 1
  AF_SHUTDOWN                      = 0

netxmsd: show watchdog
Thread                                          Interval Status
----------------------------------------------------------------------------
Item Poller                                      20      Running
Syncer Thread                                    130      Running
Poll Manager                                    60      Running

]]>https://forum.netxms.org/post/7864https://forum.netxms.org/post/7864Wed, 11 Jul 2012 17:01:09 GMT<![CDATA[Reply to Server Performance on Wed, 11 Jul 2012 16:43:53 GMT]]>Situation is definitely not normal. Can you please open server debug console (from management console or with nxadm -i), and send me output of the following commands:

show sessions
show pollers
show queue
show stats
show flags
show watchdog

You are correct, containers in infrastructure tree are for logical grouping. Polling is not related to node location in container, and node is polled only once no matter how many times it appears in the tree.

Best regards,
Victor

]]>https://forum.netxms.org/post/7863https://forum.netxms.org/post/7863Wed, 11 Jul 2012 16:43:53 GMT