<![CDATA[NetXMS Server Crashing]]>Server is, Windows 2008 R2 64 Bit with SQL Server 2008 db on a separate machine. Running 1.0.5 server with 1.0.6 clients.

The following error is happening,

We have about 450 agents and maybe a total of 1500 nodes. Only about 4-6 DCI per node, vast majority are just monitoring windows event logs for errors and criticals.

I am noticing that we are getting a fair amount of agents dropping connectivity and then restoring in blocks of about 10-20 every 10-20 minutes. The crashes only started after we went from about 300 agents installed to 450 agents installed. I put the numberofstatuspollers up to 50 at first and that didn't seem to help, now I am putting the number back down to 30. That seems related but I'm not sure. The netowrk utilization is steady at about 2-3 percent. The processor is steady at about 10-20 percent. Disk IO is nothing...

Any suggestions based on this information would be helpful.

]]>https://forum.netxms.org/topic/1016/netxms-server-crashingRSS for NodeFri, 13 Nov 2026 06:07:19 GMTFri, 05 Nov 2010 15:15:49 GMT60<![CDATA[Reply to NetXMS Server Crashing on Wed, 17 Nov 2010 08:05:47 GMT]]>Hi!

Last special build uses SO_REUSEADDR option - it makes no difference. As for MaxUserPort, things are different on Windows 2008 - http://support.microsoft.com/kb/929851. So by default we should have about 16000 ports for outgoing connections, which is much more then required.

Best regards,
Victor

]]>https://forum.netxms.org/post/4706https://forum.netxms.org/post/4706Wed, 17 Nov 2010 08:05:47 GMT<![CDATA[Reply to NetXMS Server Crashing on Wed, 17 Nov 2010 04:46:59 GMT]]>Do you guys has any reluctance on my suggestion put about

  1. Microsoft Loopback adapter
  2. Registry values for MaxUserPort and TcpTimedWaitDelay
  3. Using SO_REUSEADDR by setsockopt()
    Please give them a try. I undestand that some setup have much higher load then current setup but you need to understand that every setup/network is different!!!
]]>https://forum.netxms.org/post/4705https://forum.netxms.org/post/4705Wed, 17 Nov 2010 04:46:59 GMT<![CDATA[Reply to NetXMS Server Crashing on Tue, 16 Nov 2010 21:01:32 GMT]]>It also could be Windows 2008 issue. We have installation in Riga with 700+ agents - and I never seen issues like that. They are running on Windows Server 2003 x86.

Best regards,
Victor

]]>https://forum.netxms.org/post/4704https://forum.netxms.org/post/4704Tue, 16 Nov 2010 21:01:32 GMT<![CDATA[Reply to NetXMS Server Crashing on Tue, 16 Nov 2010 20:49:52 GMT]]>It all seemed to start after we went from about 150-200 agents to about 400+ agents.

I'm going to be moving the system to a 32 bit server today or tomorrow to see if that elimates the connection issue, I don't think it would be related but worth a try.

]]>https://forum.netxms.org/post/4703https://forum.netxms.org/post/4703Tue, 16 Nov 2010 20:49:52 GMT<![CDATA[Reply to NetXMS Server Crashing on Tue, 16 Nov 2010 20:17:07 GMT]]>Hi!

I'm still unable to reproduce this issue or find any clue to the source of the problem. Please try to replace libnetxms.dll to attached one - it has another changes in communication code. At least, it may give more meaningful error codes in "agent unreachable" messages in the log.

Also, do you experience this issue from the beginning, or it appears after latest upgrades?

Best regards,
Victor

libnetxms.dll

]]>https://forum.netxms.org/post/4702https://forum.netxms.org/post/4702Tue, 16 Nov 2010 20:17:07 GMT<![CDATA[Reply to NetXMS Server Crashing on Tue, 16 Nov 2010 15:54:45 GMT]]>Okay performed the update and I pretty much don't see a difference.

Here are a variety of entries for the error you were asking about.

This is the most common one, lots of these 98% is this.

agent unreachable, error=910, socketError=0

Then we have some of these,

agent unreachable, error=500, socketError=0

A few of these,

agent unreachable, error=500, socketError=10053
agent unreachable, error=500, socketError=10054

]]>https://forum.netxms.org/post/4701https://forum.netxms.org/post/4701Tue, 16 Nov 2010 15:54:45 GMT<![CDATA[Reply to NetXMS Server Crashing on Mon, 15 Nov 2010 22:32:29 GMT]]>Could you please try to install updated build: https://www.netxms.org/download/rc/netxms-1.0.8-rc2.exe. If it will not solve the problem, please set logging to file (by setting LogFile = some_file in netxmsd.conf) and run netxmsd.exe with -D 6 command line option (this will enable debug output). It will generate a lot of messages in the log file - I'm interested in messages containing words "agent unreachable, error=".

Best regards,
Victor

]]>https://forum.netxms.org/post/4697https://forum.netxms.org/post/4697Mon, 15 Nov 2010 22:32:29 GMT<![CDATA[Reply to NetXMS Server Crashing on Mon, 15 Nov 2010 20:42:45 GMT]]>Yes windows firewall is running, disabled for domain, enabled in private and public. Not currently logging.
Don't see any netxmsd log / error.

]]>https://forum.netxms.org/post/4695https://forum.netxms.org/post/4695Mon, 15 Nov 2010 20:42:45 GMT<![CDATA[Reply to NetXMS Server Crashing on Mon, 15 Nov 2010 17:46:24 GMT]]>Few additional questions:

Do you have a Windows firewall running on server machine?
Is there any error messages in netxmsd log?

Best regards,
Victor

]]>https://forum.netxms.org/post/4691https://forum.netxms.org/post/4691Mon, 15 Nov 2010 17:46:24 GMT<![CDATA[Reply to NetXMS Server Crashing on Mon, 15 Nov 2010 15:12:47 GMT]]>Still getting hundreds of agent timeouts per minute with negligible amounts of visible system resource utilization.

Also, even the console and command line nxadm -i timeouts are very common. Even when running the console locally.

For example the, "show queues" command at times never responds on the server. If I close it and run it a few more times it will eventually give results. According to the results nothing is queued.

The netxmsd process is at about 300mb and holding, the server has been running for at least 4 days now without a crash. It takes about 5-10 timeouts and retries with the console app to get it connected.

netxms_debug_1.JPG

netxms_debug_2.JPG

]]>https://forum.netxms.org/post/4689https://forum.netxms.org/post/4689Mon, 15 Nov 2010 15:12:47 GMT<![CDATA[Reply to NetXMS Server Crashing on Mon, 15 Nov 2010 13:49:26 GMT]]>If there are some socket related limitation that can be overcome by below guidelines
Start Registry Editor. Browse to, and then click the following key in the registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
On the Edit menu, click New, DWORD Value, and then add the following registry values

  1. Reduce the client TCP/IP socket connection timeout value from the default value of 240 seconds
    Value name: TcpTimedWaitDelay
    Value data: <Enter a decimal value between 30 and 240 here>

2.  Increase the number of ephemeral ports that can by dynamically allocated to clients:
Value name: MaxUserPort
Value data: <Enter a decimal value between 5000 and 65534 here>
 
Victor, Please consider using "SO_REUSEADDR" socket setting, by calling "setsockopt()"

Reference for all registry is from http://support.microsoft.com/default.aspx?scid=kb;en-us;314053
http://technet.microsoft.com/en-us/library/bb726981.aspx#EDAA

]]>https://forum.netxms.org/post/4688https://forum.netxms.org/post/4688Mon, 15 Nov 2010 13:49:26 GMT<![CDATA[Reply to NetXMS Server Crashing on Fri, 12 Nov 2010 18:49:41 GMT]]>292 total for port 4700, about 90 are in TIME_WAIT or SYN_SENT...

After a fresh reboot and a few minutes time, I see about 950 with 600 in TIME_WAIT.

]]>https://forum.netxms.org/post/4683https://forum.netxms.org/post/4683Fri, 12 Nov 2010 18:49:41 GMT<![CDATA[Reply to NetXMS Server Crashing on Fri, 12 Nov 2010 15:45:53 GMT]]>I suspect that it may hit some limits for open connections. There was a limit of around 4000 outgoing TCP connections in Windows 2003, I'm not sure how it is in Windows 2008. Could you please run netstat -n and count connections to external IPs on port 4700? And how many of them are not in ESTABLISHED state?

Best regards,
Victor

]]>https://forum.netxms.org/post/4681https://forum.netxms.org/post/4681Fri, 12 Nov 2010 15:45:53 GMT<![CDATA[Reply to NetXMS Server Crashing on Fri, 12 Nov 2010 15:20:33 GMT]]>Server has not crashed in almost 24 hours. The only remaning issue seems to be the connectivity loss of the agents dropping and restoring.

]]>https://forum.netxms.org/post/4679https://forum.netxms.org/post/4679Fri, 12 Nov 2010 15:20:33 GMT<![CDATA[Reply to NetXMS Server Crashing on Fri, 12 Nov 2010 08:24:05 GMT]]>Very random post... Please ensure that your network connectivity is stable. I experienced random NetXMS problem/behavior when network availability fluctuates.
Just give a try to install "Microsoft Loopback Adapter" from Add Hardware Wizard. Assign some Private IP (Other then your network). I'm sure there is no harm in installing loopback adapter

]]>https://forum.netxms.org/post/4675https://forum.netxms.org/post/4675Fri, 12 Nov 2010 08:24:05 GMT<![CDATA[Reply to NetXMS Server Crashing on Fri, 12 Nov 2010 00:01:26 GMT]]>After a couple minutes the event viewer started responding more quickly again..

The queues went to all zero.

And just as a heads up this is what my SQL server activity is looking like over the last while...

sqlstat.JPG

]]>https://forum.netxms.org/post/4671https://forum.netxms.org/post/4671Fri, 12 Nov 2010 00:01:26 GMT<![CDATA[Reply to NetXMS Server Crashing on Thu, 11 Nov 2010 23:53:11 GMT]]>Server responding very slowly now.. For example event log takes a minute to load up with a progress bar I had not seen before... The log is riddled with agent connection loss/restored messages. 50-100 clients are losing connection and restoring connections per minute.

The queues are currently,

netxmsd: show queues
Condition poller                : 0
Configuration poller            : 0
Data collector                  : 0
Database writer                  : 0
Event processor                  : 0
Network discovery poller        : 0
Node poller                      : 0
Routing table poller            : 0
Status poller                    : 132

netxmsd: show mutex
Mutex status:
  g_hMutexIdIndex: unlocked
  g_hMutexNodeIndex: unlocked
  g_hMutexSubnetIndex: unlocked
  g_hMutexInterfaceIndex: unlocked

netxmsd: show flags
Flags: 0x43300257
  AF_DAEMON                        = 1
  AF_USE_SYSLOG                    = 1
  AF_ENABLE_NETWORK_DISCOVERY      = 1
  AF_ACTIVE_NETWORK_DISCOVERY      = 0
  AF_LOG_SQL_ERRORS                = 1
  AF_DELETE_EMPTY_SUBNETS          = 0
  AF_ENABLE_SNMP_TRAPD            = 1
  AF_ENABLE_ZONING                = 0
  AF_SYNC_NODE_NAMES_WITH_DNS      = 0
  AF_CHECK_TRUSTED_NODES          = 1
  AF_WRITE_FULL_DUMP              = 0
  AF_RESOLVE_NODE_NAMES            = 1
  AF_CATCH_EXCEPTIONS              = 1
  AF_INTERNAL_CA                  = 0
  AF_DB_LOCKED                    = 1
  AF_ENABLE_MULTIPLE_DB_CONN      = 1
  AF_DB_CONNECTION_LOST            = 0
  AF_NO_NETWORK_CONNECTIVITY      = 0
  AF_EVENT_STORM_DETECTED          = 0
  AF_SERVER_INITIALIZED            = 1
  AF_SHUTDOWN                      = 0

The memory utilization is holding at about 121mb.

]]>https://forum.netxms.org/post/4670https://forum.netxms.org/post/4670Thu, 11 Nov 2010 23:53:11 GMT<![CDATA[Reply to NetXMS Server Crashing on Thu, 11 Nov 2010 22:14:25 GMT]]>The client connectivity loss/restore entries all still quite high. Dozens every few minutes, as shown by the attached event log.

netxms_debug.JPG

netxms_data_collection_graph.JPG

]]>https://forum.netxms.org/post/4669https://forum.netxms.org/post/4669Thu, 11 Nov 2010 22:14:25 GMT<![CDATA[Reply to NetXMS Server Crashing on Thu, 11 Nov 2010 21:11:46 GMT]]>Queues looks much better now. The only possibly problematic number is data collector's queue. Is it always like this, or it was just a short pike? Btw, by default server create DCIs for internal queue sizes. If those DCIs are configured, could you post a charts of them for last hour or two?

]]>https://forum.netxms.org/post/4667https://forum.netxms.org/post/4667Thu, 11 Nov 2010 21:11:46 GMT<![CDATA[Reply to NetXMS Server Crashing on Thu, 11 Nov 2010 20:45:04 GMT]]>I am noticing a tremendous amount of these disconnect and reconnects in the event log since I revised the settings. I attached screen shot. I put the status poller interval back down to 90 from 120 in case that is the cause.

memory is sitting at 110mb after being up for an hour or so..

queues look like this now,

netxmsd: show queues
Condition poller                 : 0
Configuration poller             : 523
Data collector                   : 5343
Database writer                  : 0
Event processor                  : 0
Network discovery poller         : 0
Node poller                      : 0
Routing table poller             : 0
Status poller                    : 190

netxms_debug.JPG

]]>https://forum.netxms.org/post/4665https://forum.netxms.org/post/4665Thu, 11 Nov 2010 20:45:04 GMT<![CDATA[Reply to NetXMS Server Crashing on Thu, 11 Nov 2010 19:16:58 GMT]]>Memory usage of the netxmsd before making all these changes and restarting was holding at about 130mb.

The database should be healthy, I'll watch the perfomance logs on that after these changes but the SQL server is low usage and on a higher performance segment of our SAN.

These suggestions,

StatusPollingInterval = 90
ConfigurationPollingInterval = 14400

I had already moved status interval to 90, so I bumped it to 120. The configuration I put at 14400 as you suggest and it was 3600.

Just restarted with the revised setting and removed the processor affinity. I'll check and post the stats in an hour.

]]>https://forum.netxms.org/post/4663https://forum.netxms.org/post/4663Thu, 11 Nov 2010 19:16:58 GMT<![CDATA[Reply to NetXMS Server Crashing on Thu, 11 Nov 2010 18:58:29 GMT]]>This hour stats..

netxmsd: show queues
Condition poller                : 0
Configuration poller            : 2978
Data collector                  : 0
Database writer                  : 181423
Event processor                  : 0
Network discovery poller        : 0
Node poller                      : 0
Routing table poller            : 0
Status poller                    : 2316

netxmsd: show mutex
Mutex status:
  g_hMutexIdIndex: unlocked
  g_hMutexNodeIndex: unlocked
  g_hMutexSubnetIndex: unlocked
  g_hMutexInterfaceIndex: unlocked

netxmsd: show flags
Flags: 0x43300257
  AF_DAEMON                        = 1
  AF_USE_SYSLOG                    = 1
  AF_ENABLE_NETWORK_DISCOVERY      = 1
  AF_ACTIVE_NETWORK_DISCOVERY      = 0
  AF_LOG_SQL_ERRORS                = 1
  AF_DELETE_EMPTY_SUBNETS          = 0
  AF_ENABLE_SNMP_TRAPD            = 1
  AF_ENABLE_ZONING                = 0
  AF_SYNC_NODE_NAMES_WITH_DNS      = 0
  AF_CHECK_TRUSTED_NODES          = 1
  AF_WRITE_FULL_DUMP              = 0
  AF_RESOLVE_NODE_NAMES            = 1
  AF_CATCH_EXCEPTIONS              = 1
  AF_INTERNAL_CA                  = 0
  AF_DB_LOCKED                    = 1
  AF_ENABLE_MULTIPLE_DB_CONN      = 1
  AF_DB_CONNECTION_LOST            = 0
  AF_NO_NETWORK_CONNECTIVITY      = 0
  AF_EVENT_STORM_DETECTED          = 0
  AF_SERVER_INITIALIZED            = 1
  AF_SHUTDOWN                      = 0

netxmsd: show stats
Total number of objects:    9309
Number of monitored nodes:  3649
Number of collectable DCIs:  8147

netxms_debug_1.JPG

]]>https://forum.netxms.org/post/4662https://forum.netxms.org/post/4662Thu, 11 Nov 2010 18:58:29 GMT<![CDATA[Reply to NetXMS Server Crashing on Thu, 11 Nov 2010 18:17:33 GMT]]>Few things I forgot:

  • Remove ProcessAffinityMask option from netxmsd.conf. It does not help with the crash issue, but prevents using of all CPUs in the system.

  • Some additional questions:

  1. How many memory consumes netxmsd process? Is it growing?
  2. What is the state of SQL server? So high value for database writer queue means that either NetXMS server cannot send requests as fast as needed or SQL server cannot process them as fast as needed. Increasing value for NumberOfDatabaseWriters will speed up request sending, but will SQL server be able to handle them?

Best regards,
Victor

]]>https://forum.netxms.org/post/4661https://forum.netxms.org/post/4661Thu, 11 Nov 2010 18:17:33 GMT<![CDATA[Reply to NetXMS Server Crashing on Thu, 11 Nov 2010 18:11:29 GMT]]>Hi!

Numbers are quite high, especially database writer queue. I would suggest to change some server parameters, taking into account number of nodes you have:

NumberOfStatusPollers = 100
NumberOfConfigurationPollers = 25
NumberOfRoutingTablePollers = 25

NumberOfDatabaseWriters = 5

I would also suggest to increase status and configuration polling intervals, which are one minute and one hour respectively. My recommended settings is to set configuration polling interval to at least 4 hours, and status polling interval to 90 seconds. It can be done by setting the following parameters:

StatusPollingInterval = 90
ConfigurationPollingInterval = 14400

Best regards,
Victor

]]>https://forum.netxms.org/post/4660https://forum.netxms.org/post/4660Thu, 11 Nov 2010 18:11:29 GMT