Server crash

xenth

I just noticed something very disturbing, the polling isn't going properly.

My set intervals aren't being listened to, I'll give you an example, this is one of my workstations.
I have set the interval to 60 seconds, yet it does NOT poll every 60 seconds

It should poll every 60 seconds, yet when I view the history:

05-May-2008 07:13:29 527216640
05-May-2008 07:09:30 527216640
05-May-2008 07:05:48 527216640
05-May-2008 07:01:57 527216640
05-May-2008 06:58:12 527216640
05-May-2008 06:54:25 527216640
05-May-2008 06:50:26 527216640
05-May-2008 06:46:32 527216640
05-May-2008 06:42:42 527216640
05-May-2008 06:38:50 527216640

This is a VERY big problem as well, is it related to my other problem?

Alex Kirhenshtein

Thanks for the dump, I'm checking it.

xenth

Great!

Victor Kirhenshtein

@xenth said in Server crash:

I just noticed something very disturbing, the polling isn't going properly.

My set intervals aren't being listened to, I'll give you an example, this is one of my workstations.
I have set the interval to 60 seconds, yet it does NOT poll every 60 seconds

It should poll every 60 seconds, yet when I view the history:

This is a VERY big problem as well, is it related to my other problem?

Yes, most likely they are related. If you take a look at a result of "show queues" console command, you can see quite big number (330) in data collector queue. This means that at the moment when you type this command 330 requests for collecting data was waiting for processing, because all data collectors was busy. You can try to increase number of data collectors to 40 or 50 (by changing server's parameter NumberOfDataCollectors), this may help a bit, but will not remove the problem completely.

Some additional questions: do you have SNMP on the nodes? Do you use SNMP for data collection?

Best regards,
Victor

xenth

Hi,

I set the Numberofdatacollectors to 90 and the data collector queue is now at 150ish, should I keep increasing the number of collectors?

Some of the workstations have windows XP on them with the SNMP service but I am not running any dci's for those,
I am however monitoring several routers/modems/etc with snmp.

To be more precise: About 7 or 8 nodes with SNMP that I am monitoring with an average of about 6-7 dci's per SNMP node.

Thank you for your time.

Victor Kirhenshtein

Hi!

Further increase of data collectors will not help - we have similar problem at one of our customer's site, and it's related to SNMP. I'll publish updated version of NetXMS server today, it may help.

Best regards,
Victor

xenth

Great! I can't wait to test it

Queue problem is fixed now anyway, I'll post another topic with some questions regarding it.

xenth

Hi, small update.

I set the "numberofdatacollectors" at 100 now and everything is running as smooth as ever, 0 in the queues
Every problem I had with crashing and timeouts appear to be gone.

Victor Kirhenshtein

Very interesting information... Thank you for reporting!

Best regards,
Victor

xenth

You're welcome, all is not solved unfortunately

On all the workstations I am monitoring I have disabled status polling, personally I think that's where the problem is coming from.

When everyone is working on the machines (and they are reachable by netxms) everything is fine with the current settings.

However at the moment (it's 20:00 here) I am seeing a queue of around 100 for the datacollector pollers and the same problems are happening again, I can't delete anything, timeouts and I have to reboot the netxms service.

You see what's happening is this:

08:00 - 17:00 - Server requests data from agents and gets replies - no queues, no issues.
17:00 - 08:00 - Server requests data from agents and the packets are blocked by the firewall because the destination adres does not exist at that moment - queues and lots of issues.

-UPDATE: Here's the routers log message for when it tries to poll offline nodes if it helps:

	 Exceed MAX incomplete, sent TCP RST

Putting my workstations back on status polling is not an option because they go down so frequently

Victor Kirhenshtein

It can be a problem for NetXMS, because if during status poll it determines that node is down, data collection will not be scheduled for that node. But, if there are no status polls, NetXMS will try to establish TCP connection for each parameter to be collected. With connection timeout about 1 minute, it can easily cause long data collection queues.

What is the reason for disabling status polls? If you don't want alarms, mails, etc. for SYS_NODE_DOWN events, you can just block them in the first rule of event processing policy.

Best regards,
Victor

xenth

Hi

That makes a lot of sense.

But, if I block those specific nodes from the policy that will only block them from my alarm list (which is what I want) but they will still be marked with a red dot in front of them, and in the network summary they will be displayed as critical nodes.

This is the specific reason I've blocked status polling on those nodes.

Basically what I wanted to achieve:
Only important nodes need to be marked with a red dot and displayed as critical in the network summary, is there any other way to achieve this then?

Thanks again for all your work, I appreciate it.

Victor Kirhenshtein

Node status calculated based on two sources - active alarms for that node and status of child objects (usually interfaces). If you wish that node remains in NORMAL status event if some or all interfaces goes down, you should change status propagation algorithm for child objects of that node. You can select "fixed value" propagation algorithm for interface objects and set value to NORMAL.

Best regards,
Victor

xenth

Great! This appears to be working perfectly

NetXMS Forum

Server crash