Netxms-Server crashes after tuning

[Zebble]

We had a similar challenge with the poller queue never "catching up" and would continue to grow.

Victor took a closer look.  Turning on parallel processing and increasing the Discovery.BaseSize from 1 to 8 seems to have helped immensely.  We haven't had any related problems since.

Not sure if this will help your situation.

-zeb

[victor]

Hi,

most likely you have hit RAM limit when allow thread pool to grow to 4000 threads. Could also be other memory-related issue (memory leak in server, slow database causing DB writer queues to grow, etc. Do you have pre-crash data on number of threads, queues, and memory consumption?

Also, 4000 nodes is not that much, it is quite strange that 500 threads is not enough. Such high load on poller pool indicates that there are lot of polls taking long time to complete.  Could it be that you have many nodes that timeout when polled?

Best regards,
Victor

[Sack-C-Fix]

Hello,

unfortunately, it took some time until I could deal with the topic again.
The memory usage looks OK to me, how do you recognise a slow database? At least I don't see anything in the associated logs.

Attached are a few graphics, on 11 March at 18:00 I started the server with tuned values. During the night the service stops several times, but is restarted by corosync. In the morning of 12 March, I restore the default values.

Are there any values that I should look at more closely? Or which I should monitor before the next tuning attempt? Unfortunately, the tuning topic is described somewhat briefly in the documentation, and the designations no longer seem to be correct.

Andi

---

[victor]

Hi,

problem is definitely with pollers pool usage, rest looks fine. I would suggest gradually increasing size of pollers pool, probably start with 750/1500 from current 250/500.

Do you use hook scripts?

Do you have routers with huge routing tables? If yes, you could try to disable routing table poll for such devices.

If netxmsd is crashing, does system generates core file? If sould be really helpful for debugging the issue.

Best regards,
Victor

[Sack-C-Fix]

Thanks Victor,

I will test the suggested values.

In fact, we have some adjustements in Hook::ConfigurationPoll. We set the interface expected state there depending on the name and CDP/LLDP. Is this a problem?

We also have a core network with large routing tables, but these are already deactivated. But it would be nice to be able to use them.

Core dump is not active yet, but I will switch it on. I can also try running the service with dbg to debug the crash.

Many thanks once again
Andi

[Sack-C-Fix]

Hello everybody,

I have now tried several times to tune the server, but I can't get any further.
After some time, the service stops and restarts, so a practical use is not possible.

I now have a few core files, should I upload them?

Thanks

[victor]

Hi,

please use this link for upload: https://cloud.radensolutions.com/s/88sd5KwD2SaAewE

Best regards,
Victor

[Sack-C-Fix]

Hi Victor,

I just uploaded an actual core-dump, name is "core_20210523".

Thanks

[Sack-C-Fix]

Hello,

today the server crashed again for no apparent reason. The only explanation would be that a vMotion took place in ESX 30 minutes before.

Could this be the problem?

Thanks

[Sack-C-Fix]

Okay,

unfortunately, the server can no longer be started, even with default values.
Even when I load a dump on the test server and try it there, the server stops after a few minutes.

Is there a possibility to get support for this? We had planned to buy support for next year, maybe we can buy it earlier.

Thanks

[victor]

Hello,

yes, it's possible. I can take a look tomorrow at your server. Will send you PM with contact details.

Best regards,
Victor