daily server crash - Thread "Item Poller" does not respond to watchdog thread

[tomaskir]

Definitely upgrade to 2.0.2.
There were many fixes between 2.0-RC2 and 2.0.2.

If the problem still persists on 2.0.2, let us know

[Mark.Ouwendijk]

Ok thanks for the advice and support.

I will upgrade to version 2.0.2 and then further evaluate if this issue is resolved.

Again thanks for your help.

[Mark.Ouwendijk]

Yesterday i successfully upgraded to 2.0.2 at around 14:00h.

Unfortunately this issue, "daily server crash - Thread "Item Poller" does not respond to watchdog thread", occurred again 5 hours after the successful upgrade to version 2.0.2.

* application and services still running, but no dci polling values
* Error viewing last values for each node, screenshot attached in this post.
* empty dashboard graphs

Any advice in fixing  this issue is more than welcome.

---

[victor]

Hi,

can you please add the following to netxmsd.conf:

DebugLevel = 7
CreateCrashDumps = yes

as well as ensure that LogFile points to some file, not {syslog}, and restart it. When server hangs, run

nxadm -c "raise access"

server process will crash and dump will be generated. Send dump file and log file to us for analyze.

Best regards,
Victor

[Mark.Ouwendijk]

Followed instructions.

Server is running again. Waiting for the next crash. Then i will run : nxadm -c "raise access"

I will sent log and dumpfiles after.

[Mark.Ouwendijk]

Server crashed again, graphs empty and no values. NetXMS core service still appears to be running.

Executed command line : nxadm -c "raise access"
dump files were created

Attached in this reply the most recent (zipped) log file.
Also attached in this reply the 2 dump files.

Awaiting your response.
Thanks in advance.
Kind regards,
Mark

---

log.7z

netxmsd-5608-1455526480.info

netxmsd-5608-1455526480.mdmp

[Mark.Ouwendijk]

Server crashed again, graphs empty and no values. NetXMS core service still appears to be running.

Executed command line : nxadm -c "raise access"
dump files were created

Attached in this reply the most recent (zipped) log file.
Also attached in this reply the 2 dump files.

Hope the team can solve this issue with the new information (logs and dumpfiles)

Thanks in advance.
Kind regards,
Mark

---

log.7z

netxmsd-5608-1455526480.info

netxmsd-5608-1455526480.mdmp

[Mark.Ouwendijk]

From a more recent crash attached in this reply :  (zipped)log file and dump files

Hope this addition to the troubleshooting log and dump files can help in solving the issue.

---

log.7z

netxmsd-5676-1455781950.info

netxmsd-5676-1455781950.mdmp

[Mark.Ouwendijk]

Another crash.

More log and dump files attached in this reply :  (zipped)log file and dump files

Again I hope this addition to the troubleshooting log and dump files can help in solving the issue.

Thanks in advance.

Kind regards,

Mark

---

log.7z

netxmsd-4432-1455799238.info

netxmsd-4432-1455799238.mdmp

[Mark.Ouwendijk]

Different location. Different network. Different NetXMS server, (software new version 2.0.2)

same issue : daily server crash - Thread "Item Poller" does not respond to watchdog thread, empty graphs, no polling values

created DUMP file and log files. Attached in this reply.

Hope you can find some leads within the new information regarding this issue.

---

log.txt.7z

netxmsd-1756-1456132912.info

netxmsd-1756-1456132912.mdmp

[Mark.Ouwendijk]

This issue has not been solved.

Have done some new troubleshooting.

-MySQL logging enabled and reviewed logging.
Maybe someone can tell me what to look for besides the usual error search.

-Changed some server configuration parameters, for example in one troubleshooting 'run':  increased the StatusPollingInterval from 60 seconds to 300 seconds.
and in another troubleshooting 'run' : increased PollerThreadPoolMaxSize from 250 to 500, PollerThreadPoolBaseSize from 10 to 20 and NumberOfDataCollectors from 25 to 50.

Unfortunately until now the server keeps on 'crashing' daily.

Hope there is a solution for solving this issue.

[Mark.Ouwendijk]

Still crashing.

Added some screenshots in this reply post of situation after the issue has occurred.

---

[Mark.Ouwendijk]

FYI
For one location i configured the following settings on a NetXMS server:

Statuspollerinterval          600
DefaultDCIPollingInterval  600
ConditionPollingInterval    600

Although there are unrealistic values for daily operations the server has been running for more than one day now without crashing.

[Mark.Ouwendijk]

Unfortunately the configuration mentioned in my last post (Statuspollerinterval 600, DefaultDCIPollingInterval 600, ConditionPollingInterval 600) did not last.

Server just logged the error "Thread "Item Poller" does not respond to watchdog thread" and does not poll new values anymore and displays empty graphs again.

[Mark.Ouwendijk]

This issue has been solved by installing the pre-release of the next version.
NetXMS servers have been running for more than a week without any form of crashing.

Thanks Victor and others for your support with this matter.