segfault netxmsd crash

edward.borst

here it is:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffef576700 (LWP 19648)]
0x00007ffff7d04da7 in AlarmManager::watchdogThread (this=0x7ffff7ff7740) at alarm.cpp:1090
1090 if ((m_pAlarmList[i].dwTimeout > 0) &&
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6_3.6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6.x86_64 libcom_err-1.41.12-14.el6.x86_64 libgcc-4.4.6-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 mysql-libs-5.1.66-2.el6_3.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 openssl-1.0.0-27.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007ffff7d04da7 in AlarmManager::watchdogThread (this=0x7ffff7ff7740) at alarm.cpp:1090
#1 0x00007ffff7d04f29 in WatchdogThreadStarter (pArg=<value optimized out>) at alarm.cpp:204
#2 0x000000313d407851 in start_thread () from /lib64/libpthread.so.0
#3 0x000000313cce811d in clone () from /lib64/libc.so.6

regards,
Edward

edward.borst

Hi Victor,

Did you have a chance to look at the trace?
it keeps crashing several times a day....

Alex Kirhenshtein

Hello.

Yes, we spent some time on it, but unfortunately can't find root of this problem. Can you please run server under valgrind, until crash?

edward.borst

of course we can.
any options you want to give valgrind to check?

edward.borst

it looks like this issue is coming from pre 1.2.16 alarms.
We had about 6 old alarms with a sticky acknowledge set on it.
I was looking at the alarm.cpp code to see if I could find anything.
I saw that piece of code was added last release.
So, I deleted(terminated) all of the old alarms and we did not have a crash yet.
We have had new alarms coming in with no error.

regards,
Edward

Victor Kirhenshtein

Hi!

you mean that you've found some specific place in added code which caused this crash?

Best regards,
Victor

edward.borst

Hello,
Unfortunately it crashed again. Same error
I'm running it now under valgrind, but that takes a lot of resources. I run it with default settings
Did not have a crash yet

edward.borst

Here is a part from an error from Valgrind:

==6816== Thread 7:
==6816== Invalid read of size 4
==6816==    at 0x4C5CDD0: AlarmManager::watchdogThread() (alarm.cpp:1105)
==6816==    by 0x4C5CF28: WatchdogThreadStarter(void*) (alarm.cpp:204)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)
==6816==  Address 0x2c8c57bc is 52,620 bytes inside a block of size 243,984 free'd
==6816==    at 0x4A0610F: realloc (vg_replace_malloc.c:525)
==6816==    by 0x4C5EF8A: AlarmManager::newAlarm(char*, char*, int, int, unsigned int, unsigned int, Event*, unsigned int) (alarm.cpp:359)
==6816==    by 0x4C81704: EPRule::generateAlarm(Event*) (epp.cpp:531)
==6816==    by 0x4C81D4A: EPRule::processEvent(Event*) (epp.cpp:468)
==6816==    by 0x4C81F4B: EventPolicy::processEvent(Event*) (epp.cpp:814)
==6816==    by 0x4C862C9: EventProcessor(void*) (evproc.cpp:225)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)

edward.borst

and here is another one:

==6816== Thread 66:
==6816== Conditional jump or move depends on uninitialised value(s)
==6816==    at 0x4CC549E: ClientSession::onAlarmUpdate(unsigned int, NXC_ALARM*) (session.cpp:4989)
==6816==    by 0x4C631ED: EnumerateClientSessions(void (*)(ClientSession*, void*), void*) (client.cpp:337)
==6816==    by 0x4C5F278: AlarmManager::newAlarm(char*, char*, int, int, unsigned int, unsigned int, Event*, unsigned int) (alarm.cpp:384)
==6816==    by 0x4C81704: EPRule::generateAlarm(Event*) (epp.cpp:531)
==6816==    by 0x4C81D4A: EPRule::processEvent(Event*) (epp.cpp:468)
==6816==    by 0x4C81F4B: EventPolicy::processEvent(Event*) (epp.cpp:814)
==6816==    by 0x4C862C9: EventProcessor(void*) (evproc.cpp:225)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)

Hope this helps finding the crash cause.

edward.borst

Hello,

Any progress on this?
Server is still crashing several times...

Would it be an option to downgrade to 1.2.14?
This was our previous version.

Thanks,
Edward

Victor Kirhenshtein

Hi!

can you please provide full valgrind log (assuming there are more records then you post already)? Those errors logged looks like consequences of some memory corruption that happened before.

Best regards,
Victor

edward.borst

Hi,

Here is the full log.
It is huge, so I have compressed it.

Hope this helps.

valgrind.tgz

edward.borst

any news on this?

I'm open for any ideas to speed up the resolve of this issue.

include some more info in the code?
reinstall system? move to other OS?

Thanks,
Edward

Alex Kirhenshtein

Hello.

Could you please also provide us with disassembly of two methods: AlarmManager::watchdogThread and AlarmManager::newAlarm?

This can be done with gdb:

$ gdb /opt/netxms/bin/netxmsd
(gdb) info functions AlarmManager::watchdogThread
(gdb) disassemble AlarmManager::watchdogThread
(gdb) disassemble AlarmManager::newAlarm

edward.borst

Hi Alex,

Thanks for responding!
attached is the output for the disassemble.

Maybe good to know that I found out that the crash always occur strait after a new alarm.
Not after each alarm, but after running a couple of hours (I think after a couple of new alarms)
Also it looks that it is related to a critical event. (node down or so)
I'm trying to reproduce the crash, but that is really difficult.

Today we have had not many new alarms, and it is running all day now.
I expect that as soon as start some maintenance (rebooting machines etc) we will have a crash again.

Hope this helps.
Regards,
Edward

disassemble.log

edward.borst

attached is a debug log (level where you can see a new alarm coming in.
this was a node down alarm. direct after this we got the sigsegv.

debug.log

Victor Kirhenshtein

Hi,

it seems one of the most mysterious bugs we've ever encounter. I've done load tests with lot of alarm generation and termination, and system works as expected. Seems it is some rare combination of events and/or environment. Is it possible to get remote access to your system to debug it in place? Alternatively, can you try to get core dump after crash (most likely it will note be generated by default, you'll have to enable it with ulimit) and sent to us along with compiled binaries? If you run it as virtual machine and can provide a VM image it also could help us with debugging.

Best regards,
Victor

edward.borst

Hi Victor,

Thanks for the reply!

I'm currently running it under GDB, so as soon as I have a crash I will generate a core dump.
Is that ok for you if I do it this way?

Another question:
Is is possible to fake some node down events for existing nodes?
That way I can try to reproduce the crash.

Regards,
Edward

Victor Kirhenshtein

You can gdnerate events using nxevent command line tool.

Best regards,
Victor

edward.borst

I have a core dump generated.
How would you like me to send/upload it?
it is compressed 9M. (only the core file)
comressed tar from the binaries is 70M.

Best regards,
Edward

NetXMS Forum

segfault netxmsd crash