it seems that it is update of IP topology network map. I'll take a look at it.

Best regards,
Victor

Regards,
Edward

gdb_bt.log

when system will be on 100%, please attach to netxmsd process with gdb and send me result of command

thread apply all bt

Best regards,
Victor

Some news:
I have migrated my complete system from Oracle Linux to Debian 7.6
and guess what? the problem is not reproduced yet...

I will leave this new system running for now. see how it goes.
only one thing: I see netxmsd running 100% continue.
Maybe we have to look at that some time.

Regards,
Edward

I have upgraded my system from Oracle Linux 6.3 to 6.5
glibc unfortunately is not updated in the release.
Also netxmsd crashed after giving it a flood of alarms.

So, situation is not changed so far.
Regards,
Edward

so far our analysis shows that root of the problem is that some internal data structures are not initialized properly (if you are familiar with C++ - it seems that constructor for global instance of AlarmManager class is not called), although it is not clear why this happens. I try your binaries on CentOS 6.3 (closest that I can find) and it works as expected. Is it an option to do system upgrade to latest versions of kernel and glibc and try again? If this will not help, I'll re-write alarm manager initialization and provide you with intermediate build for testing.

Best regards,
Victor

I have uploaded the following files:
core.tgz
server-binaries.tgz

do you need anything else?
regards,
Edward

Best regards,
Edward

Best regards,
Victor

Thanks for the reply!

I'm currently running it under GDB, so as soon as I have a crash I will generate a core dump.
Is that ok for you if I do it this way?

Another question:
Is is possible to fake some node down events for existing nodes?
That way I can try to reproduce the crash.

Regards,
Edward

it seems one of the most mysterious bugs we've ever encounter. I've done load tests with lot of alarm generation and termination, and system works as expected. Seems it is some rare combination of events and/or environment. Is it possible to get remote access to your system to debug it in place? Alternatively, can you try to get core dump after crash (most likely it will note be generated by default, you'll have to enable it with ulimit) and sent to us along with compiled binaries? If you run it as virtual machine and can provide a VM image it also could help us with debugging.

Best regards,
Victor

debug.log

Thanks for responding!
attached is the output for the disassemble.

Maybe good to know that I found out that the crash always occur strait after a new alarm.
Not after each alarm, but after running a couple of hours (I think after a couple of new alarms)
Also it looks that it is related to a critical event. (node down or so)
I'm trying to reproduce the crash, but that is really difficult.

Today we have had not many new alarms, and it is running all day now.
I expect that as soon as start some maintenance (rebooting machines etc) we will have a crash again.

Hope this helps.
Regards,
Edward

disassemble.log

Could you please also provide us with disassembly of two methods: AlarmManager::watchdogThread and AlarmManager::newAlarm?

This can be done with gdb:

$ gdb /opt/netxms/bin/netxmsd
(gdb) info functions AlarmManager::watchdogThread
(gdb) disassemble AlarmManager::watchdogThread
(gdb) disassemble AlarmManager::newAlarm

I'm open for any ideas to speed up the resolve of this issue.

include some more info in the code?
reinstall system? move to other OS?

Thanks,
Edward

Here is the full log.
It is huge, so I have compressed it.

Hope this helps.

valgrind.tgz

can you please provide full valgrind log (assuming there are more records then you post already)? Those errors logged looks like consequences of some memory corruption that happened before.

Best regards,
Victor

Any progress on this?
Server is still crashing several times...

Would it be an option to downgrade to 1.2.14?
This was our previous version.

Thanks,
Edward

==6816== Thread 66:
==6816== Conditional jump or move depends on uninitialised value(s)
==6816==    at 0x4CC549E: ClientSession::onAlarmUpdate(unsigned int, NXC_ALARM*) (session.cpp:4989)
==6816==    by 0x4C631ED: EnumerateClientSessions(void (*)(ClientSession*, void*), void*) (client.cpp:337)
==6816==    by 0x4C5F278: AlarmManager::newAlarm(char*, char*, int, int, unsigned int, unsigned int, Event*, unsigned int) (alarm.cpp:384)
==6816==    by 0x4C81704: EPRule::generateAlarm(Event*) (epp.cpp:531)
==6816==    by 0x4C81D4A: EPRule::processEvent(Event*) (epp.cpp:468)
==6816==    by 0x4C81F4B: EventPolicy::processEvent(Event*) (epp.cpp:814)
==6816==    by 0x4C862C9: EventProcessor(void*) (evproc.cpp:225)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)

Hope this helps finding the crash cause.

==6816== Thread 7:
==6816== Invalid read of size 4
==6816==    at 0x4C5CDD0: AlarmManager::watchdogThread() (alarm.cpp:1105)
==6816==    by 0x4C5CF28: WatchdogThreadStarter(void*) (alarm.cpp:204)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)
==6816==  Address 0x2c8c57bc is 52,620 bytes inside a block of size 243,984 free'd
==6816==    at 0x4A0610F: realloc (vg_replace_malloc.c:525)
==6816==    by 0x4C5EF8A: AlarmManager::newAlarm(char*, char*, int, int, unsigned int, unsigned int, Event*, unsigned int) (alarm.cpp:359)
==6816==    by 0x4C81704: EPRule::generateAlarm(Event*) (epp.cpp:531)
==6816==    by 0x4C81D4A: EPRule::processEvent(Event*) (epp.cpp:468)
==6816==    by 0x4C81F4B: EventPolicy::processEvent(Event*) (epp.cpp:814)
==6816==    by 0x4C862C9: EventProcessor(void*) (evproc.cpp:225)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)

you mean that you've found some specific place in added code which caused this crash?

Best regards,
Victor