NetXMS Forum

looks this is related to hosts where we have agents configured.
snmp data is collected fine.

After upgrading to 2.0M5 no data is collected anymore.
the timestamp for the values are all set to 01.01.1970 01:00:00

What could be the problem?

Thanks Edward.

Hi, After the upgrade to 2.0M5 we see extra interfaces being discovered.
Is there a way to disable this?
The interfaces are:
Loopback Pseudo-Interface
isatap.domain

Regards,
Edward

I was able to solve this error by setting
export CPPFLAGS="-march=i486"

before the ./configure

looks the default for xenserver is set to i386.....

Hi,

I have the following error when compiling the agent 1.2.16 on Xenserver:

session.o: In function `InterlockedDecrement':
/mnt/netxms/1.2.16/netxms-1.2.16/src/agent/core/../../../include/nms_threads.h:1023: undefined reference to `__sync_sub_and_fetch_4'
/mnt/netxms/1.2.16/netxms-1.2.16/src/agent/core/../../../include/nms_threads.h:1023: undefined reference to `__sync_sub_and_fetch_4'
/mnt/netxms/1.2.16/netxms-1.2.16/src/agent/core/../../../include/nms_threads.h:1023: undefined reference to `__sync_sub_and_fetch_4'
collect2: ld returned 1 exit status
make[4]: *** [nxagentd] Error 1
make[4]: Leaving directory `/mnt/netxms/1.2.16/netxms-1.2.16/src/agent/core'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/mnt/netxms/1.2.16/netxms-1.2.16/src/agent'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/mnt/netxms/1.2.16/netxms-1.2.16/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/mnt/netxms/1.2.16/netxms-1.2.16'
make: *** [all] Error 2

Any idea on how to solve this?
Thanks,

Edward

Attached is the log from gdb.
look for thread 9332, because it is this one which is currently running at 100%

Regards,
Edward

gdb_bt.log

Hi,

Some news:
I have migrated my complete system from Oracle Linux to Debian 7.6
and guess what? the problem is not reproduced yet...

I will leave this new system running for now. see how it goes.
only one thing: I see netxmsd running 100% continue.
Maybe we have to look at that some time.

Regards,
Edward

Hi Victor,

I have upgraded my system from Oracle Linux 6.3 to 6.5
glibc unfortunately is not updated in the release.
Also netxmsd crashed after giving it a flood of alarms.

So, situation is not changed so far.
Regards,
Edward

I have uploaded a new core file.
This one is generated by abrt
regards,
Edward

Thanks,

I have uploaded the following files:
core.tgz
server-binaries.tgz

do you need anything else?
regards,
Edward

I have a core dump generated.
How would you like me to send/upload it?
it is compressed 9M. (only the core file)
comressed tar from the binaries is 70M.

Best regards,
Edward

Hi Victor,

Thanks for the reply!

I'm currently running it under GDB, so as soon as I have a crash I will generate a core dump.
Is that ok for you if I do it this way?

Another question:
Is is possible to fake some node down events for existing nodes?
That way I can try to reproduce the crash.

Regards,
Edward

looks like you have had some disk problem.
maybe no diskspace or so.

you can try to repair this table (and probably other tables as well) by running:

mysqlcheck --auto-repair <database name> -u root -p

give the password for mysql root user.
This will check all the tables in de database and try to fix them.

attached is a debug log (level where you can see a new alarm coming in.
this was a node down alarm. direct after this we got the sigsegv.

debug.log

Hi Alex,

Thanks for responding!
attached is the output for the disassemble.

Maybe good to know that I found out that the crash always occur strait after a new alarm.
Not after each alarm, but after running a couple of hours (I think after a couple of new alarms)
Also it looks that it is related to a critical event. (node down or so)
I'm trying to reproduce the crash, but that is really difficult.

Today we have had not many new alarms, and it is running all day now.
I expect that as soon as start some maintenance (rebooting machines etc) we will have a crash again.

Hope this helps.
Regards,
Edward

disassemble.log

Hi mstjohn1974,

Please read https://www.netxms.org/documentation/adminguide/
This will give you a basic understanding on how netxms works.
It also tells you how to monitor and how to configure it.

Regards,
Edward

any news on this?

I'm open for any ideas to speed up the resolve of this issue.

include some more info in the code?
reinstall system? move to other OS?

Thanks,
Edward

Hi,

Here is the full log.
It is huge, so I have compressed it.

Hope this helps.

valgrind.tgz

Hello,

Any progress on this?
Server is still crashing several times...

Would it be an option to downgrade to 1.2.14?
This was our previous version.

Thanks,
Edward

and here is another one:

==6816== Thread 66:
==6816== Conditional jump or move depends on uninitialised value(s)
==6816==    at 0x4CC549E: ClientSession::onAlarmUpdate(unsigned int, NXC_ALARM*) (session.cpp:4989)
==6816==    by 0x4C631ED: EnumerateClientSessions(void (*)(ClientSession*, void*), void*) (client.cpp:337)
==6816==    by 0x4C5F278: AlarmManager::newAlarm(char*, char*, int, int, unsigned int, unsigned int, Event*, unsigned int) (alarm.cpp:384)
==6816==    by 0x4C81704: EPRule::generateAlarm(Event*) (epp.cpp:531)
==6816==    by 0x4C81D4A: EPRule::processEvent(Event*) (epp.cpp:468)
==6816==    by 0x4C81F4B: EventPolicy::processEvent(Event*) (epp.cpp:814)
==6816==    by 0x4C862C9: EventProcessor(void*) (evproc.cpp:225)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)

Hope this helps finding the crash cause.