Out of memory NetXMS v1.2.5

[millerpaint]

Hi,

One thing seems strange, it crashes with out of memory right away running under valgrind.  If I run netxmsd normally, it can run for hours before crashing.

-Kevin C.

[millerpaint]

OK, good news Alex.  The script you provided shut down NetXMS gracefully after it reached the 1GB of RAM threshold.  It took 12-15 minutes running under valgrind before it crashed.  I have attached the valgrind log.

Thanks,

-Kevin C.

---

netxmsd-valgrind.log

[victor]

Hi!

@millerpaint said in Out of memory NetXMS v1.2.5:

One thing seems strange, it crashes with out of memory right away running under valgrind.  If I run netxmsd normally, it can run for hours before crashing.

This is normal, when running under valgrind program takes tens times more memory then when it run normally. Valgrind allocates extra memory around each dynamically allocated block to detect boundary violations, etc.

Best regards,
Victor

[millerpaint]

This is normal, when running under valgrind program takes tens times more memory then when it run normally. Valgrind allocates extra memory around each dynamically allocated block to detect boundary violations, etc.

OK, that makes sense Victor.

I have attached a screenshot of my Network Discovery panel, so you can see more about the details of my configuration.  I also have (2) SNMP community strings listed, but they are not visible in the screenshot image.

I am using top to monitor the memory consumption of netxmsd - it seems to be consuming 1/10th of 1% of available RAM every few seconds, running in normal mode.

-Kevin C.

---

netdiscovery.pdf

[alk]

Unfortunately, there nothing related to your issue in this log.
Could you please change 1Gb limit to 2Gb in the script and rerun it?

@millerpaint said in Out of memory NetXMS v1.2.5:

OK, good news Alex.  The script you provided shut down NetXMS gracefully after it reached the 1GB of RAM threshold.  It took 12-15 minutes running under valgrind before it crashed.  I have attached the valgrind log.

[millerpaint]

Hi Alex,

I modified the script for a 2GB threshold and re-ran it, the new log file is attached.  Hopefully this will provide some clues as to what is going on.

When monitoring with top, it seems to start consuming RAM when the timer reaches 14:41:

netxmsd starts out using .5% available RAM of server.  Then:
14:41 - .6%
15:50 - .7%
16:53 - .8%
17:69 - .9%
18:60 - 1%
19:20 - 1.1%
20:05 - 1.2%
etc.

Thanks for your help!

-Kevin C.

---

netxmsd-valgrind.log

[millerpaint]

FYI, I have completely disabled auto-discovery, and it is still running out of memory.

-Kevin C.

[victor]

Hi!

We are still trying to figure out what could cause such a high memory consumption. Can you please do the following:

1. Run netxmsd under valgrind with additional options:

valgrind --leak-check=full --undef-value-errors=no --show-reachable=yes --log-file=netxmsd-valgrind.log netxmsd -D3

(or modify script sent by Alex by adding --show-reachable=yes to valgrind's command line).

2. Run valgrind's heap profiller:

valgrind --tool=massif --time-unit=ms --stacks=yes --threshold=0.5 --max-snapshots=1000 --log-file=netxmsd-massif.log netxmsd -D3

and send me profiler's result (it will be named massif.out.<pid>).

Also, can you please show me you configuration hook script and transformation scripts?

Best regards,
Victor

[millerpaint]

Hi Victor,

OK, I can do step 1 additional options with no problem.

With Step 2, you are asking me to run valgrind's heap profiler.  I have questions on that:

1. Is step 2 option run after I have completed running the step 1 test, and it runs out of memory?
2. Do I need to start Alex's script before running step 2?

-Kevin C.

[victor]

Hi!

Yes, you should run step 2 after step 1 is completed. You can use Alex's script for step 2 too.

Best regards,
Victor

[millerpaint]

Hi Victor,

I am unable to attach the valgrind log from Step 1, it is about 600k in size, and your forum will not allow me to post it.

Can you please raise the limit of your attachment size on this forum, or else let me know your email address?

Thanks,

-Kevin C.

[victor]

You can send it to [email protected].

Best regards,
Victor

[millerpaint]

OK thanks - all log files you have requested from valgrind have been sent to that email address.

-Kevin C.