Memory peak usage at startup, till swap begins

madmucho

Dear support team,
i observing strange behavior in times when netxms service starting, maibe service trying to cache some data from database, but this takes a lot of memory and if memory goes empty system begins to swap some process out and in this time netxms process release a lot of memory and then starts and runs ok...its strange.

My environment is debian 10, database is mariadb.

Server version 3.8.267-1

Monitored nodes 297
Collectible DCIs 22487
Active alarms 180012

Mariadb has 60Gb of data.

System has 2vCPU 10GB RAM, usually memory is at 75 percent and everything runs ok, I see problems only at startup.

You can see a few minutes delay in startup log, in this time memory usage goes to swap, and then drops to normal.

2021.04.27 16:14:32.298 *I* [logger             ] Log file opened (rotation policy 2, max size 16777216)
2021.04.27 16:14:32.298 *D* [                   ] LIB directory set to /usr/lib/x86_64-linux-gnu/netxms
2021.04.27 16:14:32.298 *I* [                   ] Code page set to UTF8
2021.04.27 16:14:32.301 *I* [                   ] Platform subagent "/usr/lib/x86_64-linux-gnu/netxms/linux.nsm" successfully loaded
2021.04.27 16:14:32.302 *I* [db.drv             ] Database driver "mariadb.ddr" loaded and initialized successfully
2021.04.27 16:14:32.303 *D* [                   ] Local administration interface listener started
2021.04.27 16:14:32.303 *D* [                   ] Successfully connected to database [email protected]
2021.04.27 16:14:32.312 *D* [db.cpool           ] Database Connection Pool initialized
2021.04.27 16:14:32.312 *D* [db.cpool           ] Database Connection Pool maintenance thread started
2021.04.27 16:14:32.312 *I* [                   ] Server ID 3007CF4CB43E2255
2021.04.27 16:14:32.316 *D* [db.lock            ] Database lock set
2021.04.27 16:14:32.322 *I* [db.writer          ] DBWriter/Housekeeper interlock is OFF
2021.04.27 16:14:32.322 *D* [core               ] Data directory set to /var/lib/netxms
2021.04.27 16:14:32.323 *D* [                   ] Global configuration loaded
2021.04.27 16:14:32.325 *D* [crypto             ] Validating ciphers
2021.04.27 16:14:32.325 *D* [crypto             ]    AES-256 enabled
2021.04.27 16:14:32.325 *D* [crypto             ]    Blowfish-256 enabled
2021.04.27 16:14:32.325 *D* [crypto             ]    IDEA disabled (config)
2021.04.27 16:14:32.325 *D* [crypto             ]    3DES enabled
2021.04.27 16:14:32.325 *D* [crypto             ]    AES-128 enabled
2021.04.27 16:14:32.325 *D* [crypto             ]    Blowfish-128 enabled
2021.04.27 16:14:32.325 *I* [crypto             ] Crypto library initialized (OpenSSL 1.1.1d  10 Sep 2019)
2021.04.27 16:14:32.325 *I* [crypto.cert        ] Server certificate not set
2021.04.27 16:14:32.334 *D* [threads.pool       ] Thread pool MAIN initialized (min=8, max=256)
2021.04.27 16:14:32.334 *D* [threads.pool       ] Thread pool AGENT initialized (min=4, max=256)
2021.04.27 16:14:32.363 *D* [watchdog           ] Watchdog thread started
2021.04.27 16:14:32.364 *D* [db.writer          ] Using 1 DCI data write queues
2021.04.27 16:14:32.364 *D* [db.writer          ] Raw DCI data flush interval is 30 seconds
2021.04.27 16:14:32.364 *D* [threads.pool       ] Thread pool NPE initialized (min=1, max=1024)
2021.04.27 16:14:32.364 *D* [                   ] SMTP mailer thread started
2021.04.27 16:14:32.365 *D* [                   ] SQLite version 3.27.2
2021.04.27 16:14:32.365 *I* [db.drv             ] Database driver "sqlite.ddr" loaded and initialized successfully
2021.04.27 16:14:34.605 *D* [                   ] SQLite version 3.27.2
2021.04.27 16:14:34.605 *I* [db.drv             ] Database driver "sqlite.ddr" loaded and initialized successfully
here something eats my memory few minutes
2021.04.27 16:17:06.738 *D* [nc                 ] Loading notification channel drivers from /usr/lib/x86_64-linux-gnu/netxms/ncdrv
2021.04.27 16:17:06.906 *D* [nc                 ] 18 notification channel drivers loaded
2021.04.27 16:17:06.906 *D* [nc                 ] 0 notification channels added
2021.04.27 16:17:06.907 *D* [ndd                ] Loading network device drivers from /usr/lib/x86_64-linux-gnu/netxms/ndd
2021.04.27 16:17:06.908 *I* [ndd                ] Network device driver CAMBIUM-CNPILOT loaded successfully
2021.04.27 16:17:06.908 *I* [ndd                ] Network device driver CAMBIUM-EPMP loaded successfully
2021.04.27 16:17:06.924 *I* [ndd                ] Network device driver SAF-INTEGRA-B loaded successfully
2021.04.27 16:17:06.925 *I* [ndd                ] Network device driver SYMBOL-WS loaded successfully
2021.04.27 16:17:06.925 *I* [ndd                ] Network device driver WESTERSTRAND loaded successfully
2021.04.27 16:17:06.926 *I* [ndd                ] Network device driver UBNT loaded successfully
2021.04.27 16:17:06.927 *I* [ndd                ] Network device driver AT loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CATALYST-GENERIC loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CATALYST-2900XL loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-ESW loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-NEXUS loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-SB loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-WLC loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-GENERIC loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver OPTIX loaded successfully
....
2021.04.27 16:17:12.453 *D* [                   ] 0 startup scripts processed
2021.04.27 16:17:12.453 *D* [                   ] Server initialization completed in 160155 milliseconds
2021.04.27 16:17:12.454 *D* [statcoll           ] Server statistic collector thread started
2021.04.27 16:17:12.454 *I* [                   ] NetXMS Server started

After server started all working ok.
HTop images attached from time after Netxms server is up and responding to console.

Any idea what can cause this peak ?

Victor Kirhenshtein

Hi!

Try to add the following option to netxmsd.conf:

DBCacheConfigurationTables = no

Best regards,
Victor

madmucho

@victor said in Memory peak usage at startup, till swap begins:

Hi!

Try to add the following option to netxmsd.conf:

DBCacheConfigurationTables = no

Best regards,
Victor

Thank You Victor, after configuration change memory footprint is much better now, memory peak is much smaller and server have enough RAM.
No swaping occurs at start.

NetXMS Forum

Memory peak usage at startup, till swap begins