NetXMS Server 6.0.0 Memory Leak

[ncabral]

Hello Everyone:

I am currently running a NetXMS Server Version 6.0.0. I recently upgraded it from 5.2.8 and I have experienced the same issue from the prior version to the current version. It appears that the server begins with low RAM utilization, but increases at a consistent rate until it maxes out the available memory, then it idles with about 500mb of free RAM available.

Above is the chart depicting RAM utilization. The server has 16gb of RAM allocated to it. I have tried different amounts of RAM. Previously it had 32gb of RAM allocated to it. The random peaks have to do with server maintenance and shouldn't be related to NXMS, but notice that there are dips, and consistent increases in memory utilization until I jump in to do something.

Each dip in memory utilization in the chart is a server reboot. The rate of increase seems to be about 1.8gb per day.

Are there any known memory leak issues with NetXMS? I assumed the issue would have been fixed with NetXMS v6.0.0.

[victor]

Can you show virtual and physical memory consumption for netxmsd process specifically?

[ncabral]

Hello, thank you for your response.

I am currently monitoring the active processes on the server, and I am starting to think it is not NetXMS that is doing it since the netxms-server process is not utilizing the most ram out of all the processes running on the server.

Currently PostgreSQL is using the largest percentage of the server's RAM. I am going to continue monitoring it to see which process increases memory utilization. Since the increase is gradual I'm going to have to watch it for a few days.

The second process in the list when sorted in descending order based on memory usage is the netxmsd process.

The server's base memory utilization is about 4gb after startup. The netxmsd process is only utilizing about half a gigabyte.

[ncabral]

@victor said in NetXMS Server 6.0.0 Memory Leak:

Can you show virtual and physical memory consumption for netxmsd process specifically?

Thanks again for suggesting I separate the memory usage by process. I am certain it is related to the PostgreSQL database. I didn't suspect it at first because I have other servers running PostgreSQL, but I don't see them behave in this manner.

Since posting this, I have also upgraded the server to NetXMS 6.0.1 and I am seeing new patterns in memory usage. Currently every 6 hours or so the memory usage drops significantly after slowly building up. I am guessing NetXMS 6 handles database connections more efficiently than NetXMS 5 did.

Also, I the NetXMS server is running in a Proxmox VM. I had been monitoring RAM usage through the charts provided by Proxmox. If you notice the following two images, it seems Proxmox isn't quite seeing the same thing as when I separate memory usage by process directly in the VM. I tried both the average view and the maximum view settings in Proxmox and I get the same consistent incline.

NetXMS Chart Showing RAM Usage Separated by Process (I didn't bother charting any other processes because PostgreSQL and NetXMS Server are the top users of RAM according to the 'top' command in the linux terminal at all times.)

Here is the RAM usage broken down by process relative to the available system RAM. (Percentage)

Proxmox Chart Showing RAM usage for the Virtual Machine

I have a suspicion that Proxmox is seeing memory that was allocated by a process at some point and still has residual data in the memory space even if it has since been released by the kernel from the process at the VM level. I suspect this because I have another VM running RouterOS and Proxmox shows 100% of the RAM in use at all times regardless of how much there is, but RouterOS shows 98% of the available RAM as free. I also have a Mimosa Management Platform server that Proxmox shows as using roughly 95% of the allocated RAM, but the VM itself shows only about 50% in use give or take 10%.

If there was an issue in prior releases of NetXMS it appears to be fixed now.

[ncabral]

I spoke too soon. The PostgreSQL server went up to 8 GB overnight and still rising. It appears the database uses whatever is available regardless of how much it is. I know now it is not a NetXMS issue. It's a PostgreSQL issue. Thanks again for the suggestions. Hopefully this post will help others.

[Filipp Sudanov]

Was Postgres tuned, may be excessivly, e.g. too big shared_buffers or something?

[ncabral]

Hello, PostgreSQL is using it's default settings for a system with 16GB of RAM. I googled the recommended sizes for shared_buffers, effective_cache_size, work_mem, maintenance_work_mem, max_connections, and the current settings for those parameters are less than the recommended for a dedicated PostgreSQL server.

Currently shared_buffers is at 2GB, work_mem is at 3MB, max_connections is 100, effective_cache_size is 7GB, maintenance_work_mem is 1GB.

I'm probably going to have to adjust the settings around to see what fixes it. I might have to reduce the number of worker processes. Currently the set maximum is 27, and the top command confirms there are a lot of them.