FYI, I am pretty sure all historical data for the data collection items was lost because there is no data prior to today.
But all the devices, the network topology, the infrastructure containers, the charts, custom data collection items, SSH Logins, everything is still there and working.
Thank you! This worked.
Specifically I did the following.
After the apt upgrade finished, I had to use mkdir to create the "/var/lib/netxms/images" directory first. Then I had to use the touch command for each of the stock image names as directed.
Once all the empty stock image files were created, I upgraded the database and it finished successfully. The netxmsd daemon started up and netxms is working again.
@alk said in Issues Upgrading from NetXMS Server 6.1.4 to 6.2.0 via apt package manager.:
I've pushed 6.2.0-2, which have workaround for this issue.
Note: workaround only handles stock images, if any custom was uploaded and then deleted directly on the filesystem - upgrade will fail.List of stock files:
092e4b35-4e7c-42df-b9b7-d5805bfac64e 1ddb76a3-a05f-4a42-acda-22021768feaf 7cd999e9-fbe0-45c3-a695-f84523b3a50c 904e7291-ee3f-41b7-8132-2bd29288ecc8 b314cf44-b2aa-478e-b23a-73bc5bb9a624 ba6ab507-f62d-4b8f-824c-ca9d46f22375 bacde727-b183-4e6c-8dca-ab024c88b999 f5214d16-1ab1-4577-bb21-063cfd45d7af f9105c54-8dcf-483a-b387-b4587dfd3cbaSimplest approach for missing custom images - create empty file, e.g. "touch /var/lib/netxms/images/...".
So, is the workaround you pushed available in the repository for APT upgrading? Or is it something I am going to have to manually do until there is an official fix?
As far as creating an empty file goes, the image it cannot read is "/var/lib/netxms/images/092e4b35-4e7c-42df-b9b7-d5805bfac64e". Will just creating an empty image file named "092e4b35-4e7c-42df-b9b7-d5805bfac64e" allow the database to upgrade?
Hello, I am trying to upgrade my NetXMS server from version 6.1.4 to 6.2.0 using the apt package manager and I am encountering the following issue when I attempt to upgrade the database before restarting the service. The service does not start after the upgrade because the database still has not been upgraded.
username@hostname:~$ sudo nxdbmgr upgrade
NetXMS Database Manager Version 6.2.0 Build 6.2-471-gce46095f23
Upgrading database...
Upgrading from version 62.12 to 62.13
Cannot read image file "/var/lib/netxms/images/092e4b35-4e7c-42df-b9b7-d5805bfac64e"
Rolling back last stage due to upgrade errors...
Database upgrade failed
username@hostname:~$
Hello, PostgreSQL is using it's default settings for a system with 16GB of RAM. I googled the recommended sizes for shared_buffers, effective_cache_size, work_mem, maintenance_work_mem, max_connections, and the current settings for those parameters are less than the recommended for a dedicated PostgreSQL server.
Currently shared_buffers is at 2GB, work_mem is at 3MB, max_connections is 100, effective_cache_size is 7GB, maintenance_work_mem is 1GB.
I'm probably going to have to adjust the settings around to see what fixes it. I might have to reduce the number of worker processes. Currently the set maximum is 27, and the top command confirms there are a lot of them.
I spoke too soon. The PostgreSQL server went up to 8 GB overnight and still rising. It appears the database uses whatever is available regardless of how much it is. I know now it is not a NetXMS issue. It's a PostgreSQL issue. Thanks again for the suggestions. Hopefully this post will help others.
@victor said in NetXMS Server 6.0.0 Memory Leak:
Can you show virtual and physical memory consumption for netxmsd process specifically?
Thanks again for suggesting I separate the memory usage by process. I am certain it is related to the PostgreSQL database. I didn't suspect it at first because I have other servers running PostgreSQL, but I don't see them behave in this manner.
Since posting this, I have also upgraded the server to NetXMS 6.0.1 and I am seeing new patterns in memory usage. Currently every 6 hours or so the memory usage drops significantly after slowly building up. I am guessing NetXMS 6 handles database connections more efficiently than NetXMS 5 did.
Also, I the NetXMS server is running in a Proxmox VM. I had been monitoring RAM usage through the charts provided by Proxmox. If you notice the following two images, it seems Proxmox isn't quite seeing the same thing as when I separate memory usage by process directly in the VM. I tried both the average view and the maximum view settings in Proxmox and I get the same consistent incline.
NetXMS Chart Showing RAM Usage Separated by Process (I didn't bother charting any other processes because PostgreSQL and NetXMS Server are the top users of RAM according to the 'top' command in the linux terminal at all times.)
Here is the RAM usage broken down by process relative to the available system RAM. (Percentage)
Proxmox Chart Showing RAM usage for the Virtual Machine
I have a suspicion that Proxmox is seeing memory that was allocated by a process at some point and still has residual data in the memory space even if it has since been released by the kernel from the process at the VM level. I suspect this because I have another VM running RouterOS and Proxmox shows 100% of the RAM in use at all times regardless of how much there is, but RouterOS shows 98% of the available RAM as free. I also have a Mimosa Management Platform server that Proxmox shows as using roughly 95% of the allocated RAM, but the VM itself shows only about 50% in use give or take 10%.
If there was an issue in prior releases of NetXMS it appears to be fixed now.
Hello, thank you for your response.
I am currently monitoring the active processes on the server, and I am starting to think it is not NetXMS that is doing it since the netxms-server process is not utilizing the most ram out of all the processes running on the server.
Currently PostgreSQL is using the largest percentage of the server's RAM. I am going to continue monitoring it to see which process increases memory utilization. Since the increase is gradual I'm going to have to watch it for a few days.
The second process in the list when sorted in descending order based on memory usage is the netxmsd process.
The server's base memory utilization is about 4gb after startup. The netxmsd process is only utilizing about half a gigabyte.
Hello Everyone:
I am currently running a NetXMS Server Version 6.0.0. I recently upgraded it from 5.2.8 and I have experienced the same issue from the prior version to the current version. It appears that the server begins with low RAM utilization, but increases at a consistent rate until it maxes out the available memory, then it idles with about 500mb of free RAM available.
Above is the chart depicting RAM utilization. The server has 16gb of RAM allocated to it. I have tried different amounts of RAM. Previously it had 32gb of RAM allocated to it. The random peaks have to do with server maintenance and shouldn't be related to NXMS, but notice that there are dips, and consistent increases in memory utilization until I jump in to do something.
Each dip in memory utilization in the chart is a server reboot. The rate of increase seems to be about 1.8gb per day.
Are there any known memory leak issues with NetXMS? I assumed the issue would have been fixed with NetXMS v6.0.0.
