Update 3.7.130 > 3.8.226: Thousands of Agent Errors
-
-
Thanks for the reply, I had seen your post and tried that but it did not work in this instance. I've checked the permissions on the NextXMS folder structure and the account running the service should have access so I don't think the file it can't find is in there. I've re-installed the WebUI a couple of times now, still no joy, the server appears to be running OK, using the java console appears to work and no info is missing, currently at a loss as to what the problem is.
DH
-
Hey Frank,
thanks for the hint with the Service User...
I check it at my next try to install V3.8.Have anyone any idea whats the "CRT Error" of the Agent means? So i can take a look at my installation to find any errors or prerequesits that i need for the update to 3.8.226?
Greetings
Marco -
I am having the exact problem with running the 3.8.226 x64 and 3.8.250 x64 agent running on windows 2012R2, 2016, and 2019 against netxms core 3.4.313 on windows server 2016. The agent will run for a few minutes and then start filling up the system event log with the following error.
Log Name: System
Source: NetXMS Win32 Agent
Date: 4/4/2021 1:14:18 PM
Event ID: 1000
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: mycomputer.mydomain.com
Description:
CRT error detected
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="NetXMS Win32 Agent" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-04-04T17:14:18.692060400Z" />
<EventRecordID>12345</EventRecordID>
<Channel>System</Channel>
<Computer>mycomputer.mydomain.com</Computer>
<Security />
</System>
<EventData>
<Data>CRT error detected</Data>
</EventData>
</Event>Once the error storm starts, agent computer has high CPU usuage and netxms core starts consuming high memory.
-
I think I have narrowed it down to having SubAgent = logwatch.nsm enabled. When a logwatch event is raised I receive the CRT error detected.
<parser trace="0" name="LogParser-EL-Application">
<file encoding="UTF-8">*Application</file>
<rules>
<rule name="ApplicationEventLogError" break="true">
<match repeatCount="0" repeatInterval="60">(.*)</match>
<event>EL_APPLICATION_ERROR</event>
<level>1</level>
</rule>
<rule name="ApplicationEventLogWarning" break="true">
<match repeatCount="0" repeatInterval="60">(.*)</match>
<event>EL_APPLICATION_WARNING</event>
<level>2</level>
</rule>
<rule name="ApplicationWindowsErrorReporting" break="true">
<match repeatCount="0" repeatInterval="1">(.*)</match>
<event>EL_APPLICATION_WINDOWS_ERROR_REPORTING</event>
<level>4</level>
<id>1001</id>
<source>Windows Error Reporting</source>
</rule>
</rules>
<macros/>
</parser> -
Hi suphu,
many thanks for the infos... We have logwatch.nsm enabled at every NetXMS Agent installation. So, i would wait with the upgrade to 3.8, until a new version is available...
Are there any infos from the NetXMS developers for this (maybe) bug?
Greetings
Marco -
Hi,
we likely found the issue. New patch release will be available in a next few days, please re-test this issue then.
Best regards,
Victor -
-
-
Hello @all,
today i installed NetXMS Server V3.8.267: Server runs, no CRT Error from Agent...
... Thanks for the Patch Viktor... I installed the WebUI Server V3.8.262 too, but the service doesn't start... I tried to run the service as "Lokales System" (as Frank suggested) but the service still doesn't start.
I attached the Log File from the WebUI service.. Maybe one of the Devs can take a look into it...
Greetings
Marco
-
Hi,
log file just states that service cannot be started. Is there any other log files?
Best regards,
Victor -
Hi Victor,
there are three files in the C:\NetXMS\WebUI\logs Directory:
commons-daemon.2021-04-13.log (attached in the last post)
nxwebui-stderr.2021-04-13.log (only one line of content "Failed to start service.")
nxwebui-stdout.2021-04-13.log (empty file)Are there any log at other locations or can i activate verbose logging for the WebUI Service?
Greetings
Marco -
Did you install it on top of old version? If yes, try to remove it and do clean install (make sure to manually delete all remaining work directories after uninstall). Select bundled JVM when installing.
Best regards,
Victor -
Hi Victor,
first i installed it over the old version > not worked.
Then i deinstalled it, deleted the c:\netxms\webui manualy and reinstall (with jre) > not worked.
What for directorys i had to clean after doing a "normal" deinstall? Is the "c:\netxms\webui" the only one?
By the way: In the c:\netxms directory is also the NetXMS Server installed...
At WebUI install i let the suggested installation path ("c:\NetXMS") unchanged. NetXMS Web UI is installed in c:\NetXMS\WebUI. The JRE is installed in c:\NetXMS\bin\jre.
Is this correct?
Greetings
Marco -
Did you change service to run under "Local System" after clean install?
Best regards,
Victor -
Hi Victor,
i have gone thru following steps:
- deinstalled NetXMS WebUI (over Windows Program deinstallation)
- manually deleted c:\NetXMS\WebUI
- installed NetXMS WebUI with netxms-webui-3.8.262-x64.exe
- Service not comes up after installation
- change service user to System Account
- start service > service start fails (please see log files, now there are more informations in log)
Greetings
Marco
-
Very strange, I just did the same and service started correctly, so it somehow related to environment. Try to install Java 11 from AdoptOpenJDK separately - maybe we miss something when pack embedded JRE. Those are log lines that are most suspicious:
[2021-04-14 09:41:19] [error] ( javajni.c:828 ) [10492] CreateJavaVM Failed with error [-1]
[2021-04-14 09:41:19] [error] ( javajni.c:828 ) [10492] Das System kann die angegebene Datei nicht finden.Best regards,
Victor -
Also found this thread: https://stackoverflow.com/questions/63012015/jni-createjavavm-fails-with-adopt-openjdk-11-0-8. Do you have JAVA_HOME environment set? If yes, try to remove it completely (preferred) or point to C:\NetXMS\bin\jre.
Best regards,
Victor -
Hi Victor,
JAVA_HOME was not set. AdoptOpenJDK is installed in V8 + V11, i deinstalled both (needed these only for the NextXMS Client). Then i reinstalled NetXMS WebUI as described in my last post.... Same errors in log..
After that, i deinstalled NetXMS WebUI an reinstalled it at another setup path (new patch: c:\NetXMS_WebUI) an voila: Service is starting and i can access NetXMS with the WebUI...
I would now leave the setup as it is.... Can i delete the c:\netxms\bin\jre path? or is this jre needed for/from the NetXMS Server (i have the NetXMS Server installed at c:\NetXMS)
Greetings
Marco -
Hi,
yes, I think you can delete c:\netxms\bin\jre. I have no idea why it was not working when installed into same directory. But we are going to merge server ad web UI installers into one anyway in version 4, so it should not be an issue after that.
Best regards,
Victor
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login