NetXMS Forum

I've set all Nodes behind the proxy as unmanaged  and disabled all DCIs od the proxy to avoid false results.
Then I've captured the lsof output of the  nxagentd process to a file before the action few seconds after the action and the diff output of the two files is following (server name is changed to xxxxx):

55a56
> nxagentd 100 root   16u  IPv4         1130371273      0t0        TCP xxxxx:netxms-agent->192.168.201.1:51656 (CLOSE_WAIT)
58a60,61
> nxagentd 100 root   20r  FIFO               0,12      0t0 1130371274 pipe
> nxagentd 100 root   21w  FIFO               0,12      0t0 1130371274 pipe

Then I captured the output 1 minute after the action and the lines were still there. 2 minutes after the action all 3 lines have disappeared from the lsof output.
This means that closing the pipes takes some time even if the process on the other side of the pipe is no longer running (checked with ps).
I did the test multiple times, each time with the same result.

The question is whether if a large number of requests come, closing pipes does not take longer a therefore the average number of open pipes is increasing.
Another question is how to modify the configuration to avoid this. CPU and free memory on the server and proxy are OK all the time and the actions DCIs are started every 15 minutes.

OK I will do it, but the Actions are used in DCIs therefore it would be better to stop disable all DCIs using the Actions and start the DCI script manually. It takes some time to prepare it.

No, Im not (as far as I know). I use SNMP proxy only but it is a Zone proxy therefore I've enabled all proxies.
It is no problem to disable SNMPTrapProxy, SyslogProxy, and TCPProxy.

I've attached the agent configuration file.

nxagentd.conf

Check this: https://www.netxms.org/forum/general-support/(windows)-web-api-install/

Can I help with some specific logging?

I had to restart the agent once again because there were the same messages in the log.
The problem with interface status didn't occur again therefore it is sure tah the main reason for the strange Status was the agent problem with the open files.
Before restart there were:
80 connections from the server
43694 pipes

I don't understand the number of open open pipes.
Currently (30 minutes after agent restart) there are  4228 open pipes but the number but that number is not only rising but also falling.
What is strange that there were only 214 running processes on the agent at this time therefore it cannot be pipes waiting for the output form the processes started by AgentExecuteActionWithOutput call or the pipes are not closed by the agent.

Current open files situation on the agent:
2279 connections form the server
12745 pipes

It was reported by and Linux agent 3.7.130 on CentOS 7.
The interface is OpenVPN interface tun0. After agent restart everything was OK.

There were lot of following messages in the agent log

2021.01.22 22:17:14.368 *E* [                   ] Unable to accept incoming connection (24 Too many open files)

When I check the number of open files by agent now and compare it with the number after start it grows.
The most open files are of these types (listed by lsof -p, the name of the server with agent is changed to xxxxx):

nxagentd 2524095 root *360u  IPv4         3489842327      0t0        TCP xxxxx:netxms-agent->192.168.201.1:36936 (CLOSE_WAIT)
nxagentd 2524095 root *361r  FIFO               0,12      0t0 3489847436 pipe

If the agent sends the Status via different TCP connection then the Oper State the above error message in the log can be the reason.

The agent is a Zone proxy and a lot of AgentExecuteActionWithOutput calls are done on the agent. This can be the reason for the open pipes, but not so many actions are started at the same time that it could exceed the maximal number of open files.
The maximal number of open files is set to 65535. I can set a higher value, but it is short term solution only.

What does the Critical interface status mean?

Thank you.
According description this setting is common for DNS and SNMP system name. Is it possible to assign SNMP name only or change the name assigned by DNS if the SNMP agent is enabled later?

My server property UseDNSNameForDiscoveredNodes has the value 0  but the Nodes accessible via Zone proxy running on Amazon VPS receive the name like "ip-192-168-213-2.eu-central-1.compute.internal" as the Object name. This name is probably some internal Amazon DNS name.
On Zone proxies which are not running on Amazon VPS the IP address is assigned but there are no such DNS names available.

Is the property UseDNSNameForDiscoveredNodes not working on proxies or did I understand something wrong?

I do notification via following nxsl script:

/*
$1 - Unique ID of event source object
$2 - Email subject
$3 - Email body
*/

node = FindObject($1);
if (node != null) {
    emails = node->getCustomAttribute("notify");
    if (emails != null && emails != "") {
         SendMail(emails, $2, $3);
    }
}

The notify  Custom Attribute contains email addresses separated by ";". The value can be set for each Node differently or as Inheritable for whole Container.
It it is missing no notification is sent.

I solve the different thresholds for different nodes similar way (Custom Attribute and nxsl script checking the threshold).

Did you mean not receive notifications (for example via email) or not see the alarms?

I have solution for the first case which even allows notification to different people for different nodes.

Thank you.
I know this workaround.

Hi Victor,

thank you for your answer. It was good idea first ask before I started  to modify the sources  ;).

If I understand this right, there is no such posibility for the NXSL function executeSSHCommand. Is it so?

I've seen that that the SSH.Command uses fixed port 22.
It would be useful to have the possibility to specify different port. Can I add the fifth optional parameter port with default value 22 to the SSH.Command? Would you add it this change to your sources?

Hi Victor,

the systemd setting was the problem.

Thank you.

After I increased the number of monitored Nodes via one proxy agent, I see lot of following messages in the agent log:

2020.12.28 13:23:52.562 *E* [                   ] Unable to accept incoming connection (24 Too many open files)
2020.12.28 13:23:53.066 *E* [                   ] Unable to accept incoming connection (24 Too many open files)
2020.12.28 13:23:56.570 *E* [                   ] Unable to accept incoming connection (24 Too many open files)

I've tried to increase the operating system value for max number of open file descriptors but even the value 65535 is not sufficient.

The CPU usage and other parameters of the proxy agent and server are OK.
I use some SNMP DCIs for and some script DCIs which start Agent actions on the proxy agent, but the number of DCIs is lower than 65535.

Objects............: 4404
Monitored nodes....: 1221 (approximately 900 via this proxy agent)
Collectible DCIs...: 14683

What does this message mean?
Is there any possibility to solve the problem?