nxapush not working?
-
Yes, the DCI parameter is "LOCAL_TEST", origin is "push".
With DebugLevel=6 in netxmsd and nxagent the only output I get is in the agent log file. Nothing shows up in the server log. Here is the agent log:
2021.03.31 13:39:08.836 *D* [ ] NamedPipeListener(nxagentd.push): accepted connection by user ats 2021.03.31 13:39:08.836 *D* [ ] ProcessPushRequest: connection established 2021.03.31 13:39:08.836 *D* [ ] ProcessPushRequest: received message CMD_PUSH_DCI_DATA 2021.03.31 13:39:08.836 *D* [ ] PushData: "LOCAL_TEST" = "7" 2021.03.31 13:39:08.836 *D* [ ] ProcessPushRequest: connection by user ats closedIs there a way to tell the agent which server it should connect to? Is this localhost by default?
-
What's exactly netxms agent and server versions?
What kind of connection is between server and agent - is it normal or tunnel? -
These are the versions we run at the moment:
$ netxmsd --version NetXMS Server Version 3.8.250 Build 3.8-250-g21bbde29bb (UNICODE) NXCP: 5.53.1.30 (AES-256, Blowfish-256, 3DES, AES-128, Blowfish-128) Built with: g++ (Debian 8.3.0-6) 8.3.0$ nxagentd -v NetXMS Core Agent Version 3.8.250 Build 3.8-250-g21bbde29bb (UNICODE)We have attempted to set up a tunnel connection following:
https://www.netxms.org/documentation/adminguide/server-management.html#self-signed-certificate-sample
placing certificates and keys in "/etc/netxmsd.conf.d/self/"Server Config:
# Log file name LogFile=/var/log/netxmsd # Increase logging verbosity, 0 (only errors) to 9 (verbose debug) DebugLevel=6 ## PostgreSQL DBDriver=pgsql.ddr DBServer=127.0.0.1 DBName=netxms DBLogin=netxms DBPassword=our_db_password TrustedCertificate = /etc/netxmsd.conf.d/self/rootCA.crt ServerCertificate = /etc/netxmsd.conf.d/self/server.crt ServerCertificateKey = /etc/netxmsd.conf.d/self/server.keyAgent Config:
# Log File LogFile=/var/log/nxagentd DebugLevel=8 # IP white list, can contain multiple records separated by comma. # CIDR notation supported for subnets. MasterServers=SERVER_IP ServerConnection=SERVER_FQHN######
The following logs are `tail -F /var/log/netxmsd /var/log/nxagentd | grep -i -e "crypto" -e "tunnel" -e "^==>"`'s output.
I have taken the liberty to:
1) prefix the filenames to each line ( awk )
2) substitute the first occurrence of "[" and "]" to "(" and ")" respectively in lines starting with "/var/log/netxmsd"
3) abstract some detailed information
######The first connection then succeeds,
/var/log/nxagentd 2021.04.08 08:20:43.023 *D* [tunnel ] Tunnel manager started /var/log/netxmsd 2021.04.08 08:20:43.026 *D* ( ) SocketListener/AgentTunnels: Incoming connection from CLIENT_IP /var/log/netxmsd 2021.04.08 08:20:43.026 *D* ( ) SocketListener/AgentTunnels: Connection from CLIENT_IP accepted /var/log/nxagentd 2021.04.08 08:20:43.026 *D* [tunnel ] SERVER_FQHN: Loading certificate from file /var/log/nxagentd 2021.04.08 08:20:43.026 *D* [tunnel ] SERVER_FQHN: Cannot open file "/var/lib/netxms/certificates/53CBA2C8E6D236B46D68AB5D430E9E1BB742786B.crt" (No such file or directory) /var/log/nxagentd 2021.04.08 08:20:43.026 *D* [tunnel ] SERVER_FQHN: Cannot open file "/var/lib/netxms/certificates/120AFD110002000400020000000000000001.crt" (No such file or directory) /var/log/netxmsd 2021.04.08 08:20:43.028 *D* (agent.tunnel ) SetupTunnel(CLIENT_IP): minimal TLS version set to 1.2 /var/log/netxmsd 2021.04.08 08:20:43.028 *D* (crypto.cert ) CreateTrustedCertificatesStore: trusted certificate "/etc/netxmsd.conf.d/self/rootCA.crt" added /var/log/netxmsd 2021.04.08 08:20:43.028 *D* (crypto.cert ) CreateTrustedCertificatesStore: added system certificate store at "/etc/ssl/certs" /var/log/netxmsd 2021.04.08 08:20:43.029 *D* (crypto.cert ) Certificate "/C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN" verification error 18 (self signed certificate) at depth 0 /var/log/nxagentd 2021.04.08 08:20:43.034 *D* [tunnel ] SERVER_FQHN: Server certificate subject is /C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN /var/log/nxagentd 2021.04.08 08:20:43.034 *D* [tunnel ] SERVER_FQHN: Server certificate issuer is /C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN /var/log/nxagentd 2021.04.08 08:20:43.034 *D* [tunnel ] SERVER_FQHN: Server certificate verification is disabled /var/log/nxagentd 2021.04.08 08:20:43.034 *D* [tunnel ] SERVER_FQHN: Sending message CMD_SETUP_AGENT_TUNNEL /var/log/netxmsd 2021.04.08 08:20:43.034 *D* (agent.tunnel ) SetupTunnel(CLIENT_IP): Agent certificate not provided /var/log/netxmsd 2021.04.08 08:20:43.035 *D* (agent.tunnel ) SetupTunnel(CLIENT_IP): assigned to poller #0 /var/log/netxmsd 2021.04.08 08:20:43.035 *D* (agent.tunnel.1 ) Tunnel started /var/log/netxmsd 2021.04.08 08:20:43.035 *D* (agent.tunnel.1 ) Received message CMD_SETUP_AGENT_TUNNEL /var/log/netxmsd 2021.04.08 08:20:43.035 *D* (agent.tunnel.1 ) Unbound tunnel initialized /var/log/netxmsd 2021.04.08 08:20:43.035 *D* (agent.tunnel.1 ) System name..............: SERVER_FQHN /var/log/netxmsd 2021.04.08 08:20:43.035 *D* (agent.tunnel.1 ) Hostname.................: SERVER_FQHN /var/log/netxmsd 2021.04.08 08:20:43.035 *D* (agent.tunnel.1 ) System information.......: Linux SERVER_FQHN 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) Platform name............: Linux-x86_64 /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) Hardware ID..............: 384518613C4CA702A5CFB61248FA459DC0E33B96 /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) Agent ID.................: 2529de7e-dc33-4595-b2b7-b3420ac838fe /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) Agent version............: 3.8.250 /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) Zone UIN.................: 0 /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) Agent proxy..............: NO /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) SNMP proxy...............: NO /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) SNMP trap proxy..........: NO /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) Syslog proxy.............: NO /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) User agent...............: NO /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) Hook script not found /var/log/netxmsd 2021.04.08 08:20:43.036 *D* (agent.tunnel.1 ) Sending message CMD_REQUEST_COMPLETED /var/log/nxagentd 2021.04.08 08:20:43.077 *D* [tunnel ] SERVER_FQHN: Received message CMD_REQUEST_COMPLETED /var/log/nxagentd 2021.04.08 08:20:43.077 *I* [ ] Tunnel with SERVER_FQHN established /var/log/nxagentd 2021.04.08 08:20:43.077 *D* [tunnel ] SERVER_FQHN: Tunnel is activeresulting in the tunnel showing up as unbound in:
# nxadm -c "show tunnels" BOUND TUNNELS ID | Node ID | EP | Chan. | Peer IP Address | System Name | Hostname | Platform Name | Agent Version | Agent Build Tag -----+---------+-----+-------+--------------------------+--------------------------+--------------------------+------------------+---------------+-------------------------- UNBOUND TUNNELS ID | EP | Peer IP Address | System Name | Hostname | Platform Name | Agent Version | Agent Build Tag -----+-----+--------------------------+--------------------------+--------------------------+------------------+---------------+------------------------------------ 1 | NO | CLIENT_IP | SERVER_FQHN | SERVER_FQHN | Linux-x86_64 | 3.8.250 | 3.8-250-g21bbde29bbIssuing an agent tunnel bind command to the server's node has netxmsd generate a key/cert pair in /var/lib/netxms/certificates/ and send it to the agent, which acknowledges the files and saves them:
/var/log/netxmsd 2021.04.08 08:24:25.841 *D* (client.session.0 ) Received message CMD_BIND_AGENT_TUNNEL /var/log/netxmsd 2021.04.08 08:24:25.841 *D* (agent.tunnel ) BindAgentTunnel: processing bind request 1 -> 1727 by user admin /var/log/netxmsd 2021.04.08 08:24:25.841 *D* (agent.tunnel.1 ) Node agent ID (00000000-0000-0000-0000-000000000000) do not match tunnel agent ID (2529de7e-dc33-4595-b2b7-b3420ac838fe) on bind /var/log/netxmsd 2021.04.08 08:24:25.841 *D* (agent.tunnel.1 ) Sending message CMD_BIND_AGENT_TUNNEL /var/log/netxmsd 2021.04.08 08:24:25.841 *D* (event.corr ) CorrelateEvent: event SYS_TUNNEL_AGENT_ID_MISMATCH id 332805 source SERVER_FQHN [1727] /var/log/netxmsd 2021.04.08 08:24:25.841 *D* (event.proc ) EVENT SYS_TUNNEL_AGENT_ID_MISMATCH [96] at {0} (ID:332805 F:0x0001 S:1 TAGS:"") FROM SERVER_FQHN: Agent ID 2529de7e-dc33-4595-b2b7-b3420ac838fe on node do not match agent ID 00000000-0000-0000-0000-000000000000 on tunnel from SERVER_FQHN (CLIENT_IP) at bind /var/log/nxagentd 2021.04.08 08:24:25.842 *D* [tunnel ] SERVER_FQHN: Received message CMD_BIND_AGENT_TUNNEL /var/log/nxagentd 2021.04.08 08:24:25.893 *D* [tunnel ] SERVER_FQHN: Sending message CMD_REQUEST_CERTIFICATE /var/log/netxmsd 2021.04.08 08:24:25.893 *D* (agent.tunnel.1 ) Received message CMD_REQUEST_CERTIFICATE /var/log/netxmsd 2021.04.08 08:24:25.893 *D* (crypto.cert ) IssueCertificate: new certificate request (CN override: 63c86ba9-08e4-4b4a-8e4a-6a09e831ddac, OU override: 84bff5a5-d832-45fe-a18b-e2ddee7519a3) /var/log/netxmsd 2021.04.08 08:24:25.895 *D* (crypto.cert ) IssueCertificate: new certificate with subject "/C=CC/O=OOOOOO/OU=84bff5a5-d832-45fe-a18b-e2ddee7519a3/CN=63c86ba9-08e4-4b4a-8e4a-6a09e831ddac" issued successfully /var/log/netxmsd 2021.04.08 08:24:25.896 *D* (agent.tunnel.1 ) New certificate issued /var/log/netxmsd 2021.04.08 08:24:25.896 *D* (node.agent ) Tunnel ID for node SERVER_FQHN [1727] set to 63c86ba9-08e4-4b4a-8e4a-6a09e831ddac /var/log/netxmsd 2021.04.08 08:24:25.896 *D* (agent.tunnel.1 ) Sending message CMD_NEW_CERTIFICATE /var/log/nxagentd 2021.04.08 08:24:25.897 *D* [tunnel ] SERVER_FQHN: Received message CMD_NEW_CERTIFICATE /var/log/nxagentd 2021.04.08 08:24:25.898 *D* [tunnel ] SERVER_FQHN: Certificate and private key saved /var/log/nxagentd 2021.04.08 08:24:25.898 *D* [tunnel ] SERVER_FQHN: Sending message CMD_REQUEST_COMPLETED /var/log/netxmsd 2021.04.08 08:24:25.898 *D* (agent.tunnel.1 ) Received message CMD_REQUEST_COMPLETED /var/log/netxmsd 2021.04.08 08:24:25.898 *D* (agent.tunnel.1 ) Certificate successfully issued and transferred to agent /var/log/netxmsd 2021.04.08 08:24:25.898 *D* (agent.tunnel.1 ) Bind successful, resetting tunnel /var/log/netxmsd 2021.04.08 08:24:25.898 *D* (agent.tunnel.1 ) Sending message CMD_RESET_TUNNEL /var/log/nxagentd 2021.04.08 08:24:25.898 *D* [tunnel ] SERVER_FQHN: Received message CMD_RESET_TUNNEL /var/log/nxagentd 2021.04.08 08:24:25.898 *D* [tunnel ] SERVER_FQHN: Receiver thread stopped (tunnel reset) /var/log/nxagentd 2021.04.08 08:24:25.898 *W* [ ] Tunnel with SERVER_FQHN closed /var/log/netxmsd 2021.04.08 08:24:25.964 *D* (client.session.0 ) Received message CMD_GET_AGENT_TUNNELS /var/log/nxagentd 2021.04.08 08:24:43.084 *D* [tunnel ] SERVER_FQHN: Resetting tunnel /var/log/netxmsd 2021.04.08 08:24:43.084 *D* (agent.tunnel.1 ) Tunnel closed by peer /var/log/netxmsd 2021.04.08 08:24:43.085 *D* (agent.tunnel.1 ) Tunnel unregistered /var/log/netxmsd 2021.04.08 08:24:43.085 *D* (agent.tunnel.1 ) Tunnel closure completed /var/log/netxmsd 2021.04.08 08:24:43.085 *D* (agent.tunnel.1 ) Tunnel shutdown /var/log/netxmsd 2021.04.08 08:24:43.085 *D* (agent.tunnel.1 ) Tunnel destroyedThe tunnel is reset automatically and all following connection attempts show up in the logs as follows:
/var/log/netxmsd 2021.04.08 08:24:43.085 *D* ( ) SocketListener/AgentTunnels: Incoming connection from CLIENT_IP /var/log/netxmsd 2021.04.08 08:24:43.085 *D* ( ) SocketListener/AgentTunnels: Connection from CLIENT_IP accepted /var/log/netxmsd 2021.04.08 08:24:43.085 *D* (agent.tunnel ) SetupTunnel(CLIENT_IP): minimal TLS version set to 1.2 /var/log/netxmsd 2021.04.08 08:24:43.086 *D* (crypto.cert ) CreateTrustedCertificatesStore: trusted certificate "/etc/netxmsd.conf.d/self/rootCA.crt" added /var/log/netxmsd 2021.04.08 08:24:43.086 *D* (crypto.cert ) CreateTrustedCertificatesStore: added system certificate store at "/etc/ssl/certs" /var/log/nxagentd 2021.04.08 08:24:43.085 *D* [tunnel ] SERVER_FQHN: Loading certificate from file /var/log/nxagentd 2021.04.08 08:24:43.089 *D* [tunnel ] SERVER_FQHN: Certificate and private key loaded /var/log/netxmsd 2021.04.08 08:24:43.089 *D* (crypto.cert ) Certificate "/C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN" verification error 18 (self signed certificate) at depth 0 /var/log/nxagentd 2021.04.08 08:24:43.092 *D* [tunnel ] SERVER_FQHN: Server certificate subject is /C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN /var/log/nxagentd 2021.04.08 08:24:43.092 *D* [tunnel ] SERVER_FQHN: Server certificate issuer is /C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN /var/log/nxagentd 2021.04.08 08:24:43.092 *D* [tunnel ] SERVER_FQHN: Server certificate verification is disabled /var/log/netxmsd 2021.04.08 08:24:43.093 *D* (crypto.cert ) Certificate "/C=CC/O=OOOOOO/OU=84bff5a5-d832-45fe-a18b-e2ddee7519a3/CN=63c86ba9-08e4-4b4a-8e4a-6a09e831ddac" verification error 7 (certificate signature failure) at depth 0 /var/log/netxmsd 2021.04.08 08:24:43.093 *D* (agent.tunnel ) SetupTunnel(CLIENT_IP): TLS handshake failed (error:00000001:lib(0):func(0):reason(1)) /var/log/nxagentd 2021.04.08 08:24:43.093 *D* [tunnel ] SERVER_FQHN: Sending message CMD_SETUP_AGENT_TUNNEL /var/log/nxagentd 2021.04.08 08:24:43.093 *D* [crypto ] OpenSSL error stack: /var/log/nxagentd 2021.04.08 08:24:43.093 *D* [crypto ] error:1409441B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error /var/log/nxagentd 2021.04.08 08:24:43.093 *D* [tunnel ] SERVER_FQHN: SSL_write error (bytes=-1 ssl_err=6 socket_err=0) /var/log/nxagentd 2021.04.08 08:24:43.093 *D* [tunnel ] SERVER_FQHN: Receiver thread stopped (MSGRECV_CLOSED) /var/log/nxagentd 2021.04.08 08:24:43.093 *W* [ ] Tunnel with SERVER_FQHN closed /var/log/nxagentd 2021.04.08 08:24:53.093 *D* [tunnel ] SERVER_FQHN: Cannot configure tunnel (request timeout) /var/log/nxagentd 2021.04.08 08:25:23.094 *D* [tunnel ] SERVER_FQHN: Resetting tunnelHow come the server is building a new key/cert pair for the client, based on data that later cannot be found for verification?
Best regards,
Florian
-
Do you have any DPI system between agent and server that could intercept ssl connections?
-
@filipp-sudanov said in nxapush not working?:
Do you have any DPI system between agent and server that could intercept ssl connections?
No. Server and agent are living on the same physical machine.
-
Using agent tunnel connection on the same machine does not have any benefits (but should work anyways).
Did you try tunnel connection from agent on another machine? Did it work?
What's exact OS version? If it's debian/ubuntu, what's the output of dpkg -l | grep libssl -
Well, okay then: How do I connect the agent with the server when living on the same machine? We don't need the agent for anything else but nxapush, so we haven't tried it from another machine.
We are running Debian Buster. libssl:
ii libssl1.1:amd64 1.1.1d-0+deb10u6 amd64 Secure -
- install netxms-agent package - you probably did that
- in /etc/nxagentd.conf there should be
MasterServers=127.0.0.0/8 - there should be a node in netxms that corresponds to your netxms server. This node is automatically created with external IP of your machine. In properties of that node -> Communications change primary host name to 127.0.0.1
- perform Poll -> Configuration (full) on that node. Poll output should say that there is communication with the agent.
- try nxapush
Also, if you need to push values from server there's nxpush utility that connects directly to netxms server.
-
Thanks! nxapush works perfectly now.
Out of curiosity - would you expect your solution to work without the full configuration poll, given a netxmsd and nxagent restart? Why?
We feel like having tried something very similar without success.---
We had some misconceptions regarding the respective feature sets of nxapush vs nxpush, resulting in us not considering the latter.
After some more fiddling with it, triggered by your suggestion, we could clear those up and find some significant advantages of using nxpush over nxapush in our use case.
Namely the more flexible reference methods for nodes.
Thank you again for that.Kind regards,
Florian
-
Yes, right, nxapush is intended to run on remote nodes, not on the server.
If the system is correctly configured initially - agent configuration file allows connection from server - the configuration poll will happen automatically and server will properly operate with the agent. If the agent is not accessible when server is launched for the first time, the server will remember that agent is not available on that node.
That's standard procedure - server remembers, if agent or SNMP is available on each node. This is seen in Object Details - Capabiliest (isAgent, etc). Capabilities are changen on Full configuration poll (and also automatically, but with very big time interval). -
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login