Problem creating tunnel between server and agent
-
-
Thanks for your reply...
@tfines said in Problem creating tunnel between server and agent:
Please post your agent config file.
ServerConnection = 192.168.10.6
ConfigIncludeDir = C:\NetXMS\etc\nxagentd.conf.d
LogFile = {syslog}
FileStore = C:\NetXMS\var
SubAgent = filemgr.nsm
SubAgent = ping.nsm
SubAgent = logwatch.nsm
SubAgent = wmi.nsm
DebugLevel = 6
LogFile = c:\netxms\log\agent.logWhat setting did you select on the server for how it handles new nodes?
New nodes need to be added manually. I have tried with manually added node and bind as well as with "create node and bind". AgentTunnel.NewNodesContainer is left empty. The node appears in the root folder.
Best...
Detlev -
Hi!
What agent versions you are using, and on what platform? We've seen such issue with Windows agents and it was solved by rebuilding agent with newer OpenSSL.
Best regards,
Victor -
@victor said in Problem creating tunnel between server and agent:
What agent versions you are using, and on what platform? We've seen such issue with Windows agents and it was solved by rebuilding agent with newer OpenSSL.
Hi Victor,
I tried the 3.3.314 windows x64 agent on both Win7-x64 and server2019-x64
Regards
Detlev -
Please try the most recent nxagent-3.3.330 version, adding EnableSSLTrace=yes to agent configuration file and setting DebugLevel=7. Please share agent log for the time when the situation occurs.
-
@filipp-sudanov said in Problem creating tunnel between server and agent:
Please try the most recent nxagent-3.3.330 version, adding EnableSSLTrace=yes to agent configuration file and setting DebugLevel=7. Please share agent log for the time when the situation occurs.
Hi Filipp,
I have updated the agent as requested and started a clean attempt. Same result as before.
Please find attached agent log
Regards
Detlev
-
Hi,
all seems going well until this point:
2020.06.02 06:04:06.788 *D* [tunnel ] 192.168.10.6: SSL_write error (bytes=-1 ssl_err=5 errno=2)
SSL error 5 means underlying socket error, which is strange because SSL negotiation seems to be completed successfully. Are you using server version 3.3.x as well? Can you check what was logged in server log during that attempt?
Best regards,
Victor -
@victor said in Problem creating tunnel between server and agent:
2020.06.02 06:04:06.788 *D* [tunnel ] 192.168.10.6: SSL_write error (bytes=-1 ssl_err=5 errno=2)
SSL error 5 means underlying socket error, which is strange because SSL negotiation seems to be completed successfully. Are you using server version 3.3.x as well? Can you check what was logged in server log during that attempt?
Hi,
yes, server is latest 3.3 version.
I ran same procedure again with debuglevel 7 on server. I have cut beginning and end of the log, but it still contains a lot of waste.
I think the most relevant line is at time 12:19:10.069 - handshake failed.
I have attached a zip file containing the server log.
Regards
Detlev
-
Please check that your server certificate has CA constraint set to TRUE. You can do that by printing certificate in text form with command like this:
openssl x509 -text -noout -in server.crtand look for "X509v3 Basic Constraints" section. For example, my test server's certificate looks like this (only relevant part of the output):
X509v3 extensions: X509v3 Basic Constraints: CA:TRUEBest regards,
Victor -
@victor said in Problem creating tunnel between server and agent:
Please check that your server certificate has CA constraint set to TRUE. You can do that by printing certificate in text form with command like this:
openssl x509 -text -noout -in server.crtand look for "X509v3 Basic Constraints" section. For example, my test server's certificate looks like this (only relevant part of the output):
X509v3 extensions: X509v3 Basic Constraints: CA:TRUEHi Victor,
there's no "X509v3 Basic Contraints" section in my certificate at all...
I will try to make a new cert during the next days and try again.
Best...
Detlev -
Hi Victor,
I have created a new certificate containing the BasicConstaint:
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUEBut the result is still the same when I try to bind the tunnel.
Excerpt from agent.log:
2020.06.05 06:02:02.529 *D* [tunnel ] 192.168.10.6: Resetting tunnel
2020.06.05 06:02:02.532 *D* [tunnel ] 192.168.10.6: Certificate and private key loaded
2020.06.05 06:02:02.532 *D* [ssl ] SSL handshake start (before SSL initialization)
2020.06.05 06:02:02.532 *D* [ssl ] SSL_connect: before SSL initialization
2020.06.05 06:02:02.532 *D* [ssl ] SSL_connect: SSLv3/TLS write client hello
2020.06.05 06:02:02.532 *D* [ssl ] SSL_connect: error in SSLv3/TLS write client hello
2020.06.05 06:02:02.534 *D* [ssl ] SSL_connect: SSLv3/TLS write client hello
2020.06.05 06:02:02.534 *D* [ssl ] SSL_connect: SSLv3/TLS read server hello
2020.06.05 06:02:02.534 *D* [ssl ] SSL_connect: TLSv1.3 read encrypted extensions
2020.06.05 06:02:02.534 *D* [ssl ] SSL_connect: SSLv3/TLS read server certificate request
2020.06.05 06:02:02.534 *D* [ssl ] SSL_connect: SSLv3/TLS read server certificate
2020.06.05 06:02:02.534 *D* [ssl ] SSL_connect: TLSv1.3 read server certificate verify
2020.06.05 06:02:02.534 *D* [ssl ] SSL_connect: SSLv3/TLS read finished
2020.06.05 06:02:02.534 *D* [ssl ] SSL_connect: SSLv3/TLS write change cipher spec
2020.06.05 06:02:02.534 *D* [ssl ] SSL_connect: SSLv3/TLS write client certificate
2020.06.05 06:02:02.536 *D* [ssl ] SSL_connect: SSLv3/TLS write certificate verify
2020.06.05 06:02:02.536 *D* [ssl ] SSL_connect: SSLv3/TLS write finished
2020.06.05 06:02:02.536 *D* [ssl ] SSL handshake done (SSL negotiation finished successfully)
2020.06.05 06:02:02.536 *D* [tunnel ] 192.168.10.6: Server certificate subject is /C=DE/ST=NW/L=Bornheim/O=TILS GmbH/OU=IT/CN=netxms.tilsnet.de
2020.06.05 06:02:02.536 *D* [tunnel ] 192.168.10.6: Server certificate issuer is /DC=de/DC=tilsnet/CN=tilsnet-CAROOT
2020.06.05 06:02:02.537 *D* [ ] TlsMessageReceiver: SSL_read error (ssl_err=5 errno=0)
2020.06.05 06:02:02.537 *D* [tunnel ] 192.168.10.6: Receiver thread stopped (MSGRECV_COMM_FAILURE)
2020.06.05 06:02:02.537 *D* [comm.vs.8 ] Requesting parameter "System.PlatformName"
2020.06.05 06:02:02.537 *W* [ ] Tunnel with 192.168.10.6 closed
2020.06.05 06:02:02.537 *D* [comm.vs.8 ] GetParameterValue(): result is 0 (SUCCESS)
2020.06.05 06:02:02.537 *D* [comm.vs.8 ] Requesting parameter "System.UName"
2020.06.05 06:02:02.537 *D* [comm.vs.8 ] GetParameterValue(): result is 0 (SUCCESS)
2020.06.05 06:02:12.537 *D* [tunnel ] 192.168.10.6: Cannot configure tunnel (request timeout)Do you have any further ideas ?
Thanks & regards
Detlev -
How server side log looks like for that tunnel? Also, please try this agent version: https://netxms.org/download/releases/3.3/nxagent-3.3.350-x64.exe - it fixes incorrect error code display in "SSL read error" message so we could see actual socket error code that could provide some clue.
Another thought - could it be that you have some kind of DPI device between agent and server that detects SSL handshake and blocks connection for some reason?Best regards,
Victor -
@victor said in Problem creating tunnel between server and agent:
How server side log looks like for that tunnel? Also, please try this agent version: https://netxms.org/download/releases/3.3/nxagent-3.3.350-x64.exe - it fixes incorrect error code display in "SSL read error" message so we could see actual socket error code that could provide some clue.
Another thought - could it be that you have some kind of DPI device between agent and server that detects SSL handshake and blocks connection for some reason?Hi Victor,
I have updated the agent to 3.3.350 and tried again with Debuglevel 7 on both, agent side and server.
Please refer to attached logs for the errors.
The NetXMS server has 4 network cards (one for each monitored subnet) to prevent passing the firewall. The simple network routing is: Agent-PC -> HPE/Aruba-Switch -> ESXi Host -> Server VM.
I also tried to bind a tunnel between the agent on the server vm and the netxms server. So data is not leaving the vm at all. In this configuration I see a second error in the agent log:
SSL_write_error (bytes=-1 ssl_err=5 errno=2)
TlsMessageReceiver: SSL_read error (ssl_err=5 errno=0)Thanks & regards
Detlev
-
Are the subnet masks the same on the agent and the server?
If they are then you may have a problem with a duplicate subnet if they are not 'zoned'.
Most tunnelling cannot work with the same subnet on each end unless it is in a 'bridge' mode.just my 2 cents.
-
From server log it looks like certificate validation error. One of common reasons is incomplete certificate chain. Did you provide root CA certificate and all intermediate CA certificates in server configuration?
Best regards,
Victor -
@victor said in Problem creating tunnel between server and agent:
Did you provide root CA certificate and all intermediate CA certificates in server configuration?
Hi Victor,
yes - I have and hopefully in the correct way..:
ServerCACertificate = v:\netxms\etc\ca.cer (contains the domain's root certificate "CAROOT")
ServerCertificate = c:\netxms\etc\cert.cer (contains the netxms server's certificate, issued by "CAROOT")
ServerCertificateKey = c:\netxms\etc\netxms.key (contains the private key for the server's certificate)there's no intermediate CA in our system...
Best...
Detlev -
Is it typo when copying here or actual typo in your config, or correct configuration (v: and c:)?
ServerCACertificate = v:\netxms\etc\ca.cer ServerCertificate = c:\netxms\etc\cert.cerAlso, are there any messages about certificate loading on server startup?
Best regards,
Victor -
@victor said in Problem creating tunnel between server and agent:
Is it typo when copying here or actual typo in your config, or correct configuration (v: and c:)?
ServerCACertificate = v:\netxms\etc\ca.cer ServerCertificate = c:\netxms\etc\cert.cerAlso, are there any messages about certificate loading on server startup?
Hi Victor,
the drive letter is just a typo... I have disabled copy function between vm console and my desktop, so I quickly typed it in.
The server's log is showing:
2020.06.22 05:54:54.459 *I* [ ] Crypto library initialized (OpenSSL 1.1.1g 21 Apr 2020)
2020.06.22 05:54:54.459 *D* [ ] Server certificate loaded
2020.06.22 05:54:54.459 *D* [ ] Using server certificate keyand the agent receives it correctly when establishing the unbound tunnel
*D* [tunnel ] 192.168.10.6: Server certificate subject is /C=DE/ST=NW/.....
*D* [tunnel ] 192.168.10.6: Server certificate issuer is /DC=de/DC=.....Best...
Detlev -
What is debug level? You should see lines like "Adding CA certificate ..." on level 3 and more on level 5.
Best regards,
Victor -
@victor said in Problem creating tunnel between server and agent:
What is debug level? You should see lines like "Adding CA certificate ..." on level 3 and more on level 5.
I have started a clean new attempt in DebugLevel 5 with SSLTrace switched on.
The agent sees the certificate of the server, and receives new certificates from server...
I have attached the agent log for your reference. It shows the "clean" start until the bind to node fails and a restart of agent after that...
Regards
Detlev
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login