I was able to resolve my issue (which, according to the logs, is the same as yours) and perhaps this will help. I'm going to start at the beginning even though you already have of the steps completed (since you get the tunnel to come up initially)
I'm on Windows 2008R2
I port forwarded TCP 4703 to my NetXMS server on my firewall
NetXMS server program is already added to Windows firewall (but check on this)
I have a Linux Mint Virtualbox that i used for the openssl commands. Windows openssl I imagine would work, but I had the Linux box and it was faster than formatting the provided openssl.conf files to a Windows install.
Follow creating the root pair as shown here:
https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html
Make sure you copy the config file from the Appendix as it is says in the directions
Follow creating the intermediate pair as shown here:
https://jamielinux.com/docs/openssl-certificate-authority/create-the-intermediate-pair.html
Make sure you copy the config file from the Appendix as it is says in the directions
When it says chmod 400, I did 444, so I could edit the files (later step)
You do not need to create the certificate chain file, as it will not be used (I tried, it doesn't work)
Edit the intermediate.cert.pem file and add the text from the intermediate.key.pem at the bottom so you have
-----BEGIN CERTIFICATE-----
stuff
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
stuff
-----END RSA PRIVATE KEY-----
I think you need a blank line at the end of the pem files. So make sure can put the cursor down under the -----END RSA PRIVATE KEY----- indicating there is a blank line there
Copy ca.cert.pem and intermediate.cert.pem to your NetXMS server. I put them in D:\NetXMS\etc\cert
Add the following to netxmsd.conf
ServerCACertificate = D:\NetXMS\etc\cert\ca.cert.pem
ServerCertificate = D:\NetXMS\etc\cert\intermediate.cert.pem
ServerCertificatePassword = secretpassword
Restart the NetXMS Core service
On the node/agent add the following to the nxagentd.conf where the x's are your public IP (or whatever the nodes will be tunneling to)
MasterServers = xxx.xxx.xxx.xxx
ServerConnection = xxx.xxx.xxx.xxx
Restart the NetXMS Agent service
In NetXMS Management Console go to Configuration/Agent Tunnel Manager. You should see the Unbound tunnel. Right click and create node and bind. I left the Primary host name blank. Although I have seen posts where you set it to 0.0.0.0 or the IP of the agent (not sure if that is the public or the local IP). When I set it to 0.0.0.0 it changes the name to not what I want. I left it blank, it has been working and I get the name I set.
Most of what I have here isn't new. I just consolidated it into one place.
This post was a big help https://www.netxms.org/forum/configuration/agent-to-server-connections/msg22308/#msg22308
