Node gets deleted automatically after some time

Woody

Hello,
I have the problem that a node gets automatically deleted after some time.
Here is some information about my system:

Server: Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-130-generic x86_64)
NetXMS version:

root@netxms:~# netxmsd -v
NetXMS Server Version 3.7.95 Build 3.7-95-ga7a4f1d5c7 (UNICODE)
NXCP: 5.53.1.29 (AES-256, Blowfish-256, 3DES, AES-128, Blowfish-128)
Built with: g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

The name of the node is: Haas HP DL380 Gen9 (iLO)

When I serach for the name in the log file i see the following errors:

2020.06.06 10:49:47.781 *I* [                   ] NetXMS Server started
2020.06.06 10:51:48.635 *W* [                   ] Failed to execute transformation script for object Haas HP DL380 Gen9 (iLO) [118319] DCI Dummy [290984] (Error 17 in line 8: Argument is not of string type and cannot be converted to string)
2020.06.06 10:51:49.923 *E* [db.drv             ] SQL query failed (Query = "INSERT INTO interface_address_list (iface_id,ip_addr,ip_netmask) VALUES (?,?,?)"): Deadlock found when trying to get lock; try restarting transaction
2020.06.06 10:52:51.877 *E* [db.drv             ] SQL query failed (Query = "INSERT INTO items (node_id,template_id,name,source,datatype,polling_interval,retention_time,status,delta_calculation,transformation,description,instance,template_item_id,flags,resource_id,proxy_node,base_units,unit_multiplier,custom_units_name,perftab_settings,system_tag,snmp_port,snmp_raw_value_type,instd_method,instd_data,instd_filter,samples,comments,guid,npe_name,instance_retention_time,grace_period_start,related_object,polling_interval_src,retention_time_src,polling_schedule_type,retention_type,snmp_version,item_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) ON DUPLICATE KEY UPDATE node_id=VALUES(node_id),template_id=VALUES(template_id),name=VALUES(name),source=VALUES(source),datatype=VALUES(datatype),polling_interval=VALUES(polling_interval),retention_time=VALUES(retention_time),status=VALUES(status),delta_calculation=VALUES(delta_calculation),transformation=VALUES(transformation),description=VALUES(description),instance=VALUES(instance),template_item_id=VALUES(template_item_id),flags=VALUES(flags),resource_id=VALUES(resource_id),proxy_node=VALUES(proxy_node),base_units=VALUES(base_units),unit_multiplier=VALUES(unit_multiplier),custom_units_name=VALUES(custom_units_name),perftab_settings=VALUES(perftab_settings),system_tag=VALUES(system_tag),snmp_port=VALUES(snmp_port),snmp_raw_value_type=VALUES(snmp_raw_value_type),instd_method=VALUES(instd_method),instd_data=VALUES(instd_data),instd_filter=VALUES(instd_filter),samples=VALUES(samples),comments=VALUES(comments),guid=VALUES(guid),npe_name=VALUES(npe_name),instance_retention_time=VALUES(instance_retention_time),grace_period_start=VALUES(grace_period_start),related_object=VALUES(related_object),polling_interval_src=VALUES(polling_interval_src),retention_time_src=VALUES(retention_time_src),polling_schedule_type=VALUES(polling_schedule_type),retention_type=VALUES(retention_type),snmp_version=VALUES(snmp_version),item_id=VALUES(item_id)"): Deadlock found when trying to get lock; try restarting transaction
2020.06.06 10:52:52.335 *E* [db.drv             ] SQL query failed (Query = "INSERT INTO items (node_id,template_id,name,source,datatype,polling_interval,retention_time,status,delta_calculation,transformation,description,instance,template_item_id,flags,resource_id,proxy_node,base_units,unit_multiplier,custom_units_name,perftab_settings,system_tag,snmp_port,snmp_raw_value_type,instd_method,instd_data,instd_filter,samples,comments,guid,npe_name,instance_retention_time,grace_period_start,related_object,polling_interval_src,retention_time_src,polling_schedule_type,retention_type,snmp_version,item_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) ON DUPLICATE KEY UPDATE node_id=VALUES(node_id),template_id=VALUES(template_id),name=VALUES(name),source=VALUES(source),datatype=VALUES(datatype),polling_interval=VALUES(polling_interval),retention_time=VALUES(retention_time),status=VALUES(status),delta_calculation=VALUES(delta_calculation),transformation=VALUES(transformation),description=VALUES(description),instance=VALUES(instance),template_item_id=VALUES(template_item_id),flags=VALUES(flags),resource_id=VALUES(resource_id),proxy_node=VALUES(proxy_node),base_units=VALUES(base_units),unit_multiplier=VALUES(unit_multiplier),custom_units_name=VALUES(custom_units_name),perftab_settings=VALUES(perftab_settings),system_tag=VALUES(system_tag),snmp_port=VALUES(snmp_port),snmp_raw_value_type=VALUES(snmp_raw_value_type),instd_method=VALUES(instd_method),instd_data=VALUES(instd_data),instd_filter=VALUES(instd_filter),samples=VALUES(samples),comments=VALUES(comments),guid=VALUES(guid),npe_name=VALUES(npe_name),instance_retention_time=VALUES(instance_retention_time),grace_period_start=VALUES(grace_period_start),related_object=VALUES(related_object),polling_interval_src=VALUES(polling_interval_src),retention_time_src=VALUES(retention_time_src),polling_schedule_type=VALUES(polling_schedule_type),retention_type=VALUES(retention_type),snmp_version=VALUES(snmp_version),item_id=VALUES(item_id)"): Deadlock found when trying to get lock; try restarting transaction
2020.06.06 10:52:52.657 *E* [db.drv             ] SQL query failed (Query = "INSERT INTO items (node_id,template_id,name,source,datatype,polling_interval,retention_time,status,delta_calculation,transformation,description,instance,template_item_id,flags,resource_id,proxy_node,base_units,unit_multiplier,custom_units_name,perftab_settings,system_tag,snmp_port,snmp_raw_value_type,instd_method,instd_data,instd_filter,samples,comments,guid,npe_name,instance_retention_time,grace_period_start,related_object,polling_interval_src,retention_time_src,polling_schedule_type,retention_type,snmp_version,item_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) ON DUPLICATE KEY UPDATE node_id=VALUES(node_id),template_id=VALUES(template_id),name=VALUES(name),source=VALUES(source),datatype=VALUES(datatype),polling_interval=VALUES(polling_interval),retention_time=VALUES(retention_time),status=VALUES(status),delta_calculation=VALUES(delta_calculation),transformation=VALUES(transformation),description=VALUES(description),instance=VALUES(instance),template_item_id=VALUES(template_item_id),flags=VALUES(flags),resource_id=VALUES(resource_id),proxy_node=VALUES(proxy_node),base_units=VALUES(base_units),unit_multiplier=VALUES(unit_multiplier),custom_units_name=VALUES(custom_units_name),perftab_settings=VALUES(perftab_settings),system_tag=VALUES(system_tag),snmp_port=VALUES(snmp_port),snmp_raw_value_type=VALUES(snmp_raw_value_type),instd_method=VALUES(instd_method),instd_data=VALUES(instd_data),instd_filter=VALUES(instd_filter),samples=VALUES(samples),comments=VALUES(comments),guid=VALUES(guid),npe_name=VALUES(npe_name),instance_retention_time=VALUES(instance_retention_time),grace_period_start=VALUES(grace_period_start),related_object=VALUES(related_object),polling_interval_src=VALUES(polling_interval_src),retention_time_src=VALUES(retention_time_src),polling_schedule_type=VALUES(polling_schedule_type),retention_type=VALUES(retention_type),snmp_version=VALUES(snmp_version),item_id=VALUES(item_id)"): Deadlock found when trying to get lock; try restarting transaction
2020.06.06 10:52:52.667 *E* [db.drv             ] SQL query failed (Query = "INSERT INTO items (node_id,template_id,name,source,datatype,polling_interval,retention_time,status,delta_calculation,transformation,description,instance,template_item_id,flags,resource_id,proxy_node,base_units,unit_multiplier,custom_units_name,perftab_settings,system_tag,snmp_port,snmp_raw_value_type,instd_method,instd_data,instd_filter,samples,comments,guid,npe_name,instance_retention_time,grace_period_start,related_object,polling_interval_src,retention_time_src,polling_schedule_type,retention_type,snmp_version,item_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) ON DUPLICATE KEY UPDATE node_id=VALUES(node_id),template_id=VALUES(template_id),name=VALUES(name),source=VALUES(source),datatype=VALUES(datatype),polling_interval=VALUES(polling_interval),retention_time=VALUES(retention_time),status=VALUES(status),delta_calculation=VALUES(delta_calculation),transformation=VALUES(transformation),description=VALUES(description),instance=VALUES(instance),template_item_id=VALUES(template_item_id),flags=VALUES(flags),resource_id=VALUES(resource_id),proxy_node=VALUES(proxy_node),base_units=VALUES(base_units),unit_multiplier=VALUES(unit_multiplier),custom_units_name=VALUES(custom_units_name),perftab_settings=VALUES(perftab_settings),system_tag=VALUES(system_tag),snmp_port=VALUES(snmp_port),snmp_raw_value_type=VALUES(snmp_raw_value_type),instd_method=VALUES(instd_method),instd_data=VALUES(instd_data),instd_filter=VALUES(instd_filter),samples=VALUES(samples),comments=VALUES(comments),guid=VALUES(guid),npe_name=VALUES(npe_name),instance_retention_time=VALUES(instance_retention_time),grace_period_start=VALUES(grace_period_start),related_object=VALUES(related_object),polling_interval_src=VALUES(polling_interval_src),retention_time_src=VALUES(retention_time_src),polling_schedule_type=VALUES(polling_schedule_type),retention_type=VALUES(retention_type),snmp_version=VALUES(snmp_version),item_id=VALUES(item_id)"): Deadlock found when trying to get lock; try restarting transaction
2020.06.06 10:52:52.682 *E* [db.drv             ] SQL query failed (Query = "INSERT INTO items (node_id,template_id,name,source,datatype,polling_interval,retention_time,status,delta_calculation,transformation,description,instance,template_item_id,flags,resource_id,proxy_node,base_units,unit_multiplier,custom_units_name,perftab_settings,system_tag,snmp_port,snmp_raw_value_type,instd_method,instd_data,instd_filter,samples,comments,guid,npe_name,instance_retention_time,grace_period_start,related_object,polling_interval_src,retention_time_src,polling_schedule_type,retention_type,snmp_version,item_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) ON DUPLICATE KEY UPDATE node_id=VALUES(node_id),template_id=VALUES(template_id),name=VALUES(name),source=VALUES(source),datatype=VALUES(datatype),polling_interval=VALUES(polling_interval),retention_time=VALUES(retention_time),status=VALUES(status),delta_calculation=VALUES(delta_calculation),transformation=VALUES(transformation),description=VALUES(description),instance=VALUES(instance),template_item_id=VALUES(template_item_id),flags=VALUES(flags),resource_id=VALUES(resource_id),proxy_node=VALUES(proxy_node),base_units=VALUES(base_units),unit_multiplier=VALUES(unit_multiplier),custom_units_name=VALUES(custom_units_name),perftab_settings=VALUES(perftab_settings),system_tag=VALUES(system_tag),snmp_port=VALUES(snmp_port),snmp_raw_value_type=VALUES(snmp_raw_value_type),instd_method=VALUES(instd_method),instd_data=VALUES(instd_data),instd_filter=VALUES(instd_filter),samples=VALUES(samples),comments=VALUES(comments),guid=VALUES(guid),npe_name=VALUES(npe_name),instance_retention_time=VALUES(instance_retention_time),grace_period_start=VALUES(grace_period_start),related_object=VALUES(related_object),polling_interval_src=VALUES(polling_interval_src),retention_time_src=VALUES(retention_time_src),polling_schedule_type=VALUES(polling_schedule_type),retention_type=VALUES(retention_type),snmp_version=VALUES(snmp_version),item_id=VALUES(item_id)"): Deadlock found when trying to get lock; try restarting transaction
2020.06.06 10:53:02.238 *E* [db.drv             ] SQL query failed (Query = "INSERT INTO interface_address_list (iface_id,ip_addr,ip_netmask) VALUES (?,?,?)"): Deadlock found when trying to get lock; try restarting transaction
2020.06.06 10:55:41.384 *W* [                   ] Failed to execute transformation script for object @HR-Soft QNAP TS-259 [198] DCI .1.3.6.1.4.1.24681.1.3.17.1.1.1 [24748] (Error 5 in line 4: Invalid operation with NULL value)
2020.06.06 11:16:56.005 *W* [                   ] Failed to execute transformation script for object Haas HP DL380 Gen9 (iLO) [118345] DCI Dummy [291118] (Error 17 in line 8: Argument is not of string type and cannot be converted to string)
2020.06.06 11:17:52.642 *E* [db.drv             ] SQL query failed (Query = "INSERT INTO interface_address_list (iface_id,ip_addr,ip_netmask) VALUES (?,?,?)"): Deadlock found when trying to get lock; try restarting transaction
2020.06.06 11:18:13.515 *I* [                   ] NetXMS Server stopped by signal

I hope somebody can help me here.

Thanks!
Woody

Victor Kirhenshtein

Hi,

does it disappear after server restart or while server is running? If former then it could be that because of SQL errors node is not saved to DB at all. If latter then it could be due to deduplication if it is turned on (you can check it in server configuration - parameter NetworkDiscovery.MergeDuplicateNodes).

Best regards,
Victor

Woody

Hi,
it does disappear after some time. After a server restart the node was still there.
NetworkDiscovery.MergeDuplicateNodes is enabled.
But the node is no duplicate. I searched for the ip adress and it didn't find anything.

Best regards,
Woody

Victor Kirhenshtein

Try to turn debug to level 6 (be aware that it will generate lot of output if you setup is big), and after node disappear again filter all messages containing name or ID of that node and post them.

Best regards,
Victor

Woody

Hi,
I updated NetXMS:

root@netxms:~# netxmsd -v
NetXMS Server Version 3.7.116 Build 3.7-116-g133064d484 (UNICODE)
NXCP: 5.53.1.29 (AES-256, Blowfish-256, 3DES, AES-128, Blowfish-128)
Built with: g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

The problem still exists in this version.

I collected the logs and filtered only the lines with the name of the node.

netxmsdLog.txt

Victor Kirhenshtein

So it is deleted as duplicate:

2021.01.11 07:59:49.527 *D* [event.proc ] EVENT SYS_DUPLICATE_NODE_DELETED [100] at {0} (ID:30074352 F:0x0001 S:1 TAGS:"") FROM NetXMS Server: Duplicate node Haas HP DL380 Gen9 (iLO) is deleted (Primary IP address 10.59.204.205 of node Haas VMWareHostB [118318] found on interface HP Ethernet 1Gb 4-port 331T Adapter of node Haas HP DL380 Gen9 (iLO) [135819])

What is node "Haas VMWareHostB" and do it really have same IP address?

Best regards,
Victor

Woody

Hi,
Thank you for helping. The problem was, that i use 1 node for HP iLO which has 8 Network Interfaces, one of the Network interfaces has the same IP Address of the ESXi VMWare Host which is installed on this Server. So the IP-Adress is a duplicate and the new Node was deleted.
I changed the Server Property to: NetworkDiscovery.MergeDuplicateNodes = 0
Now I think it works!
Thank you so much for helping!

Bye
Woody

NetXMS Forum

Node gets deleted automatically after some time