NetXMS searching for DCI on wrong nodes

[adimitrov]

Hello team ,

Another issue with the database we have.
In the logs we are observing a lot of these:

root@nms3:~# tail -f /var/log/netxms | grep 128994
[13-Nov-2016 09:57:41.336] [DEBUG] AgentConnectionEx::processBulkCollectedData: cannot find DCI with ID 128994 on node R7-******* [58915] (element 421)
[13-Nov-2016 09:58:06.033] [DEBUG] AgentConnectionEx::processBulkCollectedData: cannot find DCI with ID 128994 on node R112-******* [57842] (element 327)
[13-Nov-2016 09:58:20.910] [DEBUG] AgentConnectionEx::processBulkCollectedData: cannot find DCI with ID 128994 on node R10-******* [54272] (element 547)
[13-Nov-2016 09:58:36.890] [DEBUG] AgentConnectionEx::processBulkCollectedData: cannot find DCI with ID 128994 on node R200-******* [63335] (element 519)
[13-Nov-2016 09:58:55.141] [DEBUG] AgentConnectionEx::processBulkCollectedData: cannot find DCI with ID 128994 on node R112-******* [57842] (element 404)
[13-Nov-2016 09:59:10.085] [DEBUG] AgentConnectionEx::processBulkCollectedData: cannot find DCI with ID 128994 on node DRC-******* [58147] (element 633)
[13-Nov-2016 09:59:29.620] [DEBUG] AgentConnectionEx::processBulkCollectedData: cannot find DCI with ID 128994 on node R10-******* [54272] (element 719)

it seems that netxms is looking for this DCI on different nodes when the DCI actually is created and belong to another node.

sqlite> select * from dc_config where dci_id = 128994;
3095327485888869043|128994|1|1|ServiceCheck.Custom(***.***.***.*** ,8000)|300|1479021648|0|00000000-0000-0000-0000-000000000000|0
sqlite> select * from dc_queue where dci_id = 128994;
3095327485888869043|128994|1|1|00000000-0000-0000-0000-000000000000|1479022548|0|0
3095327485888869043|128994|1|1|00000000-0000-0000-0000-000000000000|1479022848|0|0
3095327485888869043|128994|1|1|00000000-0000-0000-0000-000000000000|1479023148|0|0
3095327485888869043|128994|1|1|00000000-0000-0000-0000-000000000000|1479023448|0|0
3095327485888869043|128994|1|1|00000000-0000-0000-0000-000000000000|1479023748|0|0
3095327485888869043|128994|1|1|00000000-0000-0000-0000-000000000000|1479024048|0|0

This is happening for a lot of DCI's for different clients and i cannot find out why, any ideas?

Best regards,
Adrian

[victor]

Hi,

are those nodes related somehow (proxy, source node for DCI, same zone)?

Best regards,
Victor

[adimitrov]

Hello Victor ,

Yes they are related to same zones and are monitored through same agent.

Best regards,
Adrian

[victor]

Hi,

do you have source node set for this DCI? Part of agent database that you show - is it from correct node or from node where this DCI should not be?

Best regards,
Victor