NetXMS Forum

isoftcom

OS version - AIX 7.2
the version of netxms is 2.0.4
i cleared the logs to have fresh logs so i detect the problem. by event logs you mean netxmsd logs right?

some logs include the following

-----------------------------------------------------------------
[21-Jan-2019 07:24:50.357] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:24:50.909] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:24:56.374] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:30:01.469] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:30:01.473] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 07:34:50.389] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:34:50.885] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:34:52.000] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:36:41.570] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:37:21.593] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:38:01.618] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:38:41.632] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:39:01.649] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:40:01.675] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:41:21.702] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:44:01.767] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:47:32.390] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:49:01.828] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:50:01.831] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:50:06.598] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:51:01.849] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:54:01.918] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:54:41.924] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:55:21.936] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:58:01.971] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:59:21.995] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:00:01.078] [DEBUG] IST clean syslg history called
[21-Jan-2019 08:00:02.000] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:01:02.018] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:02:42.077] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:03:42.096] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:04:22.101] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:05:22.109] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:07:02.148] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:14:02.234] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:14:42.260] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:15:42.292] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:16:22.307] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:18:02.327] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:19:02.345] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:20:22.362] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:20:54.132] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:20:54.137] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:22:42.407] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:24:22.453] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:26:42.511] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 08:28:22.531] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:29:22.548] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:29:59.587] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 08:30:42.565] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:33:02.599] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:33:42.610] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:34:42.624] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:35:02.631] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:36:02.654] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:37:22.683] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:39:42.724] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:40:22.747] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:41:42.765] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:41:42.766] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:42:42.826] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:44:22.867] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:46:22.920] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:47:42.952] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:49:02.972] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:49:36.626] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:49:36.627] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:50:02.994] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:52:57.180] [DEBUG] Database Connection Pool: connection 1106bc590 reconnected
[21-Jan-2019 08:52:57.847] [DEBUG] Database Connection Pool: connection 11033b730 reconnected
[21-Jan-2019 08:52:58.788] [DEBUG] Database Connection Pool: connection 1103301b0 reconnected
[21-Jan-2019 08:52:59.681] [DEBUG] Database Connection Pool: connection 1100c8d10 reconnected
[21-Jan-2019 08:52:59.970] [DEBUG] Database Connection Pool: connection 1100eedd0 reconnected
[21-Jan-2019 08:53:23.051] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:54:23.062] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:54:37.828] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 08:54:40.867] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:00:23.134] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 09:03:03.190] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 09:05:39.028] [DEBUG] IST clean syslg history called
[21-Jan-2019 09:05:40.744] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:05:41.990] [ERROR] SQL query failed (Query = "UPDATE raw_dci_values SET raw_value=?,transformed_value=?,last_poll_time=? WHERE item_id=?"): ORA-00060: deadlock detected while waiting for resource
[21-Jan-2019 09:05:42.309] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:10:43.309] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:11:52.536] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:11:52.650] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 09:11:52.786] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:12:23.353] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:13:43.374] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:14:43.393] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:18:43.447] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 09:19:43.459] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:22:03.488] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 09:25:23.555] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:26:23.576] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:27:03.589] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:28:43.676] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:31:43.746] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:33:03.763] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:33:03.767] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 09:34:23.795] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:36:23.823] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:41:43.884] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:43:03.928] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:43:11.050] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:43:43.955] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:44:43.975] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:47:44.027] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:49:04
----------------------------------------------

isoftcom

Hello Devs

I have been seeing this popping almost every after 30 seconds

DDCC client: Thread Item "Item poller" not responding

This is where Netxms is installed and other servers and its currently having a red x.

How do i sort this problem out?

isoftcom

i dont know why netxms server has become slow also it cant respond on DCI last values.

when i hit command below i see the following.

netxmsd: show watchdog
Thread Interval Status
----------------------------------------------------------------------------
Item Poller 20 Running
Syncer Thread 130 Not responding
Poll Manager 60 Not responding

i have attached netxmsd and an image of not responding.

Please help me detect the problem.

netxmsd.log1

isoftcom

Hi Alex

Thanks for the Help, let me try to do what you have advised and monitor the difference.

isoftcom

I understand that the tdata_ table holds the DCI value history, my database is big with a lot of objects in the system sitting on oracle 11g database. The problem is the value history are alot and are taking a lot of resources when pushing these history to the database. Is there way on how to tell netxms not to generate a lot of these history?

isoftcom

Hi All,

The Query "INSERT INTO tdata_122 (item_id,tdata_timestamp,record_id) VALUES (:1,:2,:3)" is very expensive, it causes all sorts of problems with the Database including the creation of deadlocks and waits. Isn't there a way we can optimize it so as to make is less expensive on resources.

isoftcom

Hi All,

The Query "INSERT INTO tdata_122 (item_id,tdata_timestamp,record_id) VALUES (:1,:2,:3)" is very expensive, it causes all sorts of problems with the Database including the creation of deadlocks and waits. Isn't there a way we can optimize it so as to make is less expensive on resources.

NetXMS Forum

isoftcom

Posts