NetXMS Forum

Hello! It's been a long time since I've posted here. Regardless....I just upgraded from 4.4.4 to 5.1.2 and now SNMP traps are no longer processed.

I was running 4.4.4 in a Debian container on Proxmox. I created a new Debian container and did a fresh install of 5.1.2. I upgraded the database via the nxdbmgr and updated my scripts with the new OID syntax. My SNMP polling of devices is working correctly, however, NetXMS is not processing the SNMP traps. 

1. I verified via packet capture that the server is receiving the traps
2. netstat -an shows that NetXMS is listening on port 162
3. The firewall was temporarily set to "allow all"
4. I verified that my server configuration variables had not changed
5. The SNMP trap monitor in the client doesn't display any of the incoming traps, and the event monitor doesn't show anything either.

I haven't tried any other 5.x versions as I was wanting to update to the latest and greatest.

Any ideas?

Here is a CSV export of the FDB entry for the workstation:

"MAC Address","Port","Interface","VLAN","Node","Type"
"00:23:24:6C:89:04","95","[0]","500","Test Workstation","static"

The workstation is actually on Gi2/0/39.  My workstation is plugged into a Polycom IP phone, so Gi2/0/39 has both MAC addresses in the mac address table.  The port is an access port (we don't currently separate voice into a dedicated VLAN).

Here's the entry for the phone:

"MAC Address","Port","Interface","VLAN","Node","Type"
"00:04:F2:9D:B0:F2","95","[0]","500","","static"

From the switch:

[26.01.2016 09:34:56] **** Poll request sent to server ****
[26.01.2016 09:34:56] Poll request accepted
[26.01.2016 09:34:56] Starting topology poll for node IT
[26.01.2016 09:34:58] VLAN list successfully retrieved from node
[26.01.2016 09:35:01] Switch forwarding database retrieved
[26.01.2016 09:35:02] Link layer topology retrieved (2 connections found)
[26.01.2016 09:35:02] Updating peer information on interfaces
[26.01.2016 09:35:02]    Local interface Gi1/0/50 linked to remote interface UPLINK SWITCH1:Ethernet2/21
[26.01.2016 09:35:02]    Local interface Gi2/0/50 linked to remote interface UPLINK SWITCH2:Ethernet2/21
[26.01.2016 09:35:02] Link layer topology processed
[26.01.2016 09:35:02] Finished topology poll for node IT
[26.01.2016 09:35:02] **** Poll completed successfully ****

And from the node object that is my workstation:

[26.01.2016 09:36:44] **** Poll request sent to server ****
[26.01.2016 09:36:44] Poll request accepted
[26.01.2016 09:36:44] Starting topology poll for node Test Workstation
[26.01.2016 09:36:44] Failed to get switch forwarding database
[26.01.2016 09:36:44] Link layer topology retrieved (0 connections found)
[26.01.2016 09:36:44] Updating peer information on interfaces
[26.01.2016 09:36:44] Link layer topology processed
[26.01.2016 09:36:44] Finished topology poll for node Test Workstation
[26.01.2016 09:36:44] **** Poll completed successfully ****

I hope that's what you were looking for.

Ah, going that route it shows all of the MAC addresses.  Although the "Find IP Address" tool still shows the indirectly attached port.

No, it's not all of the MAC addresses.  There are 3 or 4 VLANs on the switch.  I attached the output of one of them.

snmpwalk.txt

[netxms@netxms ~]# snmpwalk -v 2c -Of -c "SUPERSECRETSTRING" THE_IP_ADDRESS .1.3.6.1.2.1.17.4.3.1.1
.iso.org.dod.internet.mgmt.mib-2.dot1dBridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbAddress.'..B...' = STRING: 0:3:42:f4:ef:81
.iso.org.dod.internet.mgmt.mib-2.dot1dBridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbAddress.'.....`' = STRING: 0:1a:1e:0:9a:60
.iso.org.dod.internet.mgmt.mib-2.dot1dBridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbAddress.'......' = STRING: 0:1a:1e:0:9a:b0
.iso.org.dod.internet.mgmt.mib-2.dot1dBridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbAddress.'.&....' = STRING: 0:26:f0:dd:0:0
.iso.org.dod.internet.mgmt.mib-2.dot1dBridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbAddress.'.*j..B' = STRING: 0:2a:6a:a4:f0:42
.iso.org.dod.internet.mgmt.mib-2.dot1dBridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbAddress.'.*j.l.' = STRING: 0:2a:6a:a5:6c:c2
.iso.org.dod.internet.mgmt.mib-2.dot1dBridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbAddress.'(..^.h' = STRING: 28:c7:ce:5e:dd:68
.iso.org.dod.internet.mgmt.mib-2.dot1dBridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry.dot1dTpFdbAddress.'(..^.X' = STRING: 28:c7:ce:5e:de:58
[netxms@netxms ~]#

It shows: Driver=CATALYST-GENERIC

The static entries are NOT in the FDB table.  In fact, the FDB table only shows 7 MAC addresses.  All of the MAC addresses that it shows are DYNAMIC, but not all of the DYNAMIC entries on the switch are displayed in the FDB table.

Yes, I'm running 2.0.1.  I looked up my own MAC address and it shows me indirectly connected to another switch (my VLAN extends to a Juniper switch), rather than my directly connected Cisco 2960X (running 15.2(3)E2).  We enable port security on all switch ports in order to limit the number of MAC addresses learned and to prevent CAM table overflows.  We don't actually statically assign MAC addresses to ports, but a look at the MAC address table shows almost all MAC addresses as "STATIC".  The only DYNAMIC ones are the MAC's learned on the uplink port-channel (which doesn't have port security enabled).  Here's a snippet:

SWITCHNAME#show mac address-table | include STATIC
 All    0100.0ccc.cccc    STATIC      CPU
 All    0100.0ccc.cccd    STATIC      CPU
 All    0100.0ccd.cddc    STATIC      CPU
 All    0180.c200.0000    STATIC      CPU
 All    0180.c200.0001    STATIC      CPU
 All    0180.c200.0002    STATIC      CPU
 All    0180.c200.0003    STATIC      CPU
 All    0180.c200.0004    STATIC      CPU
 All    0180.c200.0005    STATIC      CPU
 All    0180.c200.0006    STATIC      CPU
 All    0180.c200.0007    STATIC      CPU
 All    0180.c200.0008    STATIC      CPU
 All    0180.c200.0009    STATIC      CPU
 All    0180.c200.000a    STATIC      CPU
 All    0180.c200.000b    STATIC      CPU
 All    0180.c200.000c    STATIC      CPU
 All    0180.c200.000d    STATIC      CPU
 All    0180.c200.000e    STATIC      CPU
 All    0180.c200.000f    STATIC      CPU
 All    0180.c200.0010    STATIC      CPU
 All    ffff.ffff.ffff    STATIC      CPU
 509    00a0.0304.8cfe    STATIC      Gi3/0/31
 509    00c0.e403.31dc    STATIC      Gi3/0/40
   3    00c0.b784.5333    STATIC      Gi3/0/38
   3    00c0.b7a1.e885    STATIC      Gi3/0/32
   3    00c0.b7b8.ccaf    STATIC      Gi3/0/37
   3    00c0.b7b8.d0a2    STATIC      Gi3/0/36
  30    0001.e65c.89b1    STATIC      Gi1/0/17
  30    001a.4b1f.f7ae    STATIC      Gi3/0/34
  30    001b.780d.090f    STATIC      Gi2/0/7
  30    0026.732f.b2ae    STATIC      Gi1/0/2
  30    308d.99ab.29d7    STATIC      Gi1/0/37
  30    5820.b14c.a5c4    STATIC      Gi2/0/2
  30    7446.a04e.7867    STATIC      Gi3/0/33
  30    f4ce.463d.0b3f    STATIC      Gi1/0/1
  14    0004.f296.dda5    STATIC      Gi2/0/36
  14    0004.f29a.da22    STATIC      Gi1/0/15
  14    0004.f29b.1196    STATIC      Gi1/0/21
  14    0004.f29d.22af    STATIC      Gi2/0/34

I followed the link and saw that the feature request is now closed.  What became of this feature? I'm encountering the same thing myself.  Is there a configuration option I need to change?

Ooops, I should have included that info in the original post.  We are using MS SQL Server 2014.

I'm getting this error message in my logs after upgrading from 1.2.17 to 2.0.1.

[13-Jan-2016 15:31:14.661] [ERROR] SQL query failed (Query = "INSERT INTO snmp_trap_log (trap_id,trap_timestamp,ip_addr,object_id,trap_oid,trap_varlist) VALUES (15678756,1452717074,'192.168.2.2',18337,'.1.3.6.1.4.1.9.9.315.0.0.1','.1.3.6.1.2.1.2.2.1.1.10141 == ''10141''; .1.3.6.1.2.1.31.1.1.1.1.10141 == ''GigabitEthernet1/0/41''; .1.3.6.1.4.1.9.9.315.1.2.1.1.10.10141 == ''ðÞñü~â''')"): Unclosed quotation mark after the character string '.1.3.6.1.2.1.2.2.1.1.10141 == '10141'; .1.3.6.1.2.1.31.1.1.1.1.10141 == 'GigabitEthernet1/0/41'; .1.3.6.1.4.1.9.9.315.1.2.1.1.10.10141 == ''.

It looks like the quote characters are messing up the query.  Is there something I need to do to correct this, or is there a work around?

Thanks for your help.  I'll download the new version tomorrow, then.

Hopefully this is what you need.

TABLE_QUALIFIER TABLE_OWNER TABLE_NAME COLUMN_NAME DATA_TYPE TYPE_NAME PRECISION LENGTH SCALE RADIX NULLABLE REMARKS COLUMN_DEF SQL_DATA_TYPE SQL_DATETIME_SUB CHAR_OCTET_LENGTH ORDINAL_POSITION IS_NULLABLE SS_DATA_TYPE
netxms dbo items item_id 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 1 NO 56
netxms dbo items node_id 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 2 NO 56
netxms dbo items template_id 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 3 NO 56
netxms dbo items template_item_id 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 4 NO 56
netxms dbo items name 12 varchar 1023 1023 NULL NULL 1 NULL NULL 12 NULL 1023 5 YES 39
netxms dbo items description 12 varchar 255 255 NULL NULL 1 NULL NULL 12 NULL 255 6 YES 39
netxms dbo items flags 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 7 NO 56
netxms dbo items source 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 8 NO 56
netxms dbo items snmp_port 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 9 NO 56
netxms dbo items datatype 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 10 NO 56
netxms dbo items polling_interval 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 11 NO 56
netxms dbo items retention_time 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 12 NO 56
netxms dbo items status 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 13 NO 56
netxms dbo items snmp_raw_value_type 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 14 NO 56
netxms dbo items delta_calculation 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 15 NO 56
netxms dbo items transformation -1 text 2147483647 2147483647 NULL NULL 0 NULL NULL -1 NULL 2147483647 16 NO 35
netxms dbo items instance 12 varchar 255 255 NULL NULL 1 NULL NULL 12 NULL 255 17 YES 39
netxms dbo items system_tag 12 varchar 255 255 NULL NULL 1 NULL NULL 12 NULL 255 18 YES 39
netxms dbo items resource_id 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 19 NO 56
netxms dbo items proxy_node 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 20 NO 56
netxms dbo items base_units 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 21 NO 56
netxms dbo items unit_multiplier 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 22 NO 56
netxms dbo items custom_units_name 12 varchar 63 63 NULL NULL 1 NULL NULL 12 NULL 63 23 YES 39
netxms dbo items perftab_settings -1 text 2147483647 2147483647 NULL NULL 1 NULL NULL -1 NULL 2147483647 24 YES 35
netxms dbo items instd_method 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 25 NO 56
netxms dbo items instd_data 12 varchar 255 255 NULL NULL 1 NULL NULL 12 NULL 255 26 YES 39
netxms dbo items instd_filter -1 text 2147483647 2147483647 NULL NULL 1 NULL NULL -1 NULL 2147483647 27 YES 35
netxms dbo items samples 4 int 10 4 0 10 0 NULL NULL 4 NULL NULL 28 NO 56
netxms dbo items comments -1 text 2147483647 2147483647 NULL NULL 1 NULL NULL -1 NULL 2147483647 29 YES 35

It looks like a possible issue with the DCItem::saveToDB function.  Here is the debug output from one attempt at binding a template to a node.  It looks like the "INSERT INTO items" query is off by one parameter since 15943 is the template id and not the node id:

[14-Oct-2014 16:49:33.119] [DEBUG] Successful sync query: "INSERT INTO dct_node_map (template_id,node_id) VALUES (15943,8168)" [1 ms]
[14-Oct-2014 16:49:33.119] [DEBUG] Successful sync query: "INSERT INTO dct_node_map (template_id,node_id) VALUES (15943,10760)" [0 ms]
[14-Oct-2014 16:49:33.120] [DEBUG] Successful sync query: "INSERT INTO dct_node_map (template_id,node_id) VALUES (15943,10909)" [1 ms]
[14-Oct-2014 16:49:33.120] [DEBUG] Successful sync query: "INSERT INTO dct_node_map (template_id,node_id) VALUES (15943,11082)" [0 ms]
[14-Oct-2014 16:49:33.121] [DEBUG] Successful sync query: "INSERT INTO dct_node_map (template_id,node_id) VALUES (15943,11255)" [1 ms]
[14-Oct-2014 16:49:33.121] [DEBUG] Successful sync query: "INSERT INTO dct_node_map (template_id,node_id) VALUES (15943,11384)" [0 ms]
[14-Oct-2014 16:49:33.122] [DEBUG] Successful sync query: "INSERT INTO dct_node_map (template_id,node_id) VALUES (15943,11765)" [1 ms]
[14-Oct-2014 16:49:33.122] [DEBUG] Successful sync query: "DELETE FROM acl WHERE object_id=15943" [0 ms]
[14-Oct-2014 16:49:33.122] [DEBUG] {0x7f9e68018130} Successful prepare: "SELECT item_id FROM items WHERE item_id=?" [0 ms]
[14-Oct-2014 16:49:33.122] [DEBUG] {0x7f9e68018130} bind at pos 1: "282"
[14-Oct-2014 16:49:33.122] [DEBUG] Successful prepared sync query: "SELECT item_id FROM items WHERE item_id=?" [0 ms]
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} Successful prepare: "INSERT INTO items (node_id,template_id,name,source,datatype,polling_interval,retention_time,status,delta_calculation,transformation,description,instance,template_item_id,flags,resource_id,proxy_node,base_units,unit_multiplier,custom_units_name,perftab_settings,system_tag,snmp_port,snmp_raw_value_type,instd_method,instd_data,instd_filter,samples,comments,item_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" [0 ms]
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 1: "15943"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 2: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 3: "Dummy"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 4: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 5: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 6: "300"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 7: "5"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 8: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 9: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 10: "use US::Utilities;
use US::CiscoCommon;

return ciscoEnvMonTemperatureStatusValue($node->name);"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 11: "Temperature"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 12: ""
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 13: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 14: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 15: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 16: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 17: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 18: "1"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 19: ""
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 20: "<config>
  <enabled>false</enabled>
  <autoScale>true</autoScale>
  <logScaleEnabled>false</logScaleEnabled>
  <minYScaleValue>1</minYScaleValue>
  <maxYScaleValue>100</maxYScaleValue>
  <timeRange>1</timeRange>
  <timeUnits>1</timeUnits>
  <type>0</type>
  <color>49152</color>
  <title></title>
  <name></name>
  <showThresholds>false</showThresholds>
  <parentDciId>0</parentDciId>
  <order>100</order>
</config>"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 21: ""
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 22: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 23: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 24: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 25: ""
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 26: ""
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 27: "0"
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 28: ""
[14-Oct-2014 16:49:33.123] [DEBUG] {0x7f9e6804a350} bind at pos 29: "282"
[14-Oct-2014 16:49:33.124] [DEBUG] Failed prepared sync query: "INSERT INTO items (node_id,template_id,name,source,datatype,polling_interval,retention_time,status,delta_calculation,transformation,description,instance,template_item_id,flags,resource_id,proxy_node,base_units,unit_multiplier,custom_units_name,perftab_settings,system_tag,snmp_port,snmp_raw_value_type,instd_method,instd_data,instd_filter,samples,comments,item_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" [1 ms]
[14-Oct-2014 16:49:33.124] [ERROR] SQL query failed (Query = "INSERT INTO items (node_id,template_id,name,source,datatype,polling_interval,retention_time,status,delta_calculation,transformation,description,instance,template_item_id,flags,resource_id,proxy_node,base_units,unit_multiplier,custom_units_name,perftab_settings,system_tag,snmp_port,snmp_raw_value_type,instd_method,instd_data,instd_filter,samples,comments,item_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"): Operand type clash: int is incompatible with text
[14-Oct-2014 16:49:33.125] [DEBUG] Event::expandText(event=0x7f9e6804f620 sourceObject=100 template='Database query failed (Query: %1; Error: %2)' alarmMsg='(null)' alarmKey='(null)')
[14-Oct-2014 16:49:33.125] [DEBUG] CorrelateEvent: event SYS_DB_QUERY_FAILED id 2625549 source ourserver.ourdomain.com [100]

The long answer is that I am rebuilding our NetXMS monitoring server.  The previous one was in service for two years.  With all of the new features that have been added, in addition to more experience with NetXMS, I've found better, cleaner ways to do the same monitoring.  That being said, I had created 4 templates, each with 6-12 SNMP-based DCI's and a couple of them had thresholds set.  It appears that none of the DCI's are being saved to the database.

I have a bunch of these errors:

[13-Oct-2014 16:48:07.811] [ERROR] SQL query failed (Query = "INSERT INTO items (node_id,template_id,name,source,datatype,polling_interval,retention_time,status,delta_calculation,transformation,description,instance,template_item_id,flags,resource_id,proxy_node,base_units,unit_multiplier,custom_units_name,perftab_settings,system_tag,snmp_port,snmp_raw_value_type,instd_method,instd_data,instd_filter,samples,comments,item_id) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"): Operand type clash: int is incompatible with text

I'll have to check with our Sys Admins.  This is running on a MSSQL cluster and occasionally the database gets moved to another server by the cluster service.  I wonder if that factors in.  BTW: I'm running 1.2.16.

Twice now I've had the DCI's in multiple templates disappear.  The most recent occurrence was this past weekend.  I'm the only one with management access so I know it's not something I've done.  It's frustrating to have to re-create these DCI's.  This time I'm going to export them to ease re-creating them, but the bigger issue is why are they disappearing?  What do I need to do to debug this issue?

Cisco Stackwise MIB traps and events.

cisco-stackwise-mib.xml