<![CDATA[Linux kernel kills netxms process- out of memory after upgrade 1.2.14]]>Hi,

Something wierd is happing after I upgraded my 1.2.8 netxms on a VM running centos 5.

The server is running up to one day and then the process is killed by the kernel "out of memory", do you have an idea what is this happeing? before on version 1.2.8 I've never had this issue.

The network discovery is disable. Please find below the logs from the VM.

Thanks,

Christian Diaz

Jun 12 07:30:55 dashboard kernel: netxmsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jun 12 07:30:55 dashboard kernel: netxmsd cpuset=/ mems_allowed=0
Jun 12 07:30:55 dashboard kernel: Pid: 24776, comm: netxmsd Not tainted 2.6.32-358.14.1.el6.x86_64 #1
Jun 12 07:30:55 dashboard kernel: Call Trace:
Jun 12 07:30:55 dashboard kernel: [<ffffffff810cb561>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111cd80>] ? dump_header+0x90/0x1b0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8121d2ac>] ? security_real_capable_noaudit+0x3c/0x70
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111d202>] ? oom_kill_process+0x82/0x2a0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111d141>] ? select_bad_process+0xe1/0x120
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111d640>] ? out_of_memory+0x220/0x3c0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8112c2ec>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Jun 12 07:30:55 dashboard kernel: [<ffffffff811609ea>] ? alloc_pages_current+0xaa/0x110
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111a167>] ? __page_cache_alloc+0x87/0x90
Jun 12 07:30:55 dashboard kernel: [<ffffffff81119b4e>] ? find_get_page+0x1e/0xa0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111b127>] ? filemap_fault+0x1a7/0x500
Jun 12 07:30:55 dashboard kernel: [<ffffffff81143124>] ? __do_fault+0x54/0x530
Jun 12 07:30:55 dashboard kernel: [<ffffffff811436f7>] ? handle_pte_fault+0xf7/0xb50
Jun 12 07:30:55 dashboard kernel: [<ffffffff81063360>] ? wake_up_state+0x10/0x20
Jun 12 07:30:55 dashboard kernel: [<ffffffff810a9ae0>] ? wake_futex+0x40/0x60
Jun 12 07:30:55 dashboard kernel: [<ffffffff810aa35e>] ? futex_wake+0x10e/0x120
Jun 12 07:30:55 dashboard kernel: [<ffffffff8114438a>] ? handle_mm_fault+0x23a/0x310
Jun 12 07:30:55 dashboard kernel: [<ffffffff810474e9>] ? __do_page_fault+0x139/0x480
Jun 12 07:30:55 dashboard kernel: [<ffffffff81435fd5>] ? sys_sendto+0x185/0x190
Jun 12 07:30:55 dashboard kernel: [<ffffffff8150dd80>] ? thread_return+0x4e/0x76e
Jun 12 07:30:55 dashboard kernel: [<ffffffff815137be>] ? do_page_fault+0x3e/0xa0
Jun 12 07:30:55 dashboard kernel: [<ffffffff81510b75>] ? page_fault+0x25/0x30
Jun 12 07:30:55 dashboard kernel: Mem-Info:
Jun 12 07:30:55 dashboard kernel: Node 0 DMA per-cpu:
Jun 12 07:30:55 dashboard kernel: CPU    0: hi:    0, btch:  1 usd:  0
Jun 12 07:30:55 dashboard kernel: Node 0 DMA32 per-cpu:
Jun 12 07:30:55 dashboard kernel: CPU    0: hi:  186, btch:  31 usd:  68
Jun 12 07:30:55 dashboard kernel: active_anon:63386 inactive_anon:63461 isolated_anon:0
Jun 12 07:30:55 dashboard kernel: active_file:265 inactive_file:334 isolated_file:0
Jun 12 07:30:55 dashboard kernel: unevictable:0 dirty:6 writeback:1 unstable:0
Jun 12 07:30:55 dashboard kernel: free:13251 slab_reclaimable:2690 slab_unreclaimable:13761
Jun 12 07:30:55 dashboard kernel: mapped:8752 shmem:8570 pagetables:3855 bounce:0
Jun 12 07:30:55 dashboard kernel: Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:3132kB inactive_anon:3268kB active_file:52kB inactive_file:224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15304kB mlocked:0kB dirty:0kB writeback:4kB mapped:1300kB shmem:1252kB slab_reclaimable:8kB slab_unreclaimable:72kB kernel_stack:416kB pagetables:128kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:288 all_unreclaimable? no
Jun 12 07:30:55 dashboard kernel: lowmem_reserve[]: 0 2004 2004 2004
Jun 12 07:30:55 dashboard kernel: Node 0 DMA32 free:44656kB min:44720kB low:55900kB high:67080kB active_anon:250412kB inactive_anon:250576kB active_file:1008kB inactive_file:1112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2052256kB mlocked:0kB dirty:24kB writeback:0kB mapped:33708kB shmem:33028kB slab_reclaimable:10752kB slab_unreclaimable:54972kB kernel_stack:3176kB pagetables:15292kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2816 all_unreclaimable? no
Jun 12 07:30:55 dashboard kernel: lowmem_reserve[]: 0 0 0 0
Jun 12 07:30:55 dashboard kernel: Node 0 DMA: 23*4kB 18*8kB 21*16kB 7*32kB 2*64kB 2*128kB 2*256kB 3*512kB 1*1024kB 2*2048kB 0*4096kB = 8348kB
Jun 12 07:30:55 dashboard kernel: Node 0 DMA32: 10656*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 44656kB
Jun 12 07:30:55 dashboard kernel: 18191 total pagecache pages
Jun 12 07:30:55 dashboard kernel: 9001 pages in swap cache
Jun 12 07:30:55 dashboard kernel: Swap cache stats: add 5526230, delete 5517229, find 13659110/14099219
Jun 12 07:30:55 dashboard kernel: Free swap  = 0kB
Jun 12 07:30:55 dashboard kernel: Total swap = 4128760kB
Jun 12 07:30:55 dashboard kernel: 524272 pages RAM
Jun 12 07:30:55 dashboard kernel: 43681 pages reserved
Jun 12 07:30:55 dashboard kernel: 57899 pages shared
Jun 12 07:30:55 dashboard kernel: 453313 pages non-shared
Jun 12 07:30:55 dashboard kernel: [ pid ]  uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Jun 12 07:30:55 dashboard kernel: [  429]    0  429    2766        1  0    -17        -1000 udevd
Jun 12 07:30:55 dashboard kernel: [  997]    0  997    23293      11  0    -17        -1000 auditd
Jun 12 07:30:55 dashboard kernel: [ 1013]    0  1013    62367        7  0      0            0 rsyslogd
Jun 12 07:30:55 dashboard kernel: [ 1025]    81  1025    7911        2  0      0            0 dbus-daemon
Jun 12 07:30:55 dashboard kernel: [ 1063]    0  1063    16563        3  0    -17        -1000 sshd
Jun 12 07:30:55 dashboard kernel: [ 1071]    38  1071    7540      36  0      0            0 ntpd
Jun 12 07:30:55 dashboard kernel: [ 1180]    0  1180    20216      23  0      0            0 master
Jun 12 07:30:55 dashboard kernel: [ 1189]    89  1189    20279      18  0      0            0 qmgr
Jun 12 07:30:55 dashboard kernel: [ 1190]    0  1190    29312      23  0      0            0 crond
Jun 12 07:30:55 dashboard kernel: [ 1202]    0  1202    15628      213  0      0            0 miniserv.pl
Jun 12 07:30:55 dashboard kernel: [ 1210]    0  1210    1014        2  0      0            0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1212]    0  1212    1014        2  0      0            0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1214]    0  1214    1014        2  0      0            0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1216]    0  1216    1014        2  0      0            0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1218]    0  1218    1014        2  0      0            0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1220]    0  1220    1014        2  0      0            0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1229]    0  1229    3095        1  0    -17        -1000 udevd
Jun 12 07:30:55 dashboard kernel: [ 1455]    0  1455    11713      175  0      0            0 openvpn
Jun 12 07:30:55 dashboard kernel: [ 1459]    0  1459    2765        1  0    -17        -1000 udevd
Jun 12 07:30:55 dashboard kernel: [27796]    26 27796    53961      270  0    -17        -1000 postmaster
Jun 12 07:30:55 dashboard kernel: [27798]    26 27798    44714      23  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [27800]    26 27800    54027    8439  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [27801]    26 27801    53961      48  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [27802]    26 27802    54187      189  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [27803]    26 27803    44926      166  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [29741]    0 29741  145458      575  0      0            0 nxagentd
Jun 12 07:30:55 dashboard kernel: [24680]    0 24680  1328356    76557  0      0            0 netxmsd
Jun 12 07:30:55 dashboard kernel: [24681]    26 24681    54756    5044  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [24682]    26 24682    54784    8936  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [24683]    26 24683    54463    5255  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [24684]    26 24684    54319    2435  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [24685]    26 24685    54326    1270  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [24686]    26 24686    54297      333  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [24806]    26 24806    85327    32101  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [24894]    26 24894    54494    8758  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [24895]    26 24895    54837    9009  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: [19022]    89 19022    20236      19  0      0            0 pickup
Jun 12 07:30:55 dashboard kernel: [20255]    0 20255    18737    5391  0      0            0 miniserv.pl
Jun 12 07:30:55 dashboard kernel: [20279]    0 20279    2824      47  0      0            0 sh
Jun 12 07:30:55 dashboard kernel: [20281]    0 20281    19113    1052  0      0            0 rpm
Jun 12 07:30:55 dashboard kernel: [20295]    26 20295    54363      667  0      0            0 postmaster
Jun 12 07:30:55 dashboard kernel: Out of memory: Kill process 24680 (netxmsd) score 692 or sacrifice child
Jun 12 07:30:55 dashboard kernel: Killed process 24680, UID 0, (netxmsd) total-vm:5313424kB, anon-rss:305708kB, file-rss:520kB

]]>https://forum.netxms.org/topic/3213/linux-kernel-kills-netxms-process-out-of-memory-after-upgrade-1.2.14RSS for NodeFri, 13 Nov 2026 08:24:51 GMTThu, 12 Jun 2014 13:22:39 GMT60<![CDATA[Reply to Linux kernel kills netxms process- out of memory after upgrade 1.2.14 on Sun, 15 Jun 2014 09:10:16 GMT]]>Hi!

Log file seems to be truncated. Valgrind finalizes log file only when process exists, so you should stop netxmsd first.

Best regards,
Victor

]]>https://forum.netxms.org/post/14201https://forum.netxms.org/post/14201Sun, 15 Jun 2014 09:10:16 GMT<![CDATA[Reply to Linux kernel kills netxms process- out of memory after upgrade 1.2.14 on Fri, 13 Jun 2014 17:56:54 GMT]]>Hi Victor,

Please find the required log file on the attachments. Thanks,

netxmsd-valgrind.log

]]>https://forum.netxms.org/post/14190https://forum.netxms.org/post/14190Fri, 13 Jun 2014 17:56:54 GMT<![CDATA[Reply to Linux kernel kills netxms process- out of memory after upgrade 1.2.14 on Fri, 13 Jun 2014 13:16:29 GMT]]>Hi Victor,

I will run this valgrid on my server for a few hours, as soon as the test is done, I'll send you the log. Thanks for your help.

BTW, this is the command I'm going to use, is there any additional parameters I should  add?

valgrind --leak-check=full --undef-value-errors=no --log-file=netxmsd-valgrind.log netxmsd -D3

]]>https://forum.netxms.org/post/14183https://forum.netxms.org/post/14183Fri, 13 Jun 2014 13:16:29 GMT<![CDATA[Reply to Linux kernel kills netxms process- out of memory after upgrade 1.2.14 on Fri, 13 Jun 2014 13:06:50 GMT]]>Please find below the result for the command " sh sta"

Total number of objects:    771
Number of monitored nodes:  76
Number of collectable DCIs:  1564

]]>https://forum.netxms.org/post/14180https://forum.netxms.org/post/14180Fri, 13 Jun 2014 13:06:50 GMT<![CDATA[Reply to Linux kernel kills netxms process- out of memory after upgrade 1.2.14 on Fri, 13 Jun 2014 05:30:09 GMT]]>Am just interesting, can you show output:

nxadm -c "sh sta"
]]>https://forum.netxms.org/post/14169https://forum.netxms.org/post/14169Fri, 13 Jun 2014 05:30:09 GMT<![CDATA[Reply to Linux kernel kills netxms process- out of memory after upgrade 1.2.14 on Thu, 12 Jun 2014 19:25:11 GMT]]>Hi!

Looks like memory leak in netxmsd. Could you run it under valgrind for some time and send me valgrind's log?

Best regards,
Victor

]]>https://forum.netxms.org/post/14151https://forum.netxms.org/post/14151Thu, 12 Jun 2014 19:25:11 GMT