Linux kernel kills netxms process- out of memory after upgrade 1.2.14
-
Hi,
Something wierd is happing after I upgraded my 1.2.8 netxms on a VM running centos 5.
The server is running up to one day and then the process is killed by the kernel "out of memory", do you have an idea what is this happeing? before on version 1.2.8 I've never had this issue.
The network discovery is disable. Please find below the logs from the VM.
Thanks,
Christian Diaz
Jun 12 07:30:55 dashboard kernel: netxmsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jun 12 07:30:55 dashboard kernel: netxmsd cpuset=/ mems_allowed=0
Jun 12 07:30:55 dashboard kernel: Pid: 24776, comm: netxmsd Not tainted 2.6.32-358.14.1.el6.x86_64 #1
Jun 12 07:30:55 dashboard kernel: Call Trace:
Jun 12 07:30:55 dashboard kernel: [<ffffffff810cb561>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111cd80>] ? dump_header+0x90/0x1b0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8121d2ac>] ? security_real_capable_noaudit+0x3c/0x70
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111d202>] ? oom_kill_process+0x82/0x2a0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111d141>] ? select_bad_process+0xe1/0x120
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111d640>] ? out_of_memory+0x220/0x3c0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8112c2ec>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Jun 12 07:30:55 dashboard kernel: [<ffffffff811609ea>] ? alloc_pages_current+0xaa/0x110
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111a167>] ? __page_cache_alloc+0x87/0x90
Jun 12 07:30:55 dashboard kernel: [<ffffffff81119b4e>] ? find_get_page+0x1e/0xa0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111b127>] ? filemap_fault+0x1a7/0x500
Jun 12 07:30:55 dashboard kernel: [<ffffffff81143124>] ? __do_fault+0x54/0x530
Jun 12 07:30:55 dashboard kernel: [<ffffffff811436f7>] ? handle_pte_fault+0xf7/0xb50
Jun 12 07:30:55 dashboard kernel: [<ffffffff81063360>] ? wake_up_state+0x10/0x20
Jun 12 07:30:55 dashboard kernel: [<ffffffff810a9ae0>] ? wake_futex+0x40/0x60
Jun 12 07:30:55 dashboard kernel: [<ffffffff810aa35e>] ? futex_wake+0x10e/0x120
Jun 12 07:30:55 dashboard kernel: [<ffffffff8114438a>] ? handle_mm_fault+0x23a/0x310
Jun 12 07:30:55 dashboard kernel: [<ffffffff810474e9>] ? __do_page_fault+0x139/0x480
Jun 12 07:30:55 dashboard kernel: [<ffffffff81435fd5>] ? sys_sendto+0x185/0x190
Jun 12 07:30:55 dashboard kernel: [<ffffffff8150dd80>] ? thread_return+0x4e/0x76e
Jun 12 07:30:55 dashboard kernel: [<ffffffff815137be>] ? do_page_fault+0x3e/0xa0
Jun 12 07:30:55 dashboard kernel: [<ffffffff81510b75>] ? page_fault+0x25/0x30
Jun 12 07:30:55 dashboard kernel: Mem-Info:
Jun 12 07:30:55 dashboard kernel: Node 0 DMA per-cpu:
Jun 12 07:30:55 dashboard kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jun 12 07:30:55 dashboard kernel: Node 0 DMA32 per-cpu:
Jun 12 07:30:55 dashboard kernel: CPU 0: hi: 186, btch: 31 usd: 68
Jun 12 07:30:55 dashboard kernel: active_anon:63386 inactive_anon:63461 isolated_anon:0
Jun 12 07:30:55 dashboard kernel: active_file:265 inactive_file:334 isolated_file:0
Jun 12 07:30:55 dashboard kernel: unevictable:0 dirty:6 writeback:1 unstable:0
Jun 12 07:30:55 dashboard kernel: free:13251 slab_reclaimable:2690 slab_unreclaimable:13761
Jun 12 07:30:55 dashboard kernel: mapped:8752 shmem:8570 pagetables:3855 bounce:0
Jun 12 07:30:55 dashboard kernel: Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:3132kB inactive_anon:3268kB active_file:52kB inactive_file:224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15304kB mlocked:0kB dirty:0kB writeback:4kB mapped:1300kB shmem:1252kB slab_reclaimable:8kB slab_unreclaimable:72kB kernel_stack:416kB pagetables:128kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:288 all_unreclaimable? no
Jun 12 07:30:55 dashboard kernel: lowmem_reserve[]: 0 2004 2004 2004
Jun 12 07:30:55 dashboard kernel: Node 0 DMA32 free:44656kB min:44720kB low:55900kB high:67080kB active_anon:250412kB inactive_anon:250576kB active_file:1008kB inactive_file:1112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2052256kB mlocked:0kB dirty:24kB writeback:0kB mapped:33708kB shmem:33028kB slab_reclaimable:10752kB slab_unreclaimable:54972kB kernel_stack:3176kB pagetables:15292kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2816 all_unreclaimable? no
Jun 12 07:30:55 dashboard kernel: lowmem_reserve[]: 0 0 0 0
Jun 12 07:30:55 dashboard kernel: Node 0 DMA: 23*4kB 18*8kB 21*16kB 7*32kB 2*64kB 2*128kB 2*256kB 3*512kB 1*1024kB 2*2048kB 0*4096kB = 8348kB
Jun 12 07:30:55 dashboard kernel: Node 0 DMA32: 10656*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 44656kB
Jun 12 07:30:55 dashboard kernel: 18191 total pagecache pages
Jun 12 07:30:55 dashboard kernel: 9001 pages in swap cache
Jun 12 07:30:55 dashboard kernel: Swap cache stats: add 5526230, delete 5517229, find 13659110/14099219
Jun 12 07:30:55 dashboard kernel: Free swap = 0kB
Jun 12 07:30:55 dashboard kernel: Total swap = 4128760kB
Jun 12 07:30:55 dashboard kernel: 524272 pages RAM
Jun 12 07:30:55 dashboard kernel: 43681 pages reserved
Jun 12 07:30:55 dashboard kernel: 57899 pages shared
Jun 12 07:30:55 dashboard kernel: 453313 pages non-shared
Jun 12 07:30:55 dashboard kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jun 12 07:30:55 dashboard kernel: [ 429] 0 429 2766 1 0 -17 -1000 udevd
Jun 12 07:30:55 dashboard kernel: [ 997] 0 997 23293 11 0 -17 -1000 auditd
Jun 12 07:30:55 dashboard kernel: [ 1013] 0 1013 62367 7 0 0 0 rsyslogd
Jun 12 07:30:55 dashboard kernel: [ 1025] 81 1025 7911 2 0 0 0 dbus-daemon
Jun 12 07:30:55 dashboard kernel: [ 1063] 0 1063 16563 3 0 -17 -1000 sshd
Jun 12 07:30:55 dashboard kernel: [ 1071] 38 1071 7540 36 0 0 0 ntpd
Jun 12 07:30:55 dashboard kernel: [ 1180] 0 1180 20216 23 0 0 0 master
Jun 12 07:30:55 dashboard kernel: [ 1189] 89 1189 20279 18 0 0 0 qmgr
Jun 12 07:30:55 dashboard kernel: [ 1190] 0 1190 29312 23 0 0 0 crond
Jun 12 07:30:55 dashboard kernel: [ 1202] 0 1202 15628 213 0 0 0 miniserv.pl
Jun 12 07:30:55 dashboard kernel: [ 1210] 0 1210 1014 2 0 0 0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1212] 0 1212 1014 2 0 0 0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1214] 0 1214 1014 2 0 0 0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1216] 0 1216 1014 2 0 0 0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1218] 0 1218 1014 2 0 0 0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1220] 0 1220 1014 2 0 0 0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1229] 0 1229 3095 1 0 -17 -1000 udevd
Jun 12 07:30:55 dashboard kernel: [ 1455] 0 1455 11713 175 0 0 0 openvpn
Jun 12 07:30:55 dashboard kernel: [ 1459] 0 1459 2765 1 0 -17 -1000 udevd
Jun 12 07:30:55 dashboard kernel: [27796] 26 27796 53961 270 0 -17 -1000 postmaster
Jun 12 07:30:55 dashboard kernel: [27798] 26 27798 44714 23 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [27800] 26 27800 54027 8439 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [27801] 26 27801 53961 48 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [27802] 26 27802 54187 189 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [27803] 26 27803 44926 166 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [29741] 0 29741 145458 575 0 0 0 nxagentd
Jun 12 07:30:55 dashboard kernel: [24680] 0 24680 1328356 76557 0 0 0 netxmsd
Jun 12 07:30:55 dashboard kernel: [24681] 26 24681 54756 5044 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [24682] 26 24682 54784 8936 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [24683] 26 24683 54463 5255 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [24684] 26 24684 54319 2435 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [24685] 26 24685 54326 1270 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [24686] 26 24686 54297 333 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [24806] 26 24806 85327 32101 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [24894] 26 24894 54494 8758 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [24895] 26 24895 54837 9009 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: [19022] 89 19022 20236 19 0 0 0 pickup
Jun 12 07:30:55 dashboard kernel: [20255] 0 20255 18737 5391 0 0 0 miniserv.pl
Jun 12 07:30:55 dashboard kernel: [20279] 0 20279 2824 47 0 0 0 sh
Jun 12 07:30:55 dashboard kernel: [20281] 0 20281 19113 1052 0 0 0 rpm
Jun 12 07:30:55 dashboard kernel: [20295] 26 20295 54363 667 0 0 0 postmaster
Jun 12 07:30:55 dashboard kernel: Out of memory: Kill process 24680 (netxmsd) score 692 or sacrifice child
Jun 12 07:30:55 dashboard kernel: Killed process 24680, UID 0, (netxmsd) total-vm:5313424kB, anon-rss:305708kB, file-rss:520kB -
Hi!
Looks like memory leak in netxmsd. Could you run it under valgrind for some time and send me valgrind's log?
Best regards,
Victor -
-
-
Hi Victor,
I will run this valgrid on my server for a few hours, as soon as the test is done, I'll send you the log. Thanks for your help.
BTW, this is the command I'm going to use, is there any additional parameters I should add?
valgrind --leak-check=full --undef-value-errors=no --log-file=netxmsd-valgrind.log netxmsd -D3
-
-
Hi!
Log file seems to be truncated. Valgrind finalizes log file only when process exists, so you should stop netxmsd first.
Best regards,
Victor
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login