Server crash opening one dashboard

[sqk]

Hello,

after updating to version 6.2.4 (server, agent, client...), opening one of our custom dashboards causes the server to crash; how can we identify the cause of the problem?

We tried disabling "Scripts.RestrictWriteAccess", but without success.

Thanks in advance,

Alessio.

[alk]

Hi,

Fastest way to pin this down is to run the server under gdb and let it stop at the crash - the backtrace names the code path directly.

apt install netxms-dbg
systemctl stop netxms-server
gdb /usr/bin/netxmsd
run -D3
# open the dashboard in the client, wait for the crash
bt

Post the bt output here.

netxms-dbg is required - without it the backtrace is just hex addresses and tells us nothing. Without -d or -S netxmsd stays in foreground, which is what we want; it also starts interactive console there, so add -q if that gets in the way of gdb.

Note: Scripts.RestrictWriteAccess has no effect here. Dashboard element scripts run under user security context, not the read-only one that parameter controls, so toggling it changes nothing either way.

[sqk]

Thanks Alex,

here it is:

#0  0x00007ffff74b7b77 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff4fedebf in ?? () from /lib/x86_64-linux-gnu/libmariadb.so.3
#2  0x00007ffff4fef2b0 in ?? () from /lib/x86_64-linux-gnu/libmariadb.so.3
#3  0x00007ffff4fea43e in mysql_stmt_fetch_column () from /lib/x86_64-linux-gnu/libmariadb.so.3
#4  0x00007ffff6050ebe in GetFieldInternal (result=0x7ffff4442920, iRow=<optimized out>, iColumn=4, buffer=0x7fffd8e1a800,
    bufferSize=2187953, utf8=<optimized out>) at mariadb/mariadb.cpp:693
#5  0x00007ffff783cf6e in DBGetFieldUTF8 (hResult=0x7fffde414cc0, row=0, column=4, buffer=<optimized out>, bufferSize=0) at session.cpp:561
#6  0x00007ffff783d87c in DBGetFieldBinary (hResult=hResult@entry=0x7fffde414cc0, row=row@entry=0, column=column@entry=4,
    length=length@entry=0x7fffdf5f9538) at session.cpp:846
#7  0x00007ffff7c618d6 in ClientSession::sendLibraryImage (this=this@entry=0x7ffff4a34000, request=...) at session.cpp:13810
#8  0x00007ffff7c827f9 in ClientSession::processRequest (this=0x7ffff4a34000, request=0x7fffe9141f30) at session.cpp:1651
#9  0x00007ffff7c2cde2 in __ThreadPoolExecute_Wrapper_1<ClientSession, NXCPMessage*> (arg=0x7fffe912aca0)
    at ../../../include/nms_threads.h:1046
#10 0x00007ffff77e41f5 in WorkerThread (threadInfo=0x7fffefc77af0) at tp.cpp:228
#11 0x00007ffff77e532b in ThreadCreate_Wrapper_1<WorkerThreadInfo*> (context=0x7fffefc77a00) at ../../include/nms_threads.h:354
#12 0x00007ffff770dea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007ffff7444aef in clone () from /lib/x86_64-linux-gnu/libc.so.6

[alk]

The backtrace pins it exactly: it's a bug in our MariaDB driver, triggered by an oversized entry in the image library.

Since 6.2 images are stored in the database instead of as files, and the driver allocates the buffer for a field on the stack, sized by the length of that field. Server worker threads get a 1 MB stack by default, so an image over roughly 780 KB overflows it and the process dies the moment anything asks for that image - which is what opening the dashboard does. The upgrade migrates existing image files into the database, so an image that had been fine for years starts crashing the server right after.

Two steps to get you running again.

1. Add to netxmsd.conf and restart the server:

DefaultThreadStackSize=8M

That raises the worker stacks and stops the crash right away.

2. Find the image and replace it with a smaller one:

SELECT guid, name, category, LENGTH(image_data) FROM images ORDER BY LENGTH(image_data) DESC LIMIT 5;

Anything near or above 1000000 is the one. Re-upload a downscaled version through the image library - listing and re-uploading are safe, only fetching the image bytes crashes.

Ticket is https://github.com/netxms/netxms/issues/3637 - the driver should allocate large fields on the heap, not the stack.

[sqk]

Many thanks Alex,

found images and working on them now...

+--------------------------------------+-----------------------------------------------------------------+----------+--------------------+
| guid                                 | name                                                            | category | LENGTH(image_data) |
+--------------------------------------+-----------------------------------------------------------------+----------+--------------------+
| 81652d15-df62-44ca-9f6b-55c68c6d8670 | schema sinottico REALE alimentazione Palma Piani DRAWIO-NOAVV   | Default  |            1824040 |
| 2ffbcbdb-a04e-488b-91ec-54c48de952e6 | mappa_sfondo                                                    | Default  |            1603868 |
| f5440229-8d66-4f32-a12c-1d984a2479bd | schema sinottico REALE alimentazione Trieste DRAWIO-NOAVV       | Default  |            1581020 |
| 9e5a6fdf-cc27-4440-bc3b-364912304f34 | schema sinottico REALE alimentazione CED Palmanova DRAWIO-NOAVV | Default  |            1093976 |
| 4da68704-046f-4a78-afad-cd3ecf417687 | bw_mappa_sfondo                                                 | Default  |             934132 |
| 94249c03-ba41-4fa8-bb61-ec9441bab325 | schema sinottico REALE alimentazione UD-Fiera DRAWIO-NOAVV      | Default  |             805556 |

[victor]

Version 6.2.5 that is just published should have this fixed.

[sqk]

Great job, thanks everyone!

Alessio.