alarm does not always resolve correctly

[Tucson]

Hi,

I'm monitoring the states of Windows VMs with thresholds that trigger events.
The "OS_MAJOR_PROCESS_NOT_RUNNING" event is triggered with the event key "OS_PROCESS_NOT_RUNNING%i".

This normally works as intended, but sometimes the alarm doesn't resolve when the node leaves maintenance mode.
Attached are the event logs in .csv format, as well as some details on the individual events.

Why didn't the alarm resolve with "maintenance leave" (event 8769121)?
Both "task running" events still reference "maintenance enter" (event 8766568).
eventlog.csv

event_8769122.txt

event_8769123.txt

event_8768197.txt

event_8768198.txt

[alk]

Hi,

This is a bug in the server, not your configuration: https://github.com/netxms/netxms/issues/3682

Here's what's actually happening: when the node leaves maintenance mode, the server re-generates threshold events for every threshold whose state changed during maintenance. That's where your two "task running" events at 04:00 came from. Those events can be processed while the node is still flagged as in maintenance, and then they get correlated to the "maintenance enter" event, which is why they reference 8766568. By default, EPP rules ignore correlated events, so the rule that should terminate the alarm never sees them. It's a timing race, which is why it only happens sometimes.

Workaround until it's fixed: enable Accept correlated events on the EPP rule that terminates the alarm on OS_PROCESS_OK. As a side effect, the alarm will also be terminated when the task recovers while the node is still in maintenance, which is probably what you want anyway. If the same rule also creates the alarm, split it into two rules and set the flag only on the terminating one. Otherwise alarms will be created during maintenance too.