Thanks for the help. This is what I came up with:
The idea is to filter alerts and to prioritize those that are not secondary in terms of node down status.
If node is down and node above in the infrastructure is also down, the chances are high that the problem is only at the uppermost node and all other node down events are only "flooding" the supervisors attention.
Assuming I have nodes sorted in the infrastructure to fit the needs - based on dependency.
There was one thing to decide first - whether to filter the events or just actions based on the events.
a) events start only for nodes that fulfill the filter criteria
pro: event log and/or actions based on all events contain only the uppermost alerts
con: if there are problems with other nodes too (or if the problem starts in the meantime), after the uppermost node is resolved (up) - other problems are not shown in the log or taken action upon
b) events start and show for each node down but actions (e-mails) start only for events that pass the filter
pro: solves the problem in variant a)
con: event log contains all down nodes for the duration of the uppermost node downtime, effectively making the view unusable (if there are alot of depending nodes)
I decided for variant b) (with tagging the events to have some sort of control in the event log)
Event Processing Policy
create a "copy" of the default rule 1. (Show alarm when node is down)
Filter:
IF: SYS_NODE_DOWN
AND:
nparents = GetNodeParents($node); // get parent objects of tested node
foreach(i : nparents)
{
if (i->type == 5) // if it is infrastructure object (container)...
{
oparents = GetObjectParents(i); //... get parents of the object too...
foreach(j : oparents)
{
if (j->type == 5) //... and if it is infrastructure object (container) again...
{
children = GetObjectChildren(j); //... get object children...
foreach(k : children)
{
if ((k->status == 4) && (classof(k)!="NetObj")) return 0; //... and if the status of any of the children objects is "critical" (bypassing non-node objects) do not take action.
}
}
}
}
}
return 1; // else allow action
Action: send e-mail (for example)
I used the same script as macro in the events SYS_NODE_DOWN to tag it whether the e-mail was sent or muted.
Basically adding the script into script library too and adding to the SYS_NODE_DOWN the %[Dependency] macro which is the same as above with the exceptions:
"return 0" changed to "return " (e-mail muted)"
"return 1" changed to "return " (e-mail sent)"
I decided not to differentiate what node above the chain is down (basically if any of the upper ones is down it means all below will go down as well anyway), but can pick with the like operation if needed.
Any thoughts on drawbacks ?
