Usage

Make sure you have installed the library and its dependencies already (enable the cli extra).

Create a simple Python module called example.py, and paste the following contents (or download it from here):

from lib_alert_tree.models import DerivedAlert as Derived
from lib_alert_tree.models import ExistingAlert as Existing
from lib_alert_tree.models import Relationship
from lib_alert_tree.models import severity_pair
from lib_alert_tree.cli import generate_cli

# Existing alerts are defined with their name and labels only.
FOO_WARNING = Existing("FooServiceTooManyErrors", severity="warning")
FOO_CRITICAL = Existing("FooServiceTooManyErrors", severity="critical")

# There are shortcuts for defining the severity.
BAR_BAZ_WARNING = Existing.warning("BarAlmostOutOfSpace", bar="baz")
BAR_BAZ_CRITICAL = Existing.critical("BarAlmostOutOfSpace", bar="baz")

# Derived alerts are built from a list of children and a relationship type.
# To be serialized into valid Prometheus configuration, some other attributes are
# required.
FOOBAR_WARNING = Derived.warning(
    "FooBarDegraded",
    relationship=Relationship.ANY,
    children=[FOO_WARNING, BAR_BAZ_WARNING],
    duration="1m",
    summary="The FooBar service is degraded.",
)
FOOBAR_CRITICAL = Derived.critical(
    "FooBarAtRisk",
    relationship=Relationship.ALL,
    children=[FOO_CRITICAL, BAR_BAZ_CRITICAL],
    duration="1m",
    summary="The FooBar service is at risk.",
)

# The above "pair" is very common, so we built a shortcut for it.
ROOT_WARNING, ROOT_CRITICAL = severity_pair(
    "Example",
    summary_name="The example app",
    relationship=Relationship.ANY,
    warning_children=[FOOBAR_WARNING, Existing.warning("QuxNotProgressing")],
    critical_children=[FOOBAR_CRITICAL, Existing.critical("QuxNotProgressing")],
    duration="5m",
)

main = generate_cli(
    roots={"warning": ROOT_WARNING, "critical": ROOT_CRITICAL},
    prometheus_rule_labels={"metalk8s.scality.com/monitor": ""},
)

if __name__ == "__main__":
    main()

Read through the comments for an explanation of the basic features of this library.

Now, simply run the following to display the tree of alerts you just defined:

$ python -m example show

This will print out to the console:

You can also generate a PrometheusRule manifest using:

$ python -m example gen-rule \
    --name example.rules --namespace example \
    --out example_rule.yaml

The generated manifest should look like this one. Note that only DerivedAlert instances are part of this manifest (exiting alerts should not need any additional rules, and are only referred to in the derived expr).

Use the --help option to print a help message and discover more functionalities provided by these commands.

Going Further

To see a more advanced example, have a look at the MetalK8s alert tree defined under tools/lib-alert-tree/metalk8s. An easy access is provided through a tox environment:

$ tox -e alert-tree -- show