Usage¶
Make sure you have installed the library and its
dependencies already (enable the cli
extra).
Create a simple Python module called example.py
, and paste the following
contents (or download it from here
):
from lib_alert_tree.models import DerivedAlert as Derived
from lib_alert_tree.models import ExistingAlert as Existing
from lib_alert_tree.models import Relationship
from lib_alert_tree.models import severity_pair
from lib_alert_tree.cli import generate_cli
# Existing alerts are defined with their name and labels only.
FOO_WARNING = Existing("FooServiceTooManyErrors", severity="warning")
FOO_CRITICAL = Existing("FooServiceTooManyErrors", severity="critical")
# There are shortcuts for defining the severity.
BAR_BAZ_WARNING = Existing.warning("BarAlmostOutOfSpace", bar="baz")
BAR_BAZ_CRITICAL = Existing.critical("BarAlmostOutOfSpace", bar="baz")
# Derived alerts are built from a list of children and a relationship type.
# To be serialized into valid Prometheus configuration, some other attributes are
# required.
FOOBAR_WARNING = Derived.warning(
"FooBarDegraded",
relationship=Relationship.ANY,
children=[FOO_WARNING, BAR_BAZ_WARNING],
duration="1m",
summary="The FooBar service is degraded.",
)
FOOBAR_CRITICAL = Derived.critical(
"FooBarAtRisk",
relationship=Relationship.ALL,
children=[FOO_CRITICAL, BAR_BAZ_CRITICAL],
duration="1m",
summary="The FooBar service is at risk.",
)
# The above "pair" is very common, so we built a shortcut for it.
ROOT_WARNING, ROOT_CRITICAL = severity_pair(
"Example",
summary_name="The example app",
relationship=Relationship.ANY,
warning_children=[FOOBAR_WARNING, Existing.warning("QuxNotProgressing")],
critical_children=[FOOBAR_CRITICAL, Existing.critical("QuxNotProgressing")],
duration="5m",
)
main = generate_cli(
roots={"warning": ROOT_WARNING, "critical": ROOT_CRITICAL},
prometheus_rule_labels={"metalk8s.scality.com/monitor": ""},
)
if __name__ == "__main__":
main()
Read through the comments for an explanation of the basic features of this library.
Now, simply run the following to display the tree of alerts you just defined:
$ python -m example show
This will print out to the console:

You can also generate a PrometheusRule manifest using:
$ python -m example gen-rule \
--name example.rules --namespace example \
--out example_rule.yaml
The generated manifest should look like
this one
. Note that only
DerivedAlert
instances are part of this manifest (exiting alerts should not
need any additional rules, and are only referred to in the derived expr
).
Use the --help
option to print a help message and discover more
functionalities provided by these commands.
Going Further¶
To see a more advanced example, have a look at the MetalK8s alert tree defined
under tools/lib-alert-tree/metalk8s
. An easy access is provided through
a tox
environment:
$ tox -e alert-tree -- show