A Kubernetes cluster deployed on the Google Cloud Platform using GKE, on Microsoft Azure using AKS or even using Kops or similar tools on Amazon AWS comes with built-in tooling for centralized container log management, metrics collection, tracing, node health checking and more.
Basic Cluster Addons¶
On top of the basic Kubernetes services, the following addons are deployed:
Helm / Tiller¶
Helm is a package manager for Kubernetes. It can be used to deploy various services in a Kubernetes cluster using templates to describe objects. Tiller is a cluster-side service used by the helm CLI tool to manage these deployments.
Heapster is a service which collects and exposes resource consumption metrics of containers running in a cluster. The Kubernetes Dashboard uses the Heapster service, when available, to display CPU and memory usage of Pods, Deployments and more.
To expose Services to the outside world using an Ingress object, Kubernetes requires an Ingress Controller to be running in the cluster. For this purpose, MetalK8s deploys the nginx-ingress-controller, which uses the well-known Nginx HTTP server under the hood.
Metering / Monitoring¶
Metering and monitoring of a MetalK8s cluster is handled by the Prometheus stack, including the Prometheus TSDB for metrics storage, Alertmanager to send alerts when preconfigured conditions are (not) met, and Grafana to visualize stored metrics using predefined dashboards.
We use kube-prometheus to provide operational insight into the Kubernetes cluster and containers managed by it. This includes predefined alerting rules and various Grafana dashboards.
kube-prometheus uses prometheus-operator to deploy all required services.
The node-exporter service is deployed to expose various node OS metrics, which are in turn captured by Prometheus. These metrics include CPU, memory, disk and network consumption as well as many Linux-specific values.
To ease cluster operations, several Grafana dashboards are made available, including cluster-wide views and health-checks, node OS metrics, per-Deployment or per-Pod resource usage, monitoring of the Prometheus service itself, and many more.
Do we need to list all exported deployed with kube-prometheus?
The ElasticSearch full-text indexing service is used to ingest all container logs in a central place, and make them accessible to operators. This ElasticSearch cluster is deployed using the Helm chart, with a configuration tuned for production-grade settings.
To ensure ingested logs don’t flood the ElasticSearch resources, ElasticSearch Curator is deployed with a default configuration which drops logstash-* indices on a given schedule.
Fluent Bit and fluentd¶
In MetalK8s, Fluent Bit and fluentd have a role similar to Logstash in the ELK stack.