Cluster Services

A Kubernetes cluster deployed on the Google Cloud Platform using GKE, on Microsoft Azure using AKS or even using Kops or similar tools on Amazon AWS comes with built-in tooling for centralized container log management, metrics collection, tracing, node health checking and more.

In MetalK8s, we augment a basic Kubernetes cluster deployed using the Kubespray playbook) with various tools to bring an on-premise cluster to the same level of operability.

Basic Cluster Addons

On top of the basic Kubernetes services, the following addons are deployed:

Helm / Tiller

Helm is a package manager for Kubernetes. It can be used to deploy various services in a Kubernetes cluster using templates to describe objects. Tiller is a cluster-side service used by the helm CLI tool to manage these deployments.

Heapster

Heapster is a service which collects and exposes resource consumption metrics of containers running in a cluster. The Kubernetes Dashboard uses the Heapster service, when available, to display CPU and memory usage of Pods, Deployments and more.

metrics-server

The metrics-server service is derived from Heapster, and provides an implementation of the Metrics API exposing CPU and memory consumption of containers. These metrics are in turn used by the HorizontalPodAutoscaler controller.

Ingress Controller

To expose Services to the outside world using an Ingress object, Kubernetes requires an Ingress Controller to be running in the cluster. For this purpose, MetalK8s deploys the nginx-ingress-controller, which uses the well-known Nginx HTTP server under the hood.

Metering / Monitoring

Metering and monitoring of a MetalK8s cluster is handled by the Prometheus stack, including the Prometheus TSDB for metrics storage, Alertmanager to send alerts when preconfigured conditions are (not) met, and Grafana to visualize stored metrics using predefined dashboards.

prometheus-operator

The CoreOS Prometheus Operator is deployed in the cluster to manage Prometheus instances, scrape targets and alerting rules.

kube-prometheus

We use kube-prometheus to provide operational insight into the Kubernetes cluster and containers managed by it. This includes predefined alerting rules and various Grafana dashboards.

kube-prometheus uses prometheus-operator to deploy all required services.

node-exporter

The node-exporter service is deployed to expose various node OS metrics, which are in turn captured by Prometheus. These metrics include CPU, memory, disk and network consumption as well as many Linux-specific values.

Grafana

To ease cluster operations, several Grafana dashboards are made available, including cluster-wide views and health-checks, node OS metrics, per-Deployment or per-Pod resource usage, monitoring of the Prometheus service itself, and many more.

Todo

Do we need to list all exported deployed with kube-prometheus?

Log Collection

ElasticSearch

The ElasticSearch full-text indexing service is used to ingest all container logs in a central place, and make them accessible to operators. This ElasticSearch cluster is deployed using the manifests provided in pires/kubernetes-elasticsearch-cluster, which are tuned to use production-grade settings.

ElasticSearch Curator

To ensure ingested logs don’t flood the ElasticSearch resources, ElasticSearch Curator is deployed with a default configuration which drops logstash-* indices on a given schedule.

fluentd

The fluentd service is deployed as a DaemonSet to stream all container logs into ElasticSearch.

In MetalK8s, fluentd has a role similar to Logstash in the ELK stack.

Kibana

To give operators access to the logs stored in ElasticSearch, a Kibana instance is provided.

Note

When accessing Kibana for the first time, an index pattern for the logstash-* indices needs to be configured, using @timestamp as Time Filter field name.