Solution Operator guidelines¶

An Operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling.

—coreos.com/operators

MetalK8s Solutions are a concept mostly centered around the Operator pattern. While there is no explicit requirements except the ones described below (see Requirements), we recommend using the Operator SDK as it will embed best practices from the Kubernetes community.

Requirements¶

Files¶

All Operator-related files except for the container images (see OCI images) should be stored under /operator in the ISO archive. Those files should be organized as follows:

operator
└── deploy
    ├── crds
    │   └── some_crd.yaml
    └── role.yaml

Most of these files are generated when using the Operator SDK.

Monitoring¶

MetalK8s can automatically create a Service and ServiceMonitor to expose operator metrics to Prometheus. This is opt-in: add a metrics section under spec.operator in the solution manifest.yaml:

spec:
  operator:
    metrics:
      enabled: true        # default: false
      scheme: https        # default: "https", available: "http" or "https"
      port: 8443           # default: 8443 for HTTPS, 8080 for HTTP
      path: /metrics       # default: "/metrics"

When enabled is true, MetalK8s will:

add a containerPort to the operator Deployment
create a ClusterIP Service named <solution>-operator-metrics
create a ServiceMonitor named <solution>-operator with the label metalk8s.scality.com/monitor: '' for Prometheus discovery

When scheme is https, the ServiceMonitor is configured with bearerTokenFile authentication and insecureSkipVerify: true.

Operators should still create Service and ServiceMonitor objects for each deployed component they own.

The Prometheus Operator deployed by MetalK8s has cluster-scoped permissions, and is able to read the aforementioned ServiceMonitor objects to set up monitoring of your application services.

Configuration¶

Solution Operator must implement a --config option which will be used by MetalK8s to provide various useful information needed by the Operator, such as the endpoints for the container images. The given configuration looks like:

apiVersion: solutions.metalk8s.scality.com/v1alpha1
kind: OperatorConfig
repositories:
  <solution-version-x>:
    - endpoint: metalk8s-registry/<solution-name>-<solution-version-x>
      images:
        - <image-x>:<tag-x>
        - <image-y>:<tag-y>
  <solution-version-y>:
    - endpoint: metalk8s-registry/<solution-name>-<solution-version-y>
      images:
        - <image-x>:<tag-x>
        - <image-y>:<tag-y>

In example, for an online installation without MetalK8s providing the repository, this configuration could be:

apiVersion: solutions.metalk8s.scality.com/v1alpha1
kind: OperatorConfig
repositories:
  1.0.0:
    - endpoint: registry.scality.com/zenko
      images:
        - cloudserver:1.0.0
        - zenko-quorum:1.0.0
    - endpoint: quay.io/coreos
      images:
        - prometheus-operator:v0.34.0

This configuration allows the Operator to retrieve dynamically where the container images are stored for each version of a given Solution.

Roles¶

Solution must ship a role.yaml file located in /operator/deploy directory. This file is a manifest which declares all necessary Role and ClusterRole objects needed by the Operator. MetalK8s will take care of deploying these objects, create a ServiceAccount named <solution_name>-operator and all needed RoleBinding to bind these roles to this account.

Warning

Only Role and ClusterRole kinds are allowed in this file, the deployment of the Solution fails if any other resource is found.