General Kubernetes Resource Errors

Pod Status Shows CrashLoopBackOff

If some pods are in a persistent CrashLoopBackOf state, it means that the pods are crashing because they start up then immediately exit. Kubernetes restarts them and the cycle continues. To find potential causes of this error, review the output returned from the following command:

[root@bootstrap vagrant]# kubectl -n kube-system describe pods <pod name>
 Name:                 <pod name>
 Namespace:            kube-system
 Priority:             2000000000
 Priority Class Name:  system-cluster-critical

Persistent Volume Claim (PVC) Stuck in Pending State

If after provisioning a volume for a pod (for example Prometheus) the PVC still hangs in a Pending state, perform the following checks:

  1. Check that the volumes have been provisioned and are in a Ready state.

    kubectl describe volume <volume-name>
    [root@bootstrap vagrant]# kubectl describe volume test-volume
     Name:         <volume-name>
     Status:
       Conditions:
         Last Transition Time:  2020-01-14T12:57:56Z
         Last Update Time:      2020-01-14T12:57:56Z
         Status:                True
         Type:                  Ready
    
  2. Check that a corresponding PersistentVolume exists.

    [root@bootstrap vagrant]# kubectl get pv
    NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS  STORAGECLASS             AGE       CLAIM
    <volume-name>              10Gi       RWO            Retain          Bound  <storage-class-name>     4d22h     <persistentvolume-claim-name>
    
  3. Check that the PersistentVolume matches the PersistentVolumeClaim constraints (size, labels, storage class).

    • Find the name of your PersistentVolumeClaim:

      [root@bootstrap vagrant]# kubectl get pvc -n <namespace>
      NAME                             STATUS   VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS          AGE
      <persistent-volume-claim-name>   Bound    <volume-name>          10Gi       RWO            <storage-class-name>  24h
      
    • Check if the PersistentVolumeClaim constraints match:

      [root@bootstrap vagrant]# kubectl describe pvc <persistentvolume-claim-name> -n <namespace>
      Name:          <persistentvolume-claim-name>
      Namespace:     <namespace>
      StorageClass:  <storage-class-name>
      Status:        Bound
      Volume:        <volume-name>
      Capacity:      10Gi
      Access Modes:  RWO
      VolumeMode:    Filesystem
      
  4. If no PersistentVolume exists, check that the storage operator is up and running.

    [root@bootstrap vagrant]# kubectl -n kube-system get deployments storage-operator
    NAME               READY   UP-TO-DATE   AVAILABLE   AGE
    storage-operator   1/1     1            1           4d22h
    

Access to MetalK8s GUI Fails With “undefined backend”

If you encounter an “undefined backend” error while using the MetalK8s GUI, perform the following checks:

  1. Check that the ingress controller pods are running.

    [root@bootstrap vagrant]#  kubectl -n metalk8s-ingress get daemonsets
    NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
    nginx-ingress-control-plane-controller   1         1         1       1            1           node-role.kubernetes.io/master=   4d22h
    nginx-ingress-controller                 1         1         1       1            1           <none>                            4d22h
    
  2. Check the ingress controller logs.

    [root@bootstrap vagrant]# kubectl logs -n metalk8s-ingress nginx-ingress-control-plane-controller-ftg6v
     -------------------------------------------------------------------------------
     NGINX Ingress controller
       Release:       0.26.1
       Build:         git-2de5a893a
       Repository:    https://github.com/kubernetes/ingress-nginx
       nginx version: openresty/1.15.8.2