Welcome to the last Kubernetes lecture, where we build ourselves an operator and talk about other more advanced patterns and when to use them

Last Updated: 2024-05-25

Why

Some frameworks are very typically bundled with a modern kubernetes, such as Prometheus/Grafana, Cert-Manager or ELK. The extension of the kubernetes API and integrated functionality has a few notable patterns.

What

We will cover the following topics:

What you'll build today

By the end of today's lecture, you should have

Homework (Flipped Classroom)

How should I prepare

(same as Lecture 9 - maybe your improved it all )

PreRead

What you'll need

What if K8s doesnt know how to do what you want it to do: you might want to teach it.

In the beginning, there was <a story>

https://www.codereliant.io/build-kubernetes-operator-kubebuilder/

FIG 2. https://book.kubebuilder.io/architecture

Lets see what projects are out there and how they differ

Navigating Operators and Notable Frameworks

https://landscape.cncf.io

We ll do it locally on kind , but you can install it afterwards onto your RKE2

Installation

You need kind, go and kubebuilder (if you want to run cilium connectivity tests, you need the cilium cli)

kind create cluster

Use Constanzes github repo kind-config.yaml and Makefile for reference if you want a larger kind with cilium 

mkdir -p ~/demooperator
cd ~/demooperator
kubebuilder init --domain lva.io --repo=github.com/AustrianDataLab/demooperator
kubebuilder create api --group demo --version v1 --kind Icecream

The stuff thats been generated corresponds to the above Fig. 2

Now, we need to design our Icecream Types:

Note, the annotation syntax for the preprocessor like

//+kubebuilder:printcolumn:name="SoldOut",type="string",JSONPath=".status.soldOut",description="SoldOut"

make manifests

kubectl apply -f config/samples/demo_v1_icecream.yaml
icecream.demo.lva.io/icecream-sample created

And you notice that our status fields are pretty useless for now.

So now, your first exercise is to put something useful into this reconciler loop.

The following code is on github and makes sure that we cannot sell more than 3 icecreams of the same flavour 

make run

Now, I apply last the "toomuch" icecream, and it will delete the "special" icecream and we will never really see the SoldOut status

How to get rid off stuff - properly

Finalizers

In order to guarantee a stateful deletion (meaning that certain things have to be in certains states or/and deleted in the right order)

Finalizers are often not super well tested for edge-cases. Especially not for force-deletes . So you might have to edit them out of a zombie artefact - in emergencies.

https://github.com/AustrianDataLAB/demooperator 

git checkout finalizer
make run
❯ kubectl apply -f config/samples/demo_v1_icecream.yaml
icecream.demo.lva.io/toomuch created
❯ kubectl delete -f config/samples/demo_v1_icecream.yaml
icecream.demo.lva.io "toomuch" deleted

What you can do , before you persist your changes to kubernetes (etcd) and without interfering with the Kubernetes API

Webhooks are standalone webserver, that are inserted between each API call and its execution, via a POST request "asking for permission" . Some webhooks also inject business logic

API Validation

Validating Webhooks

We do the same check as above but this time its the Webhook and not the API doing the validation and we are validating DIFFERENT flavours (so only vanilla is actually allowed now)

git checkout webhook
kubebuilder create webhook --group demo --version v1 --kind Icecream –-programmatic-validation
INFO Writing kustomize manifests for you to edit... 
INFO Writing scaffold for you to edit...          
INFO api/v1/icecream_webhook.go                   
INFO api/v1/webhook_suite_test.go                 
INFO Update dependencies:
$ go mod tidy           
INFO Running make:
$ make generate 

make manifests && \
make install && \
make docker-build && \ 
kind load docker-image --name lvaoperator controller:latest && \
make deploy

We don't have certificates yet, so let's leave it for now in pending state

Because we need to talk about it at some point - State on K8s

Operators for StatefulSets

State is hard . That's even more true in a distributed system.

Note:

Dont use CRDs as a Database for your k8s-application

A way to encrypt the internet - for free !!

https://www.thesslstore.com/blog/acme-protocol-what-it-is-and-how-it-works/

https://www.slideshare.net/slideshow/implementing-certmanager-in-k8s/258339304

https://www.rfc-editor.org/rfc/pdfrfc/rfc8555.txt.pdf

Fig (taken from https://doc.nexusgroup.com/pub/what-is-acme )

https://www.youtube.com/watch?v=k73MNzoel7Q

Lets first understand cert-manager as an operator using a simple self-signed issuer

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager --set installCRDs=true --namespace cert-manager  --create-namespace

Run the Helm install on the same cluster as you had your IceCream Operator on

Now, lets give our webhook (for the Icecream) a certificate and an issuer, so in ./config/certmanager fill out the Certificate.yaml and apply it. Dont forget to change the namespace of the Issuer , it needs to be in the same namespace as your demooperator

And, now you can see that your demooperator webhook controller will boot and is running your own reconcilers and finalizers (incl all bugs).

Now to the real thing, lets get a real certificate

For this, we move to RKE2 , because its a lot (!!) of effort without a public IP

Now, lets create a namespace , an nginx and an ingress . Please edit the following heredoc to your namespace and correct domain.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: hello-1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: hello-1
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: hello-1
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-cluster-issuer
spec:
  ingressClassName: nginx
  rules:
  - host: lva-1.caas-0026.beta.austrianopencloudcommunity.org
    http:
      paths:
      - pathType: Prefix     path: "/"
        backend:
          service:
            name: nginx-deployment
            port:
              number: 80
  tls:
    - hosts:
        - lva-1.caas-0026.beta.austrianopencloudcommunity.org
      secretName: tls-lva-1-ingress

EOF

And now, observe how the ACME challenge is being handled by the operator

It is now your exercise, to find all these steps on the RKE2/nginx

Congratulations, you've successfully completed this training on Operators and ACME

What's next?

Reference docs