Today, we are very happy to be joined by VADIM BAUER, who is a maintainer of the CNCF project Harbor. With over a decade of experience in running containers in production, he now focuses on expanding the boundaries of how OCI artifacts are managed, adopted, and used by developers. At 8gears, Vadim helps cloud providers, ISVs and enterprises adopt Harbor and use OCI capabilities. As a regular speaker at conferences, he shares his knowledge and passion

Welcome to the eighth lecture, where today we will understand registries, work with artifacts, understand what a registry-poisoning is and how to prevent it.

You will be able to ask Vadim questions about the CNCF mentorship program and being a maintainer etc

Last Updated: 2024-4-22

Why should I care about registries?

Container registries are where your artefacts are stored. So far in this class, you've already used the GitHub registry and the docker.io registry.

In order for all of us to both push as well as pull containers (and other artefacts, like e.g. helm charts) , there is the OCI standard that prescribes how a container has a runtime spec, a distribution spec and an image spec.

What

We will cover the following topics:

What you'll build today

By the end of today's lecture, you should have

Homework (Flipped Classroom)

Prepare

What you'll need

https://opencontainers.org/about/overview/

In the beginning, there was docker...

The Open Container Initiative (OCI) is an open governance structure (project), for the purpose of creating open industry standards around container formats and runtimes. The OCI was launched on June 22nd 2015 by Docker, CoreOS and other leaders in the container industry.

This entire workflow should support the UX that users have come to expect from container engines like Docker and rkt: primarily, the ability to run an image with no additional arguments:

To support this UX the OCI Image Format contains sufficient information to launch the application on the target platform (e.g. command, arguments, environment variables, etc). This specification defines how to create an OCI Image, which will generally be done by a build system, and output an image manifest, a filesystem (layer) serialization, and an image configuration.

What I need to know about it

As an end user, you need to know...

The image format specification


... so you can be sure that you can avoid vendor lock-in, since all (legit) vendors will accept and offer the exact same image formats.
... so you can interoperate various registries and place differently hosted (or differently featured) registries across your architecture, as you require

-> This specification is on the build side, e.g. prescribing what the index.json contains in order to be "pushable/understandable" (*) by any OCI registry and runnable by an OCI-compliant CRI

(*) Building, transporting, and preparing a container image to run.

Lets look at an example from our pacman pipeline

Below, you see a pipeline that :
First, uses the distribution spec to login to the OCI registry (here we use `helm`, whereas the traditional login would be `docker`)
Then, saves your chart using the image spec as you can see in the <registryname>:<tag> naming
Lastly, you push the chart, just as you would a container image. So the build instrumentation uses the image spec to communicate with the OCI registry.

The image distribution specification

... so you can push/pull from and to any OCI compliant registry and don't need to rewrite your (e.g. CI) pipeline.
And nowadays, artifacts other than containers can also be pushed/pulled if they comply with the spec.

This is the part that is implemented on the registry side.

The runtime specification


.... so you are guaranteed some predictable security and reliable runtime (e.g. avoiding out of memory, isolation and resource constraining behavior,). This means, that there can be several container-runtimes and they are interoperable (and to some extent) interchangeable
Example: not many people will have noticed when Kubernetes deprecated docker-runtime (well, actually the docker-shim) .
The reason actually was that in kubernetes , originally there were `shims` in order to run Container Runtimes (like docker ) , but they wanted all container runtimes to be interchangeable.

So, here we have several runtimes that follow different specs: all of the runtimes (in below example) follow the OCI spec but in order to talk to kubelet, a container runtime additionally needs to following CRI (container runtime interface) spec of kubernetes and docker doesn't do that directly .

https://kubernetes.io (for more info about breaking changes in k8s over history and how some of them led to lots of confusion, see this kubecon talk https://youtu.be/a03Hh1kd6KE?si=VsGnRywG6yFEy7Nr )

Question: Did this CRI spec have any influence on the registry spec?

Answer : :)

NB: too many acronyms lead to confusion, better not to assume that all people understand the fine differences.

Let's see what projects are out there and how and why they differ.

Navigating Container Landscapes

https://landscape.cncf.io/guide#provisioning--container-registry

It may be said that the AZURE/AWS/ALIBABA/iBM/GCP offerings are rather similar, apart from ‘seasonally changing special features'. Quay is the RedHat flavor thereof

Dragonfly/Kraken is a P2P-tech based image distribution system that helps with e.g. bandwidth issues.

Images contain the information needed to execute a program (within a container) and are stored in repositories, which in turn are categorized and grouped in registries. Tools that build, run, and manage containers need access to those images. Access is provided by referencing the registry (the path to access the image).

Problem Registries address

Cloud native applications are packaged and run as containers. Container registries store and provide the container images needed to run these apps.

What Registries Do Conceptually

Registries centrally store all container images in one place, they make images easily accessible for any developer and runtime.

Container registries either store and distribute images or enhance an existing registry in some way. Fundamentally, a registry is a web API that allows container runtimes and developers to store and retrieve images. Many provide interfaces to allow container scanning or signing tools to enhance the security of the images they store. Some specialize in distributing or duplicating images in a particularly efficient manner. Any environment using containers will need to use one or more registries.

How it started and which pieces survived to this day

https://goharbor.io

Harbor was created at VMware China in 2014 and initially open sourced in 2016. It is the first open-source registry compliant with the specifications of OCI.

What are Artifacts, and why do we need them?

The process of managing and controlling software artifacts throughout their lifecycle.

How do we version Artifacts and Containers?

How do images look like?

Buzzword explained

Glossary

Which types of scenarios are there in an Enterprise architecture where you'd want to have your registry/artefactory and why

1 Hosting your artifacts for the public to consume

You'd likely want to publish to a well known, established registry like https://hub.docker.com/ or https://artifacthub.io/ .

Attestation requirements for Supply Chain , CRA (CISA/ Exec Order - SBOM) https://github.com/awesomeSBOM/awesome-sbom https://www.linuxfoundation.org/blog/blog/what-is-an-sbom

https://csrc.nist.gov/CSRC/media/Projects/cyber-supply-chain-risk-management/documents/SSCA/Spring_2019/8MayAM2.3_Software_Bill_of_Materials_Robert_Martin_05_08_19_clean.pdf

UseCase:

2 Hosting your (private) artifacts for yourself to consume

Scenario: you are a small group of people and you want to move fast, maybe you don't want to or can't afford to fully cleanup your artifacts.

Reminder: Total cost of ownership (TCO) calculation

Analog: A mortgage from a bank : "buy money" to be able to afford something today.
Exercise: what were the important questions to ask when evaluating a TCO?

See here a screenshot of ACR (basic tier) cost for storing 83 GiB, which translates to about 10 Euro per month

UseCase:

3 Consuming artifacts from the public in a regulated entity

If you are a large e.g. bank, you might opt to mirror locally (and in your own network) some or all artifacts that are sanctioned by your organization to be used by all employees

This is very effortful, as you need the following components (here an example from Digital Ocean as CSP)

https://colinwilson.uk/2021/12/11/deploy-a-harbor-container-registry-with-high-availability-on-digitalocean-via-terraform/

This includes the following:

Discuss: What do you think the DevX is if you have your own mirrors?

Common issues: fear-based "security"

Please prepare your questions in Slido or input them during the lecture

Ask your questions on Slido during the lecture or before (opens April 30th)

https://app.sli.do/event/kyi8SEXsSeKTK4dCH8g7Gi

LFX/CNCF paid mentorship program

Always wanted to do Open Source, but didn't know how to get started?

vadim@8gears.com : any links to add or more graphics?

What are the main functionalities and what to use them for?

We prepared a harbor installation for you here : https://coreharbor.caas-0026.dev.austrianopencloudcommunity.org/account/sign-in?redirect_url=%2Fharbor%2Fprojects

TODO :
1) We go through account setup and then everyone prepare themselves a repo so they can push/pull

#Setup account
Get credential


2) How to setup an RKE2 clusters such that the cluster has pull rights on a so-called registry mirror?

In the example of our RKE2 clusters, we have several internal images living in an Azure Registry (ACR) and the following configuration is used to give access to the cluster. This file will be stored in /etc/rancher

https://docs.rke2.io/install/containerd_registry_configuration

3) Now we deploy something from our private registry onto our kubernetes

This means, we have to create a means to authenticate kubernetes so it can pull, and one way (assuming we don't use managed identities just yet) is to create a robot account create and put the secret into kubernetes

https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

What kind of artifacts can you push to Harbor, what features do you have available in a (good) registry, and how are they used?

We prepared a harbor installation for you here https://coreharbor.caas-0026.dev.austrianopencloudcommunity.org/account/sign-in?redirect_url=%2Fharbor%2Fprojects

What types of artifacts can we store in Harbor?

Example of some very useful features

Exercise: push your helm chart to harbor. 

$>helm create my_chart
Creating my_chart
$>helm package my_chart
Successfully packaged chart and saved it to: .../my_chart-0.1.0.tgz
$>echo "mypass" | helm registry login coreharbor.caas-0026.dev.austrianopencloudcommunity.org/my_project/ -u myuser --password-stdin
Login Succeeded
$>helm push my_chart-0.1.0.tgz oci://coreharbor.caas-0026.dev.austrianopencloudcommunity.org/my_project/
Pushed: coreharbor.caas-0026.dev.austrianopencloudcommunity.org/my_project/my_chart:0.1.0
Digest: sha256:7ed393daf1ffc94803c08ffcbecb798fa58e786bebffbab02da5458f68d0ecb0

Documentation contains additional uses. https://goharbor.io/docs/edge/working-with-projects/working-with-oci/working-with-helm-oci-charts/

Sign the artifact with cosign

$> cosign generate-key-pair
Enter password for private key: 
Enter password for private key again: 
Private key written to cosign.key
Public key written to cosign.pub

$>cosign sign --key cosign.key coreharbor.caas-0026.dev.austrianopencloudcommunity.org/vadim/my_chart:0.1.0
...
Pushing signature to: coreharbor.caas-0026.dev.austrianopencloudcommunity.org/vadim/my_chart

Questions?

How to avoid LeakyVessel Supply Chain Attack

What is Registry poisoning ?

Registry poisoning is a type of attack where an attacker publishes a malicious package to a package registry under a name that's similar/identical to a popular package, or that was previously used by a legitimate package. When developers install packages from the registry, they might accidentally install the malicious package instead of the one they intended to install.

In the context of Docker, an attacker could publish a malicious image to a Docker registry with a name that's similar/identical to a popular image. If a developer pulls and runs the image without verifying its authenticity, they end up running malicious code.

(credit: linkedin)

What was Leaky Vessel?

Earlier this year 2024, there was a lot of outcry about the largest jack-pot critical vulnerability, as there were 4 at once:

https://www.paloaltonetworks.com/blog/prisma-cloud/leaky-vessels-vulnerabilities-container-escape/

Video with PoC exploit

You can find the code for Image and deployment in our github on https://github.com/AustrianDataLAB/pacman/blob/features/workloadid/.github/workflows/publish-image-poison.yml

Using the following Dockerfile as inspired by the snyk-PoC exploit:

We can observe the following logs from the vulnerable kubernetes cluster (running a vulnerable runc)

And in the following video we see that the file LEAKYLEAKY appears on the host of k8s-agent-001 meaning that the container has root-access to the host during the container boot.

https://youtu.be/RNYz86uDXLc?si=SVduoE-WQsrGjt25

How to prevent poisoning on registry side?

In harbor, we have the option of using Immutability rules, and we can demonstrate that if we switch on such a rule, the "overpushing" is no longer possible and thus the poisoning a lot more difficult.

Further reading

If you wanted to take this even further, you could implement an admission controller on kubernetes side to make sure that only images that you specify to be allowed, actually are admitted:

ImagePolicyWebhook

Type: Validating.

The ImagePolicyWebhook admission controller allows a backend webhook to make admission decisions.

This admission controller is disabled by default.

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

https://github.com/howto-kubernetes-info/imagepolicywebhook

Congratulations, you've successfully completed the lecture on registries, you've signed an artifact and your helm chart should now be pushed to an OCI registry !

What's next?