Episode 0x0C: Private Container Registry
Table of Contents
NOTE: Many commands in this post make use of specific constants tied to my own setup. Make sure to tailor these to your own needs. These examples should serve as a guide, not as direct instructions to copy and paste.
NOTE: Check out the final code at homelab repo on my Github account.
Introduction
A simplistic view of Continuous Integration and Continuous Deployment (CI/CD) is as follows: you code your application, test it, build it, create a container image that includes all the dependencies required for your application to run, and then push that image into a container registry. Up to this part is Continuous Integration (CI). Continuous Deployment (CD) involves pulling this image, deploying it in a cluster, ensuring it remains operational, scaling it as necessary, and other related tasks.
When we self-host our server, we might prefer not to use external container registries like Docker Hub. Instead, we can leverage the object storage we configured in the previous episode to set up our own container registry.
For this purpose, we will use the official Docker registry. Let’s get started.
Registry Deployment
Similar to our Minio deployment, we will deploy our registry. Begin by generating authentication credentials.
htpasswd -Bc .htpasswd <your-username>
You will be prompted for a password. Once entered, a .htpasswd file will be created containing your <your-username>:<your-password-hash>. Next, create a sealed secret from this file.
kubectl create secret generic -n registry registry-auth-secret --dry-run=client --from-file=.htpasswd --output yaml | kubeseal -o yaml
Save the output in a YAML file, such as secrets.yaml. Now, create the configuration file for our registry, which we will also define as a sealed secret:
kubeseal apply --dry-run=client -o yaml -n registry -f - <<EOF | kubeseal -o yaml
apiVersion: v1
kind: Secret
metadata:
name: registry-config-sec
namespace: registry
stringData:
config.yml: |
version: 0.1
log:
level: debug
formatter: json
fields:
service: registry
accesslog:
disabled: false
storage:
s3:
accesskey: <minio-access-key>
secretkey: <minio-secret-key>
region: us-east-1
bucket: container-repo
regionendpoint: minio-svc.minio.svc.homelab-k8s:9000 # Minio address
secure: false
v4auth: true
chunksize: 5242880
rootdirectory: /
delete:
enabled: true
maintenance:
readonly:
enabled: false
redirect:
disable: true
http:
addr: :5000
secret: <a-local-deployment-secret>
host: https://docker.registry.<your-host-name>.com
headers:
X-Content-Type-Options: [nosniff]
EOF
This configuration allows the registry to access Minio to store images. Also, checkout the other available options by checking out the official documentations. Ensure you have an access token by logging into Minio and navigating to access keys to create one:
Next, create an alias for this access key using the Minio Client and create a bucket for storing container images, named container-repo, or another name of your choosing; just update the configuration accordingly.
mc alias set registry https://minio.api.<your-domain>.com <access-key> <secret-key>
mc mb registry/container-repo
Also, set the regionendpoint to your Minio instance using the internal FQDN format DNS name in the format servicename.namespace.svc.clustername. svc.clustername can also be omitted.
With all configurations in place, define the deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry-dpl
namespace: registry
spec:
replicas: 1
selector:
matchLabels:
app: registry-app
template:
metadata:
labels:
app: registry-app
spec:
containers:
- name: registry
image: registry:2
ports:
- containerPort: 5000
volumeMounts:
- name: config-volume
mountPath: /etc/docker/registry
- name: htpasswd
mountPath: /auth
env:
- name: REGISTRY_AUTH
value: htpasswd
- name: REGISTRY_AUTH_HTPASSWD_PATH
value: /auth/.htpasswd
- name: REGISTRY_AUTH_HTPASSWD_REALM
value: Registry Realm
volumes:
- name: config-volume
secret:
secretName: registry-config-sec
- name: htpasswd
secret:
secretName: registry-auth-secret
And then, the service and ingress configurations:
---
apiVersion: v1
kind: Service
metadata:
namespace: registry
name: registry-svc
spec:
ports:
- port: 80
targetPort: 5000
protocol: TCP
name: http-port
selector:
app: registry-app
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: registry
name: registry
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
acme.cert-manager.io/http01-edit-in-place: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "500m"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
ingressClassName: nginx
tls:
- hosts:
- docker.registry.<your-domain>.com
secretName: registry-tls-ingress
rules:
- host: docker.registry.<your-domain>.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: registry-svc
port:
name: http-port
We have configured ingress multiple times, so I won’t delve into those details. Just ensure that the proxy-body-size is set to allow uploading large image files.
ArgoCD Application
Just as we did with Minio, we’ll create an ArgoCD application for our registry. Append the following configuration to apps.yaml:
- apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: registry
namespace: argocd
spec:
destination:
namespace: registry
server: https://kubernetes.default.svc
project: default
source:
path: applications/registry
repoURL: https://github.com/<your-github-repo>.git
syncPolicy:
automated:
prune: true
syncOptions:
- CreateNamespace=true
This configuration creates an ArgoCD application for our deployment files, assuming they are in the applications/registry directory. Remember to replace <your-github-repo> with the correct registry URL.
Apply this configuration by running:
kubectl apply -f app.yaml
And watch Argo CD magic.
Testing the Container Registry
While configuring my desktop Docker client to recognize this registry proved challenging (this is Chat GPTs way of telling I couldn’t do it!), trust me, Kubernetes can pull images from it just fine. You can manually test the API as shown below:
UPDATE: After several weeks of debugging, I finally discovered the root of the issue. By design, the registry redirects requests to blobs directly to the bucket storage, bypassing the container registry to reduce unnecessary traffic. This works seamlessly with cloud-based object storage like S3 buckets. However, in our setup, requests were being redirected to minio-svc.minio.svc.homelab-k8s:9000, the address of our Minio instance. Since minio-svc.minio.svc.homelab-k8s:9000 is an FQDN specific to our Kubernetes cluster, it couldn’t be resolved from our desktop PCs, causing the requests to fail. Frustratingly, Docker Desktop for Mac did not display a useful error message to diagnose this issue initially.
To fix this, we added the following configuration to our registry to disable the redirect:
redirect:
disable: true
With this change, everything worked as expected.
Let’s now run a test to ensure our container registry is functioning correctly. Create a Simple Dockerfile:
FROM alpine:latest
Build and tag the Dockerfile for your registry:
docker build -f Dockerfile -t docker.registry.<your-domain-name>.com/alpine:test .
Login to your private container registry and push the image:
docker login docker.registry.<your-domain-name>.com -u <your-username> -p <your-password>
docker push docker.registry.<your-domain-name>.com/alpine:test
Now you should be able to curl your image in the registry
curl -u <your-username>:<your-password> -X GET https://docker.registry.<your-domain>.com/v2/alpine/tags/list
Conclustion
With ArgoCD, Minio, Sealed Secrets, and a container registry in place, our setup is well-equipped to deploy applications. In the next episode, we’ll introduce a workflow engine to orchestrate our CI steps, specifically exploring Argo Workflows. Stay tuned! 🚀