Episode 0x08: Nginx Ingress and Cert-Manager

Table of Contents

NOTE: Many commands in this post make use of specific constants tied to my own setup. Make sure to tailor these to your own needs. These examples should serve as a guide, not as direct instructions to copy and paste.

NOTE: Check out the final code at homelab repo on my Github account.

Introduction

In our previous episode, we successfully set up MetalLB in our cluster. However, accessing services via IP addresses isn’t ideal. Instead, we want to assign domain names or subdomains to our services. By utilizing an ingress controller, we also eliminate the need for each service to have a unique external IP. Instead, a single external IP can be used for the ingress, acting as a reverse proxy to our services.

Moreover, it’s essential to access our services over HTTPS. This requires a robust certificate management system. We will leverage cert-manager to manage our certificates seamlessly. Cert-Manager will obtain certificates from a variety of Issuers, and ensure the certificates are valid and up-to-date, and will attempt to renew certificates at a configured time before expiry.

Installation

To install both Nginx Ingress and cert-manager using Kubespray (as we did with MetalLB in the previous episode), you need to add the following lines to cluster_variables.yaml:

ingress_nginx_enabled: true
ingress_nginx_service_type: LoadBalancer
ingress_nginx_namespace: "ingress-nginx"
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
ingress_nginx_default: true

cert_manager_enabled: true
cert_manager_namespace: "cert-manager"

After adding these configurations, recreate your cluster. As per episode 5, you can do this by running:

ansible-playbook -i inventory.ini -e @cluster_variables.yaml --user=ansible playbook.yml

NOTE: The above command may be destructive. If the NFS subdir provisioner stops working after running it, follow the steps in episode 6 to reinstall it.

Defining Cluster Issuers

To generate certificates for our services, we need to define issuers. Below are configurations for letsencrypt-staging, letsencrypt-prod, and buypass-prod issuers. If you wish to use different issuers, follow their setup procedures.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <your-email-here>
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <your-email-here>
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: buypass-prod
spec:
  acme:
    server: https://api.buypass.com/acme/directory
    email: <your-email-here>
    privateKeySecretRef:
      name: buypass-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

Save these issuers’ configurations in a YAML file and apply it. All configurations use the http-01 challenge, where the CA verifies domain ownership via an HTTP request.

NOTE: The HTTP-01 challenge is a method used by Certificate Authorities (CAs) to validate domain ownership before issuing SSL/TLS certificates. In this challenge, the CA requires the domain owner to create a specific HTTP resource on their server. When the domain owner requests a certificate, the CA gives them a unique token, which must be placed in a well-known URL (e.g., http://<domain>/.well-known/acme-challenge/<token>). The CA then makes an HTTP request to this URL and verifies the presence and correctness of the token. If the token matches what the CA issued, the domain is verified, and the certificate can be issued.

Many people prefer to use the alternative to the HTTP-01 challenge, the DNS-01, which involves creating a DNS TXT record with a specific token to prove domain ownership, as it is easier to setup manually. However, we don’t want to do any manual steps, we can use http-01 challenge and cert-manager will take care of the rest for us. All we need to do is to setup correct route in nginx.

Testing Nginx Ingress and Cert-Manager

To verify the setup, deploy kuard on kuard.<your-domain-name>.com. Here’s how:

  1. Create an A DNS record for your domain.
  2. Ensure the traffic is routed to your cluster. If you are replicating my setup, configure your Nginx instances in the Edge VM and your VPS as detailed here and here.

Then apply the following yaml configuration on your cluster.

apiVersion: v1
kind: Namespace
metadata:
  name: kuard
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kuard
  name: kuard
spec:
  selector:
    matchLabels:
      app: kuard
  replicas: 1
  template:
    metadata:
      labels:
        app: kuard
    spec:
      containers:
        - image: gcr.io/kuar-demo/kuard-amd64:1
          imagePullPolicy: Always
          name: kuard
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  namespace: kuard
  name: kuard-svc
spec:
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  selector:
    app: kuard
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: kuard
  name: kuard-igs
  annotations: # These annotations are necessary.
    cert-manager.io/issuer: "letsencrypt-staging"
    acme.cert-manager.io/http01-edit-in-place: "true"
spec:
  ingressClassName: nginx
  tls: # TLS segment is also necessary, other wise, cert manager wont try to request a certificate
    - hosts:
        - kuard.<your-domain-name>.com
      secretName: kuard-tls # This is where your certificate gets stored
  rules:
    - host: kuard.<your-domain-name>.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kuard-svc
                port:
                  number: 80

Replace <your-domain-name> with your actual domain name in the above configuration. To debug, use the following commands:

kubectl get certificaterequests.cert-manager.io -n kuard
kubectl get challenges.acme.cert-manager.io -n kuard

If everything is set up correctly, you should see the kuard start page when navigating to https://kuard.<your-domain-name>.com. Note that using letsencrypt-staging will generate a warning in your browser. Replace it with letsencrypt-prod once you’re confident in your setup to avoid rate-limiting issues by Let’s Encrypt.

Remember to clean up the kuard namespace after your testing. Congratulations! You can now access your services via a domain name with a valid TLS certificate!

In the next episode, we’ll setup argocd! 🚀