Episode 0x0A: Sealed Secrets

NOTE: Many commands in this post make use of specific constants tied to my own setup. Make sure to tailor these to your own needs. These examples should serve as a guide, not as direct instructions to copy and paste.

NOTE: Check out the final code at homelab repo on my Github account.

Introduction

As explored in the previous episode, we’ve adopted the GitOps methodology. This approach posits that our Git repository serves as the authoritative source for our declarative deployment strategy. However, a important question arises: can we securely store secrets within Git?

Kubernetes secrets, in their default form, are not particularly secure, as they are only base64-encoded data. As a result, pushing them directly to Git poses a security risk. Yet, there exists a solution: encrypting these secrets before committing them to Git. This process is manageable using a tool that can decrypt them automatically back into Kubernetes secrets. One such tool is Bitnami Sealed Secrets. Sealed Secrets are designed to be safely stored inside a public repository, and they can only be decrypted by the controller residing within the target cluster.

Among the various methods for managing secrets in a Kubernetes cluster, HashiCorp Vault stands out as a powerful option. It, however, is more complex to set up and operate. As a future enhancement, I might consider migrating to Vault, but for now, we will stick with Sealed Secrets due to its simplicity in usage.

Installation

We will utilize Argo CD to install applications in our cluster. Argo CD supports deploying apps from a Helm repository. Let’s begin by creating an apps.yaml file, in which we declare the applications we plan to install in our cluster, adding one in each episode. Our first addition will be Sealed Secrets.

apiVersion: v1
kind: List
items:
  - apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: sealed-secrets
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://charts.bitnami.com/bitnami
        chart: sealed-secrets
        targetRevision: 2.3.0
        helm:
          releaseName: sealed-secrets
          parameters:
            - name: fullnameOverride
              value: sealed-secrets-controller
      destination:
        server: https://kubernetes.default.svc
        namespace: kube-system
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

The application manifest is straightforward. Notably, the syncPolicy is configured to be automated, meaning that a sync is triggered upon detecting changes in the source. Additionally, the selfHeal setting ensures that any manual changes made to the deployment will be automatically reverted, thereby maintaining the declared state within the namespace.

Since Application is a Kubernetes Custom Resource Definition (CRD) introduced by Argo CD in our cluster, we can apply the above configuration using kubectl:

kubectl apply -f apps.yaml

This command will create the Sealed Secrets application. Log in to the Argo CD interface to verify the application’s status:

Kubeseal CLI

We can create sealed secrets using the kubeseal CLI. To install kubeseal, follow the instructions in its documentation.

NOTE: The kubeseal CLI attempts to access the controller named sealed-secrets-controller. This is why we employ the fullnameOverride parameter when installing sealed secrets. Additionally, kubeseal assumes the controller is installed in the kube-system namespace by default, which explains the specified namespace in our deployment. With these configurations, we can use kubeseal without additional parameters.

Now, let’s create our first sealed secret:

kubectl create secret generic test-secret --dry-run=client --from-literal="key=mysecret" --output=yaml | kubeseal -o yaml

The above command creates a secret in dry-run mode (i.e., it outputs the secret without actually creating it) and then kubeseal is used to encrypt it. The resulting output will be:

---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: test-secret
  namespace: default
spec:
  encryptedData:
    key: AgANV/MahwI... # trimmed for brevity
  template:
    metadata:
      creationTimestamp: null
      name: test-secret
      namespace: default

This sealed secret can be safely stored in our public repository. Setting up and using Sealed Secrets is truly that straightforward!

In the next episode, we will install another essential component in our cluster: Object Storage. Stay tuned! 🪣 🚀