Introduction

When you deploy an application on Kubernetes, it may need secrets to function. These secrets can be passwords, SSH keys, tokens, etc.

But how do you manage these secrets? How do you store them? How do you share them with developers or allow developers to create them?

For example, you want to define a secret corresponding to an API token. And in a GitOps approach, you store your Kubernetes manifests in a Git repository. How can you ensure that the token is not in plain text in the Git repository?

There are many solutions to manage secrets in Kubernetes, I will present one to you: Sealed-Secret.

Sealed secret

Sealed Secret

Sealed Secrets is an open-source project developed by Bitnami. It allows encrypting Kubernetes secrets using a pair of public/private keys. The private key is present on the Kubernetes cluster, and the public key is used by developers to encrypt the secrets.

Once the secret is created and encrypted, only the Kubernetes cluster can decrypt it (or a person with the private key).

Installation

To install Sealed Secrets, Helm is used. It can be installed using the following command:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets -n kube-system sealed-secrets/sealed-secrets

Now that we have our Sealed secret installed, we can install Kubeseal which will help us to:

  • Create our sealed secrets (using the public key)
  • Decrypt our sealed secrets (via the cluster, or via the private key)
KUBESEAL_VERSION='0.24.0'
wget "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION:?}/kubeseal-${KUBESEAL_VERSION:?}-linux-amd64.tar.gz"
tar -xvzf kubeseal-${KUBESEAL_VERSION:?}-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Create a Sealed Secret

I will create my generic Kubernetes secret using the usual syntax. After that, I will encrypt it with Kubeseal, which will communicate with the Sealed-Secret controller. I must provide it with the name of the service linked to the controller pod as well as the namespace where it is located.

apiVersion: v1
kind: Secret
metadata:
  name: mon-secret
data:
  user: UXVlbnRpbg==
  slogan: Vml2ZSBsZSBjYWbDqQ==

cat secret.yaml | kubeseal \
    --controller-namespace kube-system \
    --controller-name sealed-secrets \
    --format yaml \
    > sealed-secret.yaml

With this command, kubeseal will connect to the sealed-secrets service to retrieve the public key and encrypt the content of our YAML. Here is the result of our command:

---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mon-secret
  namespace: default
spec:
  encryptedData:
    user: AgBdt5uTaeTSpHQ4gDFw9E3vu5ihAJvqEnO3unMK8Q7UroMl+nq/CGdqePocumtDYVAW2U5wGWZFa/KELnK7AOJ8Es4SjloDXmeq620/b3nRbbv825ixdo80vstrzH6RMBUoHKhaagTy/p3C3atACxpW9C++HUXJ/GmBfVTzGWKXUh2OQ3STAx4zDRvwlTs7d3Ot734DSBA6rAK45xzTIiXDGt3rv3TwN7te1kXxM/TMWUPB6iczYVhCb2ruaVfHAdlQnO6ytdbiUO/J5Bd/EdBhVFwcbXjsjcS6wHQeIwFDhQu1mEfnDbd4e9QCH0ef3QVAF/TlnMdwKaq5KWPnakIYABD5wWsMe73XSN9CWG0qbtjirRuZREeCbbGfjetsJFPTEgKSgnXwC/dtfXCppBSmHRN/2y4N5DNh9nzcpQQDX6mENbiL8xmxr7A31/6QHdjwtCMWbPNWpfalvS/Vp3Lr0/ZThV6KHmo73QB6RBVrhcSWjQQYi94CI/RFQWVT+LUxwrWbLBHAa7LnRElcOQ401/rLrvs+AU9eS2ni6B5snMUhYKMw7anhr6UAe21+M+UKuMK/C7n9P91avcfxpEBVPJByYZuTeqyvIsmE6uHjswZcZU7g05rHVqgfjQmuHKCEdj4hmxW4bM2QZowdH7L870Xz89txbzTKc25Alg74S80b0sw/jV7Yct/oAAtO/E34BRN8zFajZQ==
    slogan: AgBanbBAe3Viz11uu5Nc8TWRP/1FHfKjPKvsmS9RSNPkIeoO1QDisN6YB6R81ftbtzhe6YuLEYg9uSdO6UH6kRcrzB9PhouAjqs0jug9FPBQnDDoh3DL3nBfnKwfz8RWoF7upFazREqPApVZP2sJGACxG6mZJNzEPKw+FOuXGGwMDZu/GMDav4dtsjUh8073dAsDZKTzD6AlfAd8xzZ/dh2XWhCt5fL+Dlfbc8lCRh8dtD4KC7iBUJL3Qsv6dfSvLrUbdTIejiZOd16TkD1SccQQiDztDqIJvLxuHcNPxrxn8nnvTC7pqDBl6VpHUa6dzxFhZKG20mx7hSvByAR9RJu3reepd2cTm5N7XXPFJk0MPURsWRh9vsqpvbJF8j/6Bs1gYAblZEE9seC8ySNslzlUaUgLvz+MjZfDbdJ7fHd35O2P7adOEuUEmqrH7j7hXupcs4OJ35+4vUwdSeG7kGz0txlqVa5z3c8kkAMfWG80DFIyxwHrCl7CYlXYult5FUBAXCxuOYntvtDzfMNVdmsXxDPWZ14UezDRdZSWW5hCMeDKEdk5T6h/nm7AFY5+UOYeIDXJeseSoazY7lJRM6keG9xAba1Pn8lq712qqDxWuauR0YZN/TanWSa0iqZEwqoksnMwEF1vuMXo3Sr7ATVjPatWAq84uNh7p2G+DREOKDQUZgxTGBsZblr8aZSN/rD4Zea2ZeSNS0v/GITZLw==
  template:
    metadata:
      creationTimestamp: null
      name: mon-secret
      namespace: default

Tip

In the template section, we can add annotations or labels that will be applied to the secret generated by the controller.

This file can only be decrypted by the private key present on my cluster. Once we send the sealed-secret.yaml file to the cluster, the controller will automatically create a generic Kubernetes secret using its private key.

The controller used is the one present in the kube-system namespace, and it is possible to have multiple controllers on the same cluster.

But what if we don’t have access to the cluster for the kubeseal command to retrieve the public key?

It is possible to copy the key present on our kubeseal and send it to our developers who do not have access to the cluster.

kubeseal --fetch-cert > cert.pem

Tip

It is also possible to retrieve the same file at the path /v1/cert.pem from the web server of sealed-secret.

wget http://sealed-secrets-controller.kube-system.svc.cluster.local:8080/v1/cert.pem -O /dev/stdout -q

Once we have this file, we can repeat the same command to create our sealed-secret.yaml file, but specifying --cert instead of --controller-(namespace|name).

cat secret.yaml | kubeseal \
    --cert cert.pem \
    --format yaml \
    > sealed-secret.yaml

The generated secret is identical to the previous one, but without the need to connect to the cluster.

After sending the sealed-secret.yaml file to the cluster, we can see that the controller has created a generic Kubernetes secret by creating this pod:

apiVersion: v1
kind: Pod
metadata:
  name: petit-pod
spec:
 containers:
   - name: busybox
     image: busybox
     command: ['sh', '-c', 'echo "$SECRET_USER, $SECRET_SLOGAN"']
     env:
       - name: SECRET_USER
         valueFrom:
           secretKeyRef:
             name: mon-secret 
             key: user
       - name: SECRET_SLOGAN
         valueFrom:
           secretKeyRef:
             name: mon-secret 
             key: slogan
 restartPolicy: Never

Here is the result :

kubectl apply -f pod.yml 
# pod/petit-pod created
kubectl logs pod/petit-pod
# Quentin, Long live coffee

Tip

There is nothing preventing you from having multiple key pairs used by sealed-secret.

You can keep the key generated by sealed-secret and add the key from another cluster. We will see how to do this in the next section to export the keyring.

Saving your keys

If you want to reinstall your cluster or have the same key in two different clusters, you can export the key used by Sealed-Secret in the following way:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secret-key.yaml

The generated file can be imported into a cluster using the kubectl command.

kubectl apply -n kube-system -f sealed-secret-key.yaml

Metrics

Following my article on Prometheus, I have resolved to always show you the exporters present in the tools I use (Yes, it will annoy you, yes, I will still do it 😇).

It is possible to retrieve metrics from sealed-secret via the famous /metrics.

Conclusion

Sealed-secret is a very useful tool for managing the secrets of your Kubernetes cluster. Since I travel a lot: I use it a lot to work offline. I compare Sealed-Secret a lot with the combo Sops-Kustomize (which I currently use in my GitOps project), which is a very good alternative that meets the same need: encrypting YAML manifests and decrypting them with a private key present on the cluster.

This page is not as long as those on Prometheus or Consul, but I believe that Sealed-Secret is a tool worth knowing and practical in many cases that deserves to be shared.

I hope you enjoyed this article, feel free to give me feedback on Twitter or LinkedIn.