Welcome back to our series on Key Management Service (KMS) in Kubernetes! In Part 1, we laid the groundwork; now, in Part 2, we're diving into the critical concept of encryption at rest.

What Exactly is Encryption at Rest?

Simply put, encryption at rest in Kubernetes refers to how the API server encrypts data before storing it in etcd. Think of etcd as the brain of your Kubernetes cluster - it's where all your cluster's configuration data, state, and secrets live.

By default, the Kubernetes API server stores resources in etcd as plain text. This means if someone gains unauthorized access to your etcd, they can read all your sensitive data, including secrets, without any effort. This is a significant security risk.

While encryption at rest applies to any Kubernetes resource, in this series, we'll continue to focus on Secrets due to their inherently sensitive nature.

The good news is Kubernetes provides a way to encrypt this data before it hits etcd. This is primarily done through the --encryption-provider-config argument passed to the kube-apiserver process, which points to a configuration file.

Introducing EncryptionConfiguration

In Kubernetes, the encryption at rest behavior is configured using an EncryptionConfiguration resource. This powerful configuration allows you to specify which resources should be encrypted and using which encryption providers.

Let's look at an example configuration to understand its structure:

# CAUTION: This is an example configuration and should NOT be used for production clusters without careful consideration.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
      - pandas.awesome.bears.example # An example custom resource API
    providers:
      # The 'identity' provider stores resources as plain text (no encryption).
      # If listed first, it means data is NOT encrypted.
      - identity: {}
      - aesgcm:
          keys:
            - name: key1
              secret: c2VjcmV0IGlzIHNlY3VyZQ== # Base64 encoded key
            - name: key2
              secret: dGhpcyBpcyBwYXNzd29yZA== # Base64 encoded key
      - aescbc:
          keys:
            - name: key1
              secret: c2VjcmV0IGlzIHNlY3VyZQ== # Base64 encoded key
            - name: key2
              secret: dGhpcyBpcyBwYXNzd29yZA== # Base64 encoded key
      - secretbox:
          keys:
            - name: key1
              secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= # Base64 encoded key
  - resources:
      - events
    providers:
      - identity: {} # Do not encrypt Events
  - resources:
      - '*.apps' # Wildcard match (Kubernetes 1.27+)
    providers:
      - aescbc:
          keys:
          - name: key2
            secret: c2VjcmV0IGlzIHNlY3VyZSwgb3IgaXMgYXQ/Cg==
  - resources:
      - '*.*' # Wildcard match (Kubernetes 1.27+)
    providers:
      - aescbc:
          keys:
          - name: key3
            secret: c2VjcmV0IGlzIHNlY3VyZSwgSSB0aGluaw==

Key Takeaways from the EncryptionConfiguration:

• resources: This array specifies which Kubernetes resources you want to apply encryption to. You can target specific resources (e.g., secrets), or use wildcard matches (e.g., '*.apps', '*.*') available in Kubernetes 1.27 and later.

• providers: This is an ordered list of encryption providers. Kubernetes attempts to use the first provider in the list to encrypt new data. When decrypting, it tries providers in order until one succeeds.

• identity: {}: This provider means no encryption; data is stored in plain text. If this is the first provider for a resource, it means new data for that resource will not be encrypted.

• aesgcm, aescbc, secretbox: These are different encryption algorithms. You define named keys (base64 encoded) for each.

Demo: Encrypting Secrets in Minikube

Let's walk through a practical example using a Minikube cluster to see encryption at rest in action.

1. Start Your Minikube Cluster:
Make sure your Minikube cluster is up and running.

2. Create the EncryptionConfiguration File:
Create a file named encryption-conf.yaml with the following content:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets # We are only encrypting secrets for this demo
    providers:
      - aescbc: # Using AES-CBC encryption
          keys:
            - name: key1
              # IMPORTANT: This is a randomly generated key.
              # Use a strong, unique, and base64-encoded key in production.
              secret: lvAp17Ae2o/yTdxz2qyC6zjVzuS+sBdhkwCccgsSsUg=
      - identity: {} # Fallback to identity (plaintext) if AES-CBC fails, or for older data

Understanding the Demo Configuration:

• We're specifically targeting secrets for encryption.

• We're using the aescbc provider with a single key named key1. Remember to use a truly random and strong key for production environments! The secret value needs to be base64 encoded.

• The identity: {} provider is included as a fallback. This is crucial for smooth rotation and decryption of older data, or if the primary encryption provider encounters an issue.

3. Configure the API Server to Use the Encryption Configuration:
Now, we need to tell the kube-apiserver to use this configuration file. In Minikube, you can achieve this by modifying the API server's Pod definition.

First, locate the kube-apiserver pod definition (it's usually in /etc/kubernetes/manifests/ on the control plane node).

Next, you need to modify the kube-apiserver static pod manifest to include the --encryption-provider-config argument and mount the directory containing your encryption-conf.yaml file.

Here's an example of how the kube-apiserver pod manifest snippet would look after modification:

apiVersion: v1
kind: Pod
metadata:
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    # ... other kube-apiserver arguments
    - --encryption-provider-config=/etc/kubernetes/enc/encryption-conf.yaml # <--- Add this line
    volumeMounts:
    - mountPath: /etc/kubernetes/enc # <--- Mount path for your config file
      name: enc-vol
      readOnly: true
    # ... other volume mounts
  volumes:
  - hostPath:
      path: /etc/kubernetes/encryption # <--- Host path where your config file lives
      type: DirectoryOrCreate
    name: enc-vol
  # ... other volumes

Important: You will need to place your encryption-conf.yaml file in the /etc/kubernetes/encryption directory on your Minikube VM (or the control plane node for a full cluster) for the hostPath volume mount to work correctly.

After saving the changes, the kube-apiserver pod will automatically restart, applying the new encryption configuration.

4. Create a Secret:
Let's create a new secret in a test namespace:

kubectl create namespace test
kubectl -n=test create secret generic new-secret --from-literal=key1=supersecret

5. Retrieve the Secret (as perceived by kubectl):
Now, let's retrieve the secret using kubectl:

kubectl get secret new-secret -o yaml -n test

You'll get output similar to this:

apiVersion: v1
data:
  key1: c3VwZXJzZWNyZXQ= # Still base64 encoded!
kind: Secret
metadata:
  creationTimestamp: "2025-05-31T10:50:26Z"
  name: new-secret
  namespace: test
  resourceVersion: "6614"
  uid: a8655bde-be5f-4624-b905-61524de56ebe
type: Opaque

Notice that the data field still shows a base64-encoded value. This is crucial: kubectl always displays secrets in their base64-encoded form. This output doesn't tell us whether the secret is encrypted at rest in etcd.

6. Verify Encryption in Etcd:
This is where the magic happens! We'll directly inspect how the secret is stored in etcd.
First, exec into the etcd-minikube pod:

kubectl -n=kube-system exec -it etcd-minikube -- sh

Then, use etcdctl to retrieve the raw value of the secret from etcd:

ETCDCTL_API=3 etcdctl --cacert /var/lib/minikube/certs/etcd/ca.crt --cert /var/lib/minikube/certs/etcd/server.crt --key /var/lib/minikube/certs/etcd/server.key get /registry/secrets/test/new-secret

You'll see output that looks something like this:

/registry/secrets/test/new-secret
k8s:enc:aescbc:v1:key1:??|??<z@0-ki_<Q?
                                       jғ?[3Ox??t%??q??d??e??%?gy??ER8{s????
                                                                            ?r?UO%{?+C??h
                                                                                         ???w?II?; ??v??r????q?????t?YA?"??j?" ??f9?$FD?9T?.F?\?<???kc/Q?W0?
                                                                                                                                                                                   ?;\U??l?n???nW?^HlA?C?Ռ]=?U{j??|??pe?
?Z?Y?XY9??
          ?uuO?2??xK[޹??~7v

This is the key difference!

• Without EncryptionConfiguration, the data in etcd would be easily readable.

• With EncryptionConfiguration applied, you see an unreadable, garbled string prefixed with k8s:enc:aescbc:v1:key1:. 

This prefix tells you:

• k8s:enc: This data is encrypted by Kubernetes.
• aescbc: The encryption algorithm used is AES-CBC.
• v1: The version of the encryption scheme.
• key1: The specific key (key1 from our EncryptionConfiguration) that was used for encryption.

This confirms that your secret is now encrypted at rest in etcd!