🚀 Automating Kubernetes Sealed Secrets Management with Jenkins in a Multi-Cloud Environment

🗒️ Introduction

Managing sensitive data like API keys, passwords, and certificates securely in Kubernetes can be challenging—especially in enterprise environments with multiple clusters. In this guide, we'll explore how to automate Kubernetes Sealed Secrets management using Jenkins, ensuring security, scalability, and efficiency across multi-cloud environments such as AKS, GKE, and EKS.

🛡️ Background

In enterprise environments, sensitive information like API keys, passwords, and certificates must be managed securely. Kubernetes, widely used for container orchestration, stores such data as secrets. However, plain-text Kubernetes secrets are not safe for version control systems or manual handling.

To address this, Sealed Secrets encrypt sensitive data so it can be safely stored and shared. Deploying and managing these Sealed Secrets efficiently in Amazon AKS requires automation to ensure security, scalability, and ease of use.

⚠️ The Challenge

Enterprises often face several challenges:

Security Risks: Storing secrets in plain text makes them vulnerable to unauthorized access.
Manual Errors: Manual handling increases the risk of mistakes.
Complex Management: Managing secrets across environments (dev, staging, prod) is time-consuming.
Lack of Automation: Kubernetes lacks built-in automation for secret encryption and deployment.

💡 The Solution: Jenkins + Docker-in-Docker (DIND)

A Jenkins pipeline, combined with a Docker-in-Docker (DIND) container, automates the secure management of Sealed Secrets in AKS clusters. Key features include:

🔐 Strong Encryption: Uses the Sealed Secrets Controller, ensuring only the AKS cluster can decrypt data.
📦 Centralized Management: Jenkins standardizes secret generation and deployment across environments.
🛡️ Security Compliance: Secrets are encrypted before storage to meet strict security standards.
🔑 RBAC Integration: Kubernetes Role-Based Access Control (RBAC) adds an extra layer of security.

🔗 Check the GitHub Repository for Reference

⚙️ Jenkins Pipeline Workflow

1️⃣ Prepare the Workspace

Set up the environment for processing secrets and generating artifacts:

mkdir -p /tmp/jenkins-k8s-apply
mkdir -p ${WORKSPACE}/sealed-secrets-artifacts
rm -f /tmp/jenkins-k8s-apply/* || true
ls -la /tmp/jenkins-k8s-apply || echo "Directory is empty"

Why? A clean workspace ensures no residual sensitive data remains. ✅

2️⃣ Process Base64 Encoded Secrets

Decode the provided secrets.yaml file:

echo ${SECRETS_YAML} > /tmp/jenkins-k8s-apply/secrets.yaml.b64
base64 --decode < /tmp/jenkins-k8s-apply/secrets.yaml.b64 > /tmp/jenkins-k8s-apply/secrets.yaml
ls -l /tmp/jenkins-k8s-apply/secrets.yaml
head -n 5 /tmp/jenkins-k8s-apply/secrets.yaml | grep -v 'data:' || echo "File appears to be empty"

3️⃣ Apply Kubernetes Config & Fetch Public Certificate

Fetch the Sealed Secrets Controller’s public certificate:

ls -l ${KUBECONFIG}  # Validate kubeconfig

docker run --rm \
  -v ${KUBECONFIG}:/tmp/kubeconfig \
  -v /tmp/jenkins-k8s-apply/secrets.yaml:/tmp/secrets.yaml \
  docker-dind-kube-secret kubeseal \
    --controller-name=sealed-secrets \
    --controller-namespace=kube-system \
    --kubeconfig=/tmp/kubeconfig \
    --fetch-cert > /tmp/jenkins-k8s-apply/sealed-secrets-cert.pem

ls -l /tmp/jenkins-k8s-apply/sealed-secrets-cert.pem  # Validate certificate

4️⃣ Create Sealed Secrets

Encrypt the secrets using the public certificate:

docker run --rm \
  -v ${KUBECONFIG}:/tmp/kubeconfig \
  -v /tmp/jenkins-k8s-apply/secrets.yaml:/tmp/secrets.yaml \
  -v /tmp/jenkins-k8s-apply/sealed-secrets-cert.pem:/tmp/sealed-secrets-cert.pem \
  docker-dind-kube-secret sh -c "kubeseal \
    --controller-name=sealed-secrets \
    --controller-namespace=kube-system \
    --format yaml \
    --cert /tmp/sealed-secrets-cert.pem \
    --namespace=${NAMESPACE} \
    < /tmp/secrets.yaml" > ${WORKSPACE}/sealed-secrets-artifacts/sealed-secrets.yaml

5️⃣ Generate Documentation

Create metadata to track secret deployments:

echo "Generated on: $(date)" > ${WORKSPACE}/sealed-secrets-artifacts/README.txt
echo "Namespace: ${NAMESPACE}" >> ${WORKSPACE}/sealed-secrets-artifacts/README.txt
echo "Controller: sealed-secrets" >> ${WORKSPACE}/sealed-secrets-artifacts/README.txt
echo "Controller Namespace: kube-system" >> ${WORKSPACE}/sealed-secrets-artifacts/README.txt

🎯 Key Benefits

🔒 Enhanced Security: Encryption before storage mitigates the risk of data leaks.
⚡ Automation: Reduces manual effort, ensuring consistent workflows.
🌍 Scalability: Easily handles multiple environments and clusters.
✅ Compliance: Simplifies regulatory audits with secure secret management.

Seamlessly integrates with existing CI/CD workflows and AWS services—perfect for scaling across clusters. 🚀

📦 Part 2: Automating Kubernetes Sealed Secrets in Multi-Cloud Environments

In Part 2, we’ll expand this approach to multi-cloud environments, covering:

AKS (Non-Production) ✅
GKE (Production Cluster 1) ☁️
EKS (Production Cluster 2) 🛡️

Stay tuned for advanced multi-cloud secret management techniques! 💼

Dinesh Reddy @jettidi