When it comes to managing APIs in Kubernetes, APISIX offers a powerful and flexible platform. Recently, I encountered a unique challenge while integrating TLS certificates with APISIX,specially when automating certificate updates managed by cert-manager. cert-manager generates secrets in a standardized format (tls.crt
, tls.key
, ca.crt
), but APISIX expects a different format: (cert
, key
) - ApisixTls CRD
. Here's my journey and the solution that eventually worked, with Argo Events providing a real-time, automated workflow.
Problem Overview
Challenge: cert-manager automatically generates and renews certificates but stores them in a Kubernetes Secret object with keys like tls.crt
and tls.key
. However, APISIX requires a specific format - a Kubernetes secret with keys named cert
(for the certificate) and key
(for the private key) - interesting why Apache APISIX? Without the correct format, APISIX wouldn't be able to use the updated certificates.
Possible solutions:
- When provisioning with Terraform, you can create the Secret and ApisixTls CRD using an intermediate secret. However, remapping the TLS secret during Terraform provisioning doesn’t update the mapped secret and can only handle the key’s state.
Goals:
- Automate the TLS certificate injection into APISIX.
- Ensure the process triggers whenever a TLS secret is updated or created by cert-manager.
- Use Argo Events to listen for changes in TLS secrets, transforming them and injecting them in the APISIX-compatible format.
Solution Architecture Overview
Here's how the solution is structured:
- cert-manager generates the TLS certificates.
- Argo Events listens for events related to TLS secret creation or updates and sends corresponding API requests to the APISIX Admin API.
- APISIX ingests the transformed secrets and applies them automatically.
Tools and Technologies
- APISIX: Manages and routes API traffic.
- cert-manager: Manages and renews TLS certificates automatically.
- Argo Events: Observes changes in TLS secrets and triggers automation pipelines.
Prerequisites: Installing APISIX, cert-manager, and Argo Events
Step 1: Install APISIX
To install APISIX, follow the APISIX documentation. In this article, we assume you’ve installed APISIX with the Admin API in the gateway namespace.
Step 2: Set up cert-manager
cert-manager installation is straightforward using Helm:
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace
Step 3: Install Argo Events
Argo Events can also be installed with Helm:
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argo-events argo/argo-events --namespace argo --create-namespace
The installation process is described briefly. Please refer to the corresponding documentation or contact me for
details.
Solution: Configuring Argo Events to Watch for cert-manager TLS Secrets
To make the solution work, I set up Argo Events to watch for changes in cert-manager's TLS secrets. Here's the YAML manifest setup I used to transform cert-manager secrets into a format compatible with APISIX.
Step 1: Create Service Account and Roles
# serviceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: argo-events-sa
namespace: argo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: secrets-mapper-role
rules:
- apiGroups: [ "" ]
resources: [ "secrets" ]
verbs: [ "*" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: argo-secret-mapper-role-binding
subjects:
- kind: ServiceAccount
name: argo-events-sa
namespace: argo
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
Step 2: Configure the EventBus
Define an EventBus to manage NATS messaging within Argo Events.
# eventBus.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
name: default
namespace: argo
spec:
nats:
native: { }
Step 3: Set Up the Event Source for Watching TLS Secret Changes
This event source watches for any changes to Kubernetes TLS secrets annotated with apisix.io/tls. When it detects an
addition or update, it will trigger a transformation process.
# eventSource.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: secrets-mapper-event-source
namespace: argo
spec:
template:
serviceAccountName: argo-events-sa
resource:
apisix-tls-secret:
version: v1
resource: secrets
eventTypes:
- ADD
- UPDATE
- DELETE
filter:
fields:
- key: metadata.annotations.apisix\.io\/tls
operation: "="
value: "true"
apisix-mtls-secret:
version: v1
resource: secrets
eventTypes:
- ADD
- UPDATE
- DELETE
filter:
fields:
- key: metadata.annotations.apisix\.io\/tls
operation: "="
value: "mtls"
Step 4: Create a secret with Admin token
# apisix-creds.yaml
kind: Secret
metadata:
name: apisix-admin-credentials
namespace: argo
data:
admin: base64-encoded-apisix-admin-token
We use the reflector’s annotations to create the secret in the gateway namespace, automatically updating the secret in
the Argo namespace during credential rotation. That’s another story, though!
Step 5: Create the Sensor to Transform and Inject Secrets into APISIX
The sensor will transform the secret fields and inject them into APISIX. This involves reading the secret data,converting it to base64-decoded strings, and sending it to the APISIX admin API.
# sensor.yaml
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: tls-apisix-registrar
namespace: argo
spec:
template:
serviceAccountName: argo-events-sa
container:
env:
- name: LOG_LEVEL
value: error
dependencies:
- name: tls-secret
eventSourceName: secrets-mapper-event-source
eventName: apisix-tls-secret
transform:
jq: |
.url = "http://apisix-admin.gateway.svc.cluster.local:9180/apisix/admin/ssls/" +
.body.metadata.annotations["cert-manager.io/certificate-name"] + "-" +
.body.metadata.namespace |
.method = if .type == "ADD" or .type == "UPDATE" then "PUT" else "DELETE" end |
.certs.cert = (.body.data["tls.crt"] | @base64d) |
.certs.key = (.body.data["tls.key"] | @base64d) |
.certs.snis = (.body.metadata.annotations["cert-manager.io/alt-names"] | split(",")) |
.certs.type = "server"
- name: mtls-secret
eventSourceName: secrets-mapper-event-source
eventName: apisix-mtls-secret
transform:
jq: |
.url = "http://apisix-admin.gateway.svc.cluster.local:9180/apisix/admin/ssls/" +
.body.metadata.annotations["cert-manager.io/certificate-name"] + "-" +
.body.metadata.namespace |
.method = if .type == "ADD" or .type == "UPDATE" then "PUT" else "DELETE" end |
.certs.cert = (.body.data["tls.crt"] | @base64d) |
.certs.key = (.body.data["tls.key"] | @base64d) |
.certs.ca = (.body.data["ca.crt"] | @base64d) |
.certs.snis = (.body.metadata.annotations["cert-manager.io/alt-names"] | split(",")) |
.certs.type = "server"
triggers:
- template:
name: log-trigger
conditions: "tls-secret || mtls-secret"
log:
intervalSeconds: 20
- template:
name: apisix-register-tls
conditions: "tls-secret"
http:
url: "{url}"
method: "PUT"
headers:
content-type: "application/json"
payload:
- src:
dependencyName: tls-secret
dataKey: certs.snis
useRawData: true
dest: snis
- src:
dependencyName: tls-secret
dataKey: certs.cert
dest: cert
- src:
dependencyName: tls-secret
dataKey: certs.key
dest: key
- src:
dependencyName: tls-secret
dataKey: certs.type
dest: type
secureHeaders:
- name: "X-API-KEY"
valueFrom:
secretKeyRef:
name: apisix-admin-credentials
key: admin
parameters:
- src:
dependencyName: tls-secret
dataKey: url
dest: http.url
- src:
dependencyName: tls-secret
dataKey: method
dest: http.method
- template:
name: apisix-register-mtls
conditions: "mtls-secret"
http:
url: "{url}"
method: "PUT"
headers:
content-type: "application/json"
payload:
- src:
dependencyName: mtls-secret
dataKey: certs.snis
useRawData: true
dest: snis
- src:
dependencyName: mtls-secret
dataKey: certs.cert
dest: cert
- src:
dependencyName: mtls-secret
dataKey: certs.key
dest: key
- src:
dependencyName: mtls-secret
dataKey: certs.type
dest: type
- src:
dependencyName: mtls-secret
dataKey: certs.ca
dest: client.ca
secureHeaders:
- name: "X-API-KEY"
valueFrom:
secretKeyRef:
name: apisix-admin-credentials
key: admin
parameters:
- src:
dependencyName: mtls-secret
dataKey: url
dest: http.url
- src:
dependencyName: mtls-secret
dataKey: method
dest: http.method
retryStrategy:
steps: 3
duration: 3s
policy:
status:
allow:
- 200
- 201
Replace the Apisix Admin API url if it doesn't match your configuration: http://apisixadmin.gateway.svc.cluster.local:9180/
Apply all manifests via kubectl or other tools.
kubectl apply -f .
Result verification:
Check the pods on the argo namespace, verify specific pods are running and ready.
kubectl get pods | grep -E 'bus|mapper|registrar'
eventbus-default-stan-0 2/2 Running 0 76m
eventbus-default-stan-1 2/2 Running 0 76m
eventbus-default-stan-2 2/2 Running 0 75m
secrets-mapper-event-source-eventsource-wkrr9-7cd7d7d6cb-db74n 1/1 Running 0 76m
tls-apisix-registrar-sensor-rwh7f-fcf8d5bcc-6qtsd 1/1 Running 0 18m
Testing certificate registration:
# certificate/test.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned-issuer
namespace: argo
spec:
selfSigned: { }
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: apisix-example-cert
namespace: argo
spec:
secretName: apisix-example-tls # Name of the Secret that will be created
duration: 2160h # 90d
renewBefore: 360h # 15d
commonName: example.com
dnsNames:
- example.com
- www.example.com
- api.example.com
issuerRef:
name: selfsigned-issuer
kind: Issuer
secretTemplate:
annotations:
apisix.io/tls: "true" # annotation for eventsource
Apply the issuer and certificate:
kubectl apply -f ./certificate/
Verify the certificate added on Apisix Dashboard:
Delete the test certificate – verify it has been deleted on the APISIX Dashboard or API:
For debugging, check the logs from the corresponding pod:
tls-apisix-registrar-sensor-…
Important Note:
This solution does not automatically register existing certificates in APISIX. To sync previously created certificates, you should:
- Delete the existing TLS Secret (managed by Cert-Manager).
- Let Cert-Manager recreate it - triggering Argo Events to register it in APISIX.
Conclusion:
Managing TLS certificates in Kubernetes can be complex, especially when integrating with API gateways like APISIX.
Cert-Manager simplifies certificate issuance and renewal, but APISIX requires certificates in a specific format – creating a need for automation.
- In this guide, we explored how Argo Events bridges this gap by:
- Detecting changes to Cert-Manager-generated TLS Secrets in real time.
- Transforming the certificate data into APISIX's expected format (cert and key).
- Automatically updating APISIX via its Admin API whenever a certificate is created, updated, or deleted.
Key Takeaways
Fully Automated Certificate Sync - No manual intervention needed for new or renewed certificates.
Real-Time Updates - APISIX stays in sync with Cert-Manager without delays.
Scalable & Kubernetes-Native - Works seamlessly with existing K8s tooling.
🚀 Need a Production-Ready Kubernetes Solution?
I've battle-tested this solution in my platform Laralord.dev where the base package includes fully automated provisioning of:
- 🚦 APISIX (API Gateway)
- 🔐 Cert-Manager (TLS Automation)
- ⚡ ArgoCD (GitOps)
- ...and more!
✨ Why Choose Laralord?
Feature | Benefit | |
---|---|---|
Pre-Configured Automation | APISIX and Cert-Manager ready out-of-the-box - save dozens of configuration hours | ⚙️ |
Simplified Deployments | Designed for Laravel but works with any app stack - perfect for SaaS/microservices | 🚀 |
Enterprise-Grade Infrastructure | Built-in multi-tenancy, auto-scaling and security hardening | 🔒 |
👉 Visit Laralord.dev to supercharge your Kubernetes workflows today!
📚 Further Reading
💬 Have questions or experiences to share?
Drop a comment below! 👇 We'd love to hear about your implementation!