Automatic OpenTelemetry for Kubernetes Jobs (Hands on)
Adam Gardner

Adam Gardner @agardnerit

About: CNCF Ambassador, DevRel working in Observability. I blog at https://agardner.net and YouTube at https://www.youtube.com/@agardnerit

Joined:
Dec 8, 2023

Automatic OpenTelemetry for Kubernetes Jobs (Hands on)

Publish Date: Jan 15 '24
0 0

Use the tracepusher Kubernetes Operator to automatically generate OpenTelemetry traces for Kubernetes Job and CronJob resources.

Step 0: Create a cluster & Start Jaeger

If you don't already have a cluster, create one now:

kind create cluster

# Or install jaeger on Kubernetes with the jaeger operator
# https://www.jaegertracing.io/docs/1.53/operator/
docker run --rm --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.53
Enter fullscreen mode Exit fullscreen mode

Step 1: Install tracepusher Custom Resource Definitions on cluster

kubectl apply -f https://raw.githubusercontent.com/agardnerIT/tracepusher/main/operator/crds.yml
Enter fullscreen mode Exit fullscreen mode

Step 2: Create a JobTracer

Modify the OpenTelemetry collector URL as appropriate for your setup and then apply the file. One JobTracer per namespace please!

This file instructs tracepusher to monitor for all Job resources in the given namespace (you can ignore on a job-by-job basis with annotations that you'll see next):

cat <<EOF | kubectl create -f -
---
apiVersion: tracers.tracepusher.github.io/v1
kind: JobTracer
metadata:
  name: tracer
  namespace: default
spec:
  # Specify your OpenTelemetry collector endpoint
  # Here I've given the docker URI
  collectorEndpoint: "http://host.docker.internal:4318"
EOF
Enter fullscreen mode Exit fullscreen mode

Step 3: Create a Normal k8s Job / CronJob

Create a Job / CronJob as normal. The tracepusher annotations are optional and will override the behaviour of the above JobTracer

cat <<EOF | kubectl create -f -
---
apiVersion: batch/v1
kind: Job
metadata:
  name: pi
  namespace: default
  #annotations:
  #  tracepusher/ignore: "true"
  #  tracepusher/collector: "http://example.com:4318"
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl:5.34.0
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 0
EOF
Enter fullscreen mode Exit fullscreen mode

tracepusher job tracer result

Comments 0 total

    Add comment