Setup Cluster monitoring using Prometheus, Grafana and Loki

Prometheus is An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.
Grafana is a multi-platform open source analytics and interactive visualization web application. It can produce charts, graphs, and alerts.
Loki is a log aggregation system designed to store and query logs.

Prerequisite

Helm
Kubernetes

Setup

Install kube-premetheus-stack

Use helm to install kube-premetheus-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

then you can run helm search repo prometheus-community to see the charts

If you can't see kube-prometheus-stack you can search by type
helm search repo prometheus-community/kube-prometheus-stack

If you want to install different version, you can search the available versions
helm search repo prometheus-community/kube-prometheus-stack --versions

and install it using

helm install prometheus prometheus-community/kube-prometheus-stack --version 45.7.1 --namespace monitoring --create-namespace

kube-prometheus-stack contains:

prometheus operator
node exporter
kube state metrics
grafana

Install Loki

Before installing Loki, you need to create a Persistent Volumes if you are planning to use local storage instead of cloud storage

Create PVC

# loki-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: storage-loki-0
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ''
  claimRef:
    namespace: monitoring
    name: storage-loki-0
  hostPath:
    path: "/mnt/data/loki"
    type: DirectoryOrCreate

# loki-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: storage-loki-0
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: storage-loki-0
  storageClassName: ""  # Important: disables dynamic provisioning

type kubectl apply -f loki-pv.yaml and
type kubectl apply -f loki-pvc.yaml

Now we can install loki

helm repo add grafana http://grafana.github.io/helm-charts

helm repo update

helm search repo grafana

# you will see helm chart name 'grafana/loki'

helm install loki grafana/loki --namespace monitoring --version 6.29.0 -f loki-values.yaml

# loki-values.yaml

# Deployment mode: SingleBinary (simple), SimpleScalable (medium), Distributed (large)
deploymentMode: SingleBinary

# Number of replicas
singleBinary:
  replicas: 1

# Component replicas
write:
  replicas: 0
read:
  replicas: 0
backend:
  replicas: 0

# Caching
chunksCache:
  enabled: false
resultsCache:
  enabled: false

# Helm test
test:
  enabled: false

# All loki settings combined under one key
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem

  useTestSchema: true

  persistence:
    enabled: true
    existingClaim: storage-loki-0

Prometheus UI

To access Prometheus UI

kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090

go to http://localhost:9090

Grafana UI

To access Grafana UI

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

go to http://localhost:3000

username: admin
password: prom-operator or you can check
to make sure by typing kubectl get secret -n monitoring prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode;

Add Loki data source

In Grafana UI go to Configuration > Data sources, click on add data source and search for loki

add http://loki:3100 to URL and click on Save & Test

To test sending datas to loki

# port-forwarding the gateway to your local machine

kubectl port-forward --namespace monitoring svc/loki-gateway 3100:80

open a new tab

# send test datas

curl -H "Content-Type: application/json" -XPOST -s "http://127.0.0.1:3100/loki/api/v1/push"  \
--data-raw "{\"streams\": [{\"stream\": {\"job\": \"test\"}, \"values\": [[\"$(date +%s)000000000\", \"fizzbuzz\"]]}]}"

------
# verify that Loki did receive the data

curl "http://127.0.0.1:3100/loki/api/v1/query_range" --data-urlencode 'query={job="test"}'