Cilium Monitoring using Prometheus & Grafana

Table of Contents

We will use 2 approaches

  1. If you don’t have an existing Prometheus and Grafana stack running on your Cluster
  2. If you have an existing Prometheus and Grafana stack running on your Cluster

Approach 1. If you don’t have an existing Prometheus and Grafana stack running on your Cluster

Install Prometheus & Grafana

  • Grafana: A visualization dashboard with Cilium Dashboard pre-loaded.

  • Prometheus: a time series database and monitoring system.

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.6/examples/kubernetes/addons/prometheus/monitoring-example.yaml

Example Output

namespace/cilium-monitoring created
serviceaccount/prometheus-k8s created
configmap/grafana-config created
configmap/grafana-cilium-dashboard created
configmap/grafana-cilium-operator-dashboard created
configmap/grafana-hubble-dashboard created
configmap/grafana-hubble-l7-http-metrics-by-workload created
configmap/prometheus created
clusterrole.rbac.authorization.k8s.io/prometheus configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
service/grafana created
service/prometheus created
deployment.apps/grafana created
deployment.apps/prometheus created

This deployment of Prometheus and Grafana will automatically scrape the Cilium and Hubble metrics.

kubectl get pods -n cilium-monitoring

Example output

NAME                          READY   STATUS    RESTARTS   AGE
grafana-c84dc68d5-p8nbp       1/1     Running   0          33s
prometheus-868bb5f59d-46772   1/1     Running   0          32s
kubectl -n cilium-monitoring port-forward service/grafana --address 0.0.0.0 --address :: 3000:3000

Deploy Cilium and Hubble with metrics enabled

Cilium, Hubble, and Cilium Operator do not expose metrics by default. Enabling metrics for these services will open ports 9962, 9965, and 9963 respectively on all nodes of your cluster where these components are running.

The metrics for Cilium, Hubble, and Cilium Operator can all be enabled independently of each other with the following Helm values:

prometheus.enabled=true: Enables metrics for cilium-agent.

operator.prometheus.enabled=true: Enables metrics for cilium-operator.

hubble.metrics.enabled: Enables the provided list of Hubble metrics. For Hubble metrics to work, Hubble itself needs to be enabled with hubble.enabled=true

Setup Helm repository:

helm repo add cilium https://helm.cilium.io/

Deploy Cilium via Helm as follows to enable all metrics:

helm install cilium cilium/cilium --version 1.16.6 \
  --namespace kube-system \
  --reuse-values \
  --set prometheus.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set operator.prometheus.enabled=true \
  --set hubble.enabled=true \
  --set hubble.metrics.enableOpenMetrics=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}"

If cilium is already installed, use helm upgrade instead of helm install

How to access Grafana

Expose the port on your local machine

kubectl -n cilium-monitoring port-forward service/grafana --address 0.0.0.0 --address :: 3000:3000

Access it via your browser: localhost

How to access Prometheus

Expose the port on your local machine

kubectl -n cilium-monitoring port-forward service/prometheus --address 0.0.0.0 --address :: 9090:9090

Access it via your browser: localhost

Refer to Monitoring & Metrics for more details about the individual metrics.

Approach 2. If you have an existing Prometheus and Grafana stack running on your Cluster

  1. Make sure Prometheus is added as a data source in Grafana.
  2. Then import below dashboards in Grafana.

Access Grafana in your browser –> Click on New –> Choose import –> Paste the above jsons one by one –> Click on Load


Conclusion

You have successfully set up Prometheus and Grafana for Cilium.


© 2025 Jatin Sharma. All rights reserved.