Monitoring Karpenter Using Prometheus & Grafana
I have used
prometheus-community/kube-prometheus-stackforprometheus&grafanadeployment.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring --create-namespace -f prometheus-values.yaml
prometheus-values.yaml
server:
persistentVolume:
enabled: true
storageClass: gp2-standard
size: 10Gi
alertmanager:
persistentVolume:
enabled: true
storageClass: gp2-standard
size: 2Gi
pushgateway:
persistentVolume:
enabled: true
storageClass: gp2-standard
size: 5Gi
kube-state-metrics:
metricLabelsAllowlist:
- pods=[*]
- deployments=[app.kubernetes.io/name,app.kubernetes.io/component,app.kubernetes.io/instance]
prometheus:
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
grafana:
persistence:
enabled: true
storageClassName: gp2-standard
accessModes:
- ReadWriteOnce
size: 5Gi
adminUser: admin
adminPassword: admin
service:
type: ClusterIP
kubectl get po -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 28h
prometheus-grafana-6b6f5b59c-shbcw 3/3 Running 0 71m
prometheus-kube-prometheus-operator-549df9cbc6-vhm6l 1/1 Running 0 71m
prometheus-kube-state-metrics-85f5f89d9-28rdq 1/1 Running 0 28h
prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 28h
prometheus-prometheus-node-exporter-7xz97 1/1 Running 0 46h
prometheus-prometheus-node-exporter-mxhqc 1/1 Running 0 46h
prometheus-prometheus-node-exporter-rsz49 1/1 Running 0 85m
prometheus-prometheus-node-exporter-zksgr 1/1 Running 0 46h
kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 46h
prometheus-grafana ClusterIP 10.100.13.116 <none> 80/TCP 46h
prometheus-kube-prometheus-alertmanager ClusterIP 10.100.8.122 <none> 9093/TCP,8080/TCP 46h
prometheus-kube-prometheus-operator ClusterIP 10.100.54.58 <none> 443/TCP 46h
prometheus-kube-prometheus-prometheus ClusterIP 10.100.118.158 <none> 9090/TCP,8080/TCP 46h
prometheus-kube-state-metrics ClusterIP 10.100.199.121 <none> 8080/TCP 46h
prometheus-operated ClusterIP None <none> 9090/TCP 46h
prometheus-prometheus-node-exporter ClusterIP 10.100.212.62 <none> 9100/TCP 46h
Prometheus & Grafana access
# Port Forwarding Promotheus
kubectl port-forward service/prometheus-kube-prometheus-prometheus 3000:9090 -n monitoring
# # Port Forwarding Grafana
kubectl port-forward service/prometheus-grafana 3000:80 -n monitoring
Monitoring karpenter
Edit karpenter.yaml and add below ServiceMonitor configuration to allow prometheus to scrape karpenter metrics.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: karpenter-monitoring
namespace: karpenter
labels:
helm.sh/chart: karpenter-1.3.3
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.3.3"
app.kubernetes.io/managed-by: Helm
spec:
namespaceSelector:
matchNames:
- karpenter
selector:
matchLabels:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
endpoints:
- honorLabels: true
interval: 30s
path: /metrics
port: http-metrics
Grafana Dashboard
1. Grafana dashboard to display all useful Karpenter metrics 2. Grafana dashboard to display all useful Karpenter metrics