prometheus 101

prometheus101.md

Links

• https://prometheus.io/docs/instrumenting/exporters/
• https://grafana.com/oss/prometheus/exporters/
• https://grafana.com/grafana/dashboards/
• https://www.kubetail.com/

Avoid scenarios where a particular node is overloaded while another node in the cluster is underutilized.
kubectl top shows usage, not allocation. Allocation is what causes the insufficient CPU problem

Some interesting prometheus-alerts: https://samber.github.io/awesome-prometheus-alerts/rules.html#kubernetes

# How many NEW restarts happened in the last 15 minutes: time series of all the values in the last 30 minutes
# Might fire multiple times per pod (once per container)
increase(kube_pod_container_status_restarts_total{namespace=~"ehealth-public|ehealth-private|ehealth-audit"}[15m]) > 1

# Total pod restarts in the last 15 minutes: Aggregates restarts per pod (summing all containers inside that pod)
sum(increase(kube_pod_container_status_restarts_total{namespace=~"ehealth-public|ehealth-private|ehealth-audit"}[15m])) by (namespace,pod)

# active memory a container is currently using (in bytes) converted to MB
# use namespace in summation If two pods with the same name exist in different namespaces, their memory usage will be summed together 
sum by (pod, namespace) (container_memory_working_set_bytes{namespace=~"ehealth-public|ehealth-private|ehealth-audit", container!="", container!="POD"}) / 1024 / 1024

# PodOOM Killed: new OOMKill events in the last 15 minutes
# Increase needs at least 2 samples
increase(kube_pod_container_status_terminated_reason{reason="OOMKilled"}[15m]) > 0
sum by (pod, namespace) 

(increase(kube_pod_container_status_terminated_reason{reason="OOMKilled"}[15m])) or vector(0)


# PodStuckAfterOOMKill: Pods that were OOMKilled and haven't recovered to Running state within the last 6 hours
(sum(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}) by (pod,namespace) > 0 and on(pod) kube_pod_status_phase{phase="Running"} == 0 and on(pod) kube_pod_created{} > (time() - 21600)) or vector(0)
0

Start from here: kubectl get po -A|grep -E "ImagePullBackOff|CrashLoopBackOff|Terminating|Init"