Kubernetes Watch Workshop

WatchWorkshop.md

Kubernetes Watch

Kubernetes Watch reports changes on the resource defined by the URL and is configured through the query string.

• It is the mechanism that backs Pepr Watch and Reconcile.
• It is how Kubernetes Controllers and Informers track changes to resources
• It can return several content-types like JSON, YAML, protobuf, CBOR and probably more

URL Construction

Core resources use /api and omit group, others use /apis

/apis/GROUP/VERSION/namespaces/NAMESPACE/*

Examples:

/api/v1/namespaces
/api/v1/pods
/api/v1/namespaces/my-namespace/pods
/apis/apps/v1/deployments
/apis/apps/v1/namespaces/my-namespace/deployments
/apis/apps/v1/namespaces/my-namespace/deployments/my-deployment

Demo:

Proxy kube-apiserver locally

kubectl proxy&

Create a pod

kubectl run n --image=nginx 

Get a list of pods across all namespaces kubectl get po -A

curl "localhost:8001/api/v1/pods" | jq '.items[].metadata.name' 

Get a list of pods from default namespace

curl "localhost:8001/api/v1/namespaces/default/pods" | jq '.items[].metadata.name' 

Get a specific pod

curl "localhost:8001/api/v1/namespaces/default/pods/n" | jq '{name: .metadata.name, resourceVersion: .metadata.resourceVersion}'

Query Params

Query params are how you configure how you get resources. The ones that I am aware of are:

• watch bool - Sends back back events (KFC)
• resourceVersion int - A pod at a given state in time, resourceVersion increments up (KFC)
• allowWatchBookmarks bool - Send bookmark events indicating whether new events have happened (not used in KFC)
• labelSelector map[string]string - Obvious (not used in KFC)
• fieldSelector map[string]string - Selector for given fields (used only for metadata.name in KFC)

ResourceVersion only gets re-assigned in KFC during a relist event.

Demo:

Start a watch all pods since from no particular resource version

curl -N --no-buffer "http://localhost:8001/api/v1/pods?watch=true" | jq '{name: .object.metadata.name, type: .type, resourceVersion: .object.metadata.resourceVersion}'

Label the pod to trigger an MODIFIED event

k label po/n color=red

  "type": "MODIFIED",

Use the fieldSelector to watch pods where name is n

curl -N --no-buffer "http://localhost:8001/api/v1/pods?watch=true&fieldSelector=metadata.name=n" | jq '{name: .object.metadata.name, type: .type, resourceVersion: .object.metadata.resourceVersion}'

Trigger a new MODIFIED event

k label po/n color=blue --overwrite

  "type": "MODIFIED",

Create a new pod with name n in a new namespace

k run n -n kube-public --image=nginx

  "type": "MODIFIED",

... 😮 says modified but it is a new pod

We need to be more specific, lets re-do our watch and include namespace in the fieldSelector

curl -N --no-buffer "http://localhost:8001/api/v1/pods?watch=true&fieldSelector=metadata.name=n,metadata.namespace=kube-public" | jq '{name: .object.metadata.name, type: .type, resourceVersion: .object.metadata.resourceVersion}'

  "type": "ADDED",

What else can we do

subresources

Get logs from n

curl "http://localhost:8001/api/v1/namespaces/default/pods/n/log" 

Pull node metrics ( I am using k3d with metric server)

 # find the name of a node
curl -N --no-buffer "http://localhost:8001/apis/metrics.k8s.io/v1beta1/nodes/$(kubectl get nodes -o json | jq -r '.items[0].metadata.name')" | jq

Summary

Pepr's watch/reconcile program the Watch class in KFC to watch a given resource. The watch starts watching from a given resource version and acting upon events when it dispatches the info back to the Pepr callbacks.