cfs_notes.md

cpu requests	cpu limits	cfs_quota	cfs_period
1	1	100,000 (us)	100,000 (us)
1	2	200,000 (us)	100,000 (us)
0.5 (500m)	0.5	50,000 (us)	100,000 (us)
0.5	NONE	-1	100,000 (us)

Quota and period are found in /sys/fs/cgroup/cpu/cpu.cfs_quota_us and /sys/fs/cgroup/cpu/cpu.cfs_period_us respectively.

Basically, quota is set by the limits. The formula is:

• limits * 100000 (e.g. 0.4 cpu would be 0.4 * 100000 = 40000 microseconds, i.e. 40ms, i.e. of every 100ms period the app can only use 40ms)
• or if using m then limits * 100

You can also view throttling metrics in cpu.stat. Inside cpu.stat you’ll find:

• nr_periods – number of periods that any thread in the cgroup was runnable
• nr_throttled – number of runnable periods in which the application used its entire quota and was throttled
• throttled_time – sum total amount of time (nanoseconds) individual threads within the cgroup were throttled

Get current “throttled percentage”

cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat  | awk '{print $2}' | head -n2 | paste -s -d'/' | bc -l | xargs echo 1/ | bc -l

Example of pod running on guaranteed QoS, with

root@some-name:/data# cat /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us 
100000
root@some-name:/data# cat /sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us 
100000
root@some-name:/data# cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat 
nr_periods 8006
nr_throttled 675
throttled_time 1921546352
root@some-name:/data# cat /sys/fs/cgroup/cpuset/cpuset.cpu_exclusive 
0
root@some-name:/data# cat /sys/fs/cgroup/cpuset/cpuset.cpus 
16

This pod was run with 1 cpu limit, 1 cpu request.

When running with limit 2:

$ cat /sys/fs/cgroup/cpuset/cpuset.cpus 
0-15,17-31

i.e.
11111111
11111110
11111111
11111111

i.e.
fffeffff

For this to work in k8s, however, you need to run the kubelet with --cpu-manager-policy=static.

Example of /proc/57/status:

Cpus_allowed:   fffeffff
Cpus_allowed_list:      0-15,17-31

More info

https://www.kernel.org/doc/html/latest/scheduler/sched-bwc.html