`perf` Explainer

perf.md

perf Introduction

Using the perf tool centers around the following concepts:

1. events - these are hardware interrupts, hardware counters, or software events can be tracked
2. targets - while these CPUs, processes, or threads are executing, events will be tracked (tracking is disabled otherwise)
3. filters - these are predicates on event contexts used to control which events are tracked in which contexts

As I understand it, perf can operate in two main modes (as indicated by the perf stat and perf record subcommands):

1. perf stat - count events that occur as target executes, produce a set of global event counts, e.g., count the total number of CPU cycles that elapsed while a process was executing.

   This command introduces very low monitoring overhead, but does not track event contexts and thus does not support filters.

2. perf record - count events that occur as target executes in a particular execution context (if the context satisfies the filter), produce a set of per-context event counts. Note that an execution context includes things like:

   • the name of the currently executing function (which will be unknown unless debugging symbols are present)
   • which object file contains the currently executing function

   Note that perf record cannot record and associate literally every event occurrence as they happen too frequently. Instead, it will sample events (with a user-specified frequency) and only record sampled events.

   Also note that, by default, perf record will track the event cycles which occurs once every CPU cycle. The effect of this is that it will estimate (by sampling) how many CPU cycles were spent executing each function that your target program/thread runs (or that runs on your target CPU).

For our purposes, we generally want perf record mode, as we want to know what context a particular event was associated with.

perf record

Here is a brief explainer of the perf record command (there are many more options that are not covered here):

perf record [-e <event|{event...}>] [--filter=<filter>] \
            [-F <freq>] [-c <count>]                    \
            [-g] [--call-graph fp|lbr|dwarf]            \
            [-o|--output <path>]                        \
            [-a] [-C|--cpu CPU...] [-p|--pid PID...]  [-t|--tid TID...] [--] [command]

where:

• Line 1 options specify which events to track:

  • -e specifies the single event or brace-enclosed, comma-separated {event...} list to track

    If absent, the default event is cycles, i.e., this event occurs every CPU cycle.

  • --filter <filter> specifies the event/context filter predicate; if absent, all tracked events are counted

• Line 2 options specify how often to track events (these are mutually exclusive):

  • -F specifies a desired frequency to track events
  • -c specifies that 1 out of every count events will be tracked

  If -F and -c are absent, I believe the default is -F 1000, but I'm not sure about this.

• Line 3 options specify call graph tracking (this extends the execution context to include the stack trace and not just the currently executing function):

  • -g enables call graph tracking
  • --call-graph <unwind-mode> specifies how perf attempts to unwind call stacks:
    • fp - (default) efficient but based on frame pointers which doesn't work on code compiled with option -fomit-frame-pointer
    • lbr - an accurate and efficient mode that is only supported by modern CPUs
    • dwarf - accurate but slow, based on DWARF call frame information

• Line 4 options specify where/how to record results:

  • -o - specifies path to output file where results are saved; if absent, default is perf.data

• Line 5 options specify targets (in order of granularity):

  • -a - specifies all CPUs as targets (i.e., the entire system is profiled)
  • -C - specifies the named CPUs as targets
  • -p - specifies the named running processes as targets
  • -t - specifies the named threads as targets
  • command - executes command as a new process, specifies that process as a target, and automatically terminates when command process terminates

  Note: if command is not specified, perf will not terminate until the user presses Ctrl+C.

  To make perf record data for a fixed duration of time for a non-command target (for example, the entire system), you can use the following pattern:

  perf record -a sleep 5
  

  Since -a was passed, the entire system will be profiled; however, since <command> was passed, the profiling will terminate when the <command> process terminates (which, for sleep 5, will occur after 5 seconds). This also works for -C, -p, and -t

  If instead you did:

  perf record -a
  

  It will only terminate when Ctrl+C is pressed.

perf report

This tool consumes perf record data and visualizes it. The main options are listed below (see the man page or tutorial for more options):

perf report [-i|--input <path>] [--stdio|--tui] [-g]

• -i specifies an input file; if absent, default is perf.data
• --stdio generates a report file on stdout as text while --tui presents data using a terminal interface
• -g tells the tool to visualize call graph hierarhcies

Additional Notes

1. Since the perf command talks to low-level hardware counters (and can monitor the entire system), it typically must be run with administrator privileges, so use sudo if necessary.

2. Note that perf is actually a part of the Linux kernel. This means, typically, you must install a kernel specific version. To do this on Ubuntu, one does:

   apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r`
   

3. Data that is produced by perf record can go stale over time. I see two methods to deal with this:

   1. Generate a report using perf report ASAP after running perf record --- this trick avoids the staleness issue and is easy to do.

   2. However, there may be another possible approach that uses perf archive --- essentially, this tool scans your perf.data file, gathers debugging symbols for all of the libraries that it references, and dumps them into compressed archive. This archive can then be unpacked in the perf build-id directory, which by default, is $HOME/.debug.

4. Running perf record in a Docker container requires building the Docker container with the --privileged flag which gives the container root-like permissions on the host system --- this means such containers should be run only when performing profiling.

Additional Resources

• The perf wiki and its tutorial
• Bredan Gregg's perf Examples
• https://github.com/bpradipt/perf-container - information about running perf in a container

Of these resources, so far, I found the tutorial most helpful --- but the examples page is a nice quick reference if you know what you're doing.

perfsum.py

#!/usr/bin/python3
"""
This script takes multiple ASCII reports generated by perf record + report and aggregates the per-function percentages across runs.
It tries to minimize the name of really long C++ template functions to simplify the aggregated report.
It also collapses function name distinctions based on:
- class method object refness/constness specifiers
- geneated plt tables
"""

from pathlib import Path
from collections import defaultdict
import sys

def read_file(filepath, maxobjlen=0):
  total = 0
  data = defaultdict(lambda: 0)
  for line in Path(filepath).read_text().splitlines():
      idx = line.find("%")
      if idx <= 0: continue
      chunks  = line.split()
      percent = float(chunks[0][:-1])
      obj     = chunks[2]
      func    = ' '.join(chunks[4:]).strip()
      data[(obj,func)] += percent
      maxobjlen = max(len(obj), maxobjlen)
  return data, maxobjlen

def merge_files(datalist):
  cnt = len(datalist)
  if cnt == 0: return dict()
  total = datalist.pop()

  for data in datalist:
    for k,v in data.items():
      total[k] += v

  for (k1,k2),v in total.items():
    total[(k1,k2)] = v / cnt

  return total

def countMatchers(s, ind, opener, closer, stack=1):
    assert ind >= 0
    while ind > 0 and stack > 0:
        if s[ind] == opener:
            stack += 1
        elif s[ind] == closer:
            stack -= 1
            if stack == 0: break
        ind -= 1
    return ind if stack == 0 else None

def truncate(s, cutoff):
    if len(s) <= cutoff: return s
    # remove suffixes
    s = s.removesuffix(' &')
    s = s.removesuffix(' const')
    s = s.removesuffix('@plt')
    # remove arg types
    if s.endswith(')') and (match := countMatchers(s, len(s)-2, ')', '(')):
        s = s[:match+1] + '...' + ')'
    if len(s) <= cutoff: return s
    # remove last set of template args
    if (lastOpener := s.rfind('>')) > 0 and (match := countMatchers(s, lastOpener-1, '>', '<')):
        s = s[:match+1] + '...' + s[lastOpener:]
    if len(s) <= cutoff: return s
    # truncate name
    over = (len(s) + 1 - cutoff) // 2
    mid = len(s) // 2
    fstPart, sndPart = mid - over, mid + over
    trunc = s[:fstPart] + '...' + s[sndPart:]
    return trunc

def main():
  threshold = float(sys.argv[1])
  cutoff = int(sys.argv[2]) + 3
  filepaths = sys.stdin.read().split()
  datalist = []
  maxobjlen = 0
  for filepath in filepaths:
    data, maxobjlen = read_file(filepath, maxobjlen)
    datalist.append(data)
  data = merge_files(datalist)

  sum = 0.0
  for (k1,k2),v in dict(sorted(data.items(), key=lambda item: item[1], reverse=True)).items():
      if v > threshold:
        k1 = f"{k1:<{maxobjlen}}".replace(',', ';')
        k2 = truncate(k2, cutoff - 3).replace(',', ';')
        k2 = f"{k2:<{cutoff}}"
        print(f"{k1}, {k2}, {v:.3f}")
        sum += v
  print(f"Total percent: {sum:.2f}")

if __name__ == "__main__":
    main()