Created
March 19, 2014 03:36
Revisions
-
dhedlund created this gist
Mar 19, 2014 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,342 @@ class: center, middle # Dynamic Tracing with DTrace and SystemTap --- # DTrace * Originally developed by Sun Microsystems, now Oracle * Works with most UNIXes. Comes pre-installed on Solaris and Mac OS X * CDDL license, not compatible with the GPL > DTrace has been described as a tool that "allows you to ask arbitrary > questions about what the system is doing, and get answers." --- # SystemTap * Only works with Linux * GPL license * Current project members include Red Hat, IBM, Hitachi, and Oracle. --- # Both... * Provide a similar feature set * Are loaded into the kernel as modules to compiled in * Must be run as root (or with special groups) * Use Kprobes to implant probes and register event handlers(?) --- # Kprobes * A probe is an automated breakpoint that is **implanted dynamically** in executing (kernel-space) modules **without the need to modify their underlying source.** * They are particularly **advocated in production environments** where the use of interactive debuggers is undesirable. * Probe event handlers **run as extensions to the system breakpoint interrupt handler** and are expected to have little or no dependence on system facilities. * Because of this design point, probes are able to be **implanted in the most hostile environments without adversely skewing system performance.** _https://sourceware.org/systemtap/kprobes/_ --- # Hello World ### DTrace ``` perl # hello-world.d BEGIN { trace("hello world\n"); exit(0); } ``` $ dtrace -s hello-world.d ### SystemTap ``` perl # hello-world.stp probe begin { print("hello world\n") exit() } ``` $ stap hello-world.stp --- # Hello World ### DTrace $ dtrace -n 'BEGIN { trace("hello world\n"); exit(0) }' ### SystemTap $ stap -e 'probe begin { print("hello world\n"); exit() }' --- # They're Pretty Similar... ## Probes <table class="comparison"> <thead> <tr> <th>DTrace</th> <th>SystemTap</th> </tr> </thead> <tbody> <tr> <td>BEGIN</td> <td>begin</td> </tr> <tr> <td>END</td> <td>end</td> </tr> <tr> <td>syscall:::entry</td> <td>syscall.\*</td> </tr> <tr> <td>syscall:::return</td> <td>syscall.\*.return</td> </tr> <tr> <td>syscall::read:entry</td> <td>syscall.read</td> </tr> <tr> <td>profile:::tick-10s</td> <td>timer.s(10)</td> </tr> </tbody> </table> --- # They're Pretty Similar... ## Built-in Variables <table class="comparison"> <thead> <tr> <th>DTrace</th> <th>SystemTap</th> </tr> </thead> <tbody> <tr> <td>execname</td> <td>execname()</td> </tr> <tr> <td>uid</td> <td>uid()</td> </tr> <tr> <td>pid</td> <td>pid()</td> </tr> <tr> <td>timestamp</td> <td>gettimeofday_ns()</td> </tr> <tr> <td>arg0..N</td> <td>(custom variable: see `stap -L PROBE`)</td> </tr> <tr> <td>$target</td> <td>target()</td> </tr> </tbody> </table> --- # They're Pretty Similar... ## Functions <table class="comparison"> <thead> <tr> <th>DTrace</th> <th>SystemTap</th> </tr> </thead> <tbody> <tr> <td>stack()</td> <td>print_backtrace()</td> </tr> <tr> <td>quantize()</td> <td>@hist_log()</td> </tr> <tr> <td>lquantize()</td> <td>@hist_linear()</td> </tr> <tr> <td>exit(status)</td> <td>exit()</td> </tr> </tbody> </table> --- # Listing Syscall Probes ### DTrace $ dtrace -ln syscall:::entry ### SystemTap $ stap -l 'syscall.\*' ``` plain syscall.accept syscall.access syscall.acct syscall.add_key syscall.adjtimex syscall.alarm ``` --- class: center, middle # Comparative Examples --- # Return size of read() syscall ### DTrace $ dtrace -n 'syscall::read:return { @bytes = quantize(arg1); }' ### SystemTap $ stap -e 'global bytes; probe syscall.read.return { bytes <<< $return; } probe end { print(@hist_log(bytes)); }' --- # Count syscalls by process ### DTrace $ dtrace -n 'syscall:::entry { @[execname] = count(); }' ### SystemTap $ stap -e 'global ops; probe syscall.\* { ops[execname()] <<< 1; }' --- # New procs w/ name and args ### DTrace $ dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }' ### SystemTap $ stap -e 'probe process.begin { printf("%s\n", cmdline_str()); }' --- # More Examples ### SystemTap * https://sourceware.org/systemtap/examples/ * ```bash $ git clone git://sourceware.org/git/systemtap.git $ cd /systemtap/testsuite/systemtap.examples/ ``` ### Feature Comparison * https://sourceware.org/systemtap/wiki/SystemtapDtraceComparison --- # SystemTap ### Safety * https://sourceware.org/systemtap/tutorial/Analysis.html#SECTION00046000000000000000 ### User-space Tracing The current iteration of SystemTap allows for a multitude of options when probing kernel-space events for a wide range of kernels. However, **SystemTap's ability to probe user-space events is dependent on kernel support (the Utrace mechanism) that is unavailable in many kernels. Thus, only some kernel versions support user-space probing.** At present, the developmental efforts of the SystemTap community are geared towards improving SystemTap's user-space probing capabilities. _https://sourceware.org/systemtap/wiki/utrace/arch/HowTo_ But utrace is dead? http://stackoverflow.com/questions/12134041/is-utrace-project-dead --- # Other Tracing Tools * Linux Trace Toolkit - next generation: http://lttng.org/ * Perf: https://perf.wiki.kernel.org/index.php/Main_Page * OProfile: http://oprofile.sourceforge.net/about/ * Frysk: https://sourceware.org/frysk/ --- # Resources * http://www.postgresql.org/docs/9.2/static/dynamic-trace.html * http://avsej.net/2012/systemtap-and-ruby-20/ * /usr/share/systemtap/tapsets/ --- # DTrace Issues * https://github.com/dtrace4linux/linux/issues/58