From: eLinux.org
Kernel Instrumentation
Here is a listing of some instrumentation systems for the kernel:
Contents
Existing Instrumentation Systems
TimePegs
Andrew Morton's system for measuring intervals between kernel events:
See http://www.zipworld.com.au/~akpm/linux/timepeg.txt
Patches at:
http://www.zip.com.au/~akpm/linux/index.html#timepegs
Printk Times
Produces printk's with extra time data on them. As of kernel 2.6.11 this is part of the mainline kernel enabled by CONFIG_PRINTK_TIME. Previous versions can add it via a very simple patch. It works for bootup time measurements, or other places where you can just jam in a printk or two.
See Printk Times
Boot Tracer
Starting from 2.6.28 the kernel has this new feature to optimize the boot time. It records the timings of the initcalls. Its aim is to be parsed by the scripts/bootgraph.pl tool to produce graphics about boot inefficiencies, giving a visual representation of the delays during initcalls. Users need to boot with the "initcall_debug" and "printk.time=1" parameters, and run "dmesg | perl scripts/bootgraph.pl > output.svg" to generate the final data.
Kernel Function Instrumentation (KFI)
A system which uses a compiler flag to instrument most of the functions in the kernel. Timing data is recorded at each function entry and exit. The data can be extracted and displayed later with a command-line program.
The kernel portion of this is available in the CELF tree now.
Grep for CONFIG_KFI.
See the page Kernel Function Instrumentation page for some preliminary notes.
FIXTHIS - need to isolate this as a patch.
Linux Trace Toolkit
Kernel Tracer (in IKD patch)
This is part of a general kernel tools package, maintained by Andrea Arcangeli.
See http://www.kernel.org/pub/linux/kernel/people/andrea/ikd/README
The ktrace implementation is in the file kernel/debug/profiler.c It was originally written by Ingo Molnar, Richard Henderson and/or Andrea Arcangeli
It uses the compiler flag -pg to add profiling instrumentation to the kernel.
Function trace in KDB
Last year (Jan 2002) Jim Houston sent a patch to the kernel mailing list which provides support compiler-instrumented function calls.
See http://www.ussg.iu.edu/hypermail/linux/kernel/0201.3/0888.html
ftrace
Ftrace is a simple function tracer which initially came from the -rt patches but was mainlined in 2.6.27. Compiler profiling features are used to insert an instrumentation call (with gcc -pg option) that can be overwritten with a NOP sequence to ensure overhead is minimal with tracing disabled (this is enabled through CONFIG_DYNAMIC_FTRACE). There are a number of tracers in the kernel that use ftrace to trace high level events such as irq enabling/disabling preemption enabling/disabling, scheduler events and branch profiling.
The interface to access ftrace can be found in /debugfs/tracing, and is very extensively documented in Documentation/trace/ftrace.txt.
Steven Rostedt, ftrace's main developer, gave a conference on ftrace at the Ottawa Linux Symposium 2008. The video and slides.
SystemTap / Kprobes
SystemTap is a sophisticated kernel instrumentation tool that can be scripted with its own language to gather information about a running kernel. It uses the Kprobes infrastructure to implement it's tracing.
Notes
Some random thoughts on instrumentation:
- Most instrumentation systems need lots of memory to buffer the data produced
- Some instrumentation systems support filters or triggers to allow for better control over the information saved
- instrumentation systems tend to introduce overhead or otherwise interfere with the thing they are measuring
- instrumentation systems tend to pollute the cache lines for the processor
- There doesn't seem to be a single API to support in-kernel timing instrumentation which is supported on lots of different architectures. This is the main reason for CELF's current project to define an Instrumentation API
