From: eLinux.org
Crash Diagnostics
kernel crash debugging technique using "crash" utility.
Introduction
Most frequent issue in debugging kernel programs is lack of information; lack of information in terms of stack trace, kernel logs and crash screen-shots. Crash utility provides solution for this. It provides gdb like interface for debugging.
Prerequisites
vmlinux object file.
This is kernel image with debugging information. By default, we get compressed kernel image with no debug information. For getting, vmlinux object file, you need to install "kernel-debuginfo" rpm. If you want to debug any module, make sure it is compiled with gcc flag '-g'.
Installation
We receive crash utility as rpm.
rpm name is of format: crash-$version.$arch.rpm
Here, $version shows version number and $arch shows system architecture.
For getting crash dump: For getting crash dump, we need to add "crashkernel" option to grub command line. i.e. we can have:
ro root=LABEL=/1 rhgb quiet crashkernel=128M@16M
instead of,
ro root=LABEL=/1 rhgb quiet
Here, parameter "crashkernel=128M@16M" reserves 128MB of physical memory starting at 16MB. This reserved memory is used to preload and run the capture kernel (i.e. to capture crash dump). This command line option ensures that, whenever crash occurs, it stores crash dump at "/var/crash" and they are stored according to date and time.
There are other options to get crash dump like diskdump, netdump, etc.
Running crash Utility: 1. Debugging last kernel panic:
#crash <patht to vmlinux> /var/crash/crash_dump_name
-
Watching current running kernel:
#crash
This will prompt to crash shell. Just as example, following are commands:
crash> help
crash> bt ---> for backtrace
crash> log ---> for dumping current system buffer
For more information on this, you can visit at: http://www.redhatmagazine.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/ http://people.redhat.com/anderson/crash_whitepaper/
