Your Kernel just crashed or one of your drive is not working!! What do you do?

Well, this article gives an introduction to some kernel debugging tools for Linux. These tools makes the kernel internals more transparent. These tools help you to trace the kernel execution process and examine its memory and data structures.

The tools discussed here are :

1. Kernel debugger, kdb

2. Kernel GNU debugger, kgdb

3. GNU debugger, gdb

4. JTAG- based debuggers.

Of the mentioned tools, the kdb and kgdb were introduced as patches to the kernel code. The plain debugger gdb doesn’t need the patching process with kernel code. The JTAG (Joint Test Action Group) based debuggers are hardware assisted and powerful tools, but are expensive.

Here I will explain the installation and usage of the kdb tool. The rest of the tools are briefed.

Kernel debugger, kdb

Kdb is a project maintained by the Silicon Graphics. The main advantage of kdb is that you can debug the kernel that you are running on.

To install the kernel debugger:

1. Download and unzip the patches and apply them to your linux source tree.

Of the 2 patches, the ‘common’ patch contains the changes for the generic kernel code, and the other is the

2. You can get the required patches from

ftp://oss.sgi.com/projects/kdb/download/latest

You should choose the patches depending upon the kernel you are compiling. I compiled the latest 2.6.26 kernel. So I chose the following patches :

kdb-v4.4-2.6.26-rc9-common-1

kdb-v4.4-2.6.26-rc9-x86-1

3. Patch them to your kernel tree.

# cd /usr/src/kernels/linux – Path to your untarred linux kernel source

# patch -p1 < kdb-v4.4-2.6.9-rc4-common-1

# patch -p1 < kdb-v4.4-2.6.9-rc4-i386-1

If your patches are not successful you will get files with extensions .rej in the directory.

3. If successful, recompile the new kernel with the patches enabled using xconfig/menuconfig.

While compiling make sure the CONFIG_KDB option is enabled. You can also enable CONFIG_FRAME_POINTER for better debugging. But this uses an extra register and slightly slower kernel code. These options comes under ‘Kernel Hacking’ section.

The KDB commands to be executed during the initialization process can be defined in a plain text file called kdb_cmds, which exists in the kdb directory of the linux source tree. This file can also be used to define environment variables for setting the display and print options.

For me file was in this path : /usr/src/kernels/linux/kdb

The flags passed to kdb are of three

These flags can be passed at boot time to ensure that the KDB is On/Off. The Early flag helps to troubleshoot the problems during boot time.

KDB commands

KDB allows memory and register modification, applying breakpoints, and stack tracing.

Memory display and modification

Commonly used commands are : md, mdr, mm, and mmW.

1. The md command takes an address/symbol and a line count and displays memory starting at the address for line-count number of lines.

2. The mdr command takes an address/symbol and a byte count and displays the raw contents of memory starting at the specified address for byte-count number of bytes.

3. The mm command modifies memory contents.

4. The mmW command changes W bytes starting at the address.

Register display and modification

Commonly used commands are : rd, rm, and ef.

1. The rd command (without any arguments) displays the contents of the processor registers.

2. The rm command modifies the contents of a register.

3. The ef command takes an address as an argument and displays an exception frame at the specified address.

Stack tracing

The main stack tracing commands are bt, btp, btc, and bta.

1. The bt command attempts to provide information on the stack for the current thread.

2. The btp command takes a process ID as an argument and does a stack traceback for that particular

3. The btc command does a stack traceback for the running process on each live CPU.

4. The bta command does a traceback for all processes in a particular state. Without any argument, it does a traceback for all processes. Optionally, various arguments can be passed to this command. The processes in a particular state will be processed depending on the argument. The options and the corresponding states are as follows:

* D: Uninterruptible state

* R: Running

* S: Interruptible sleep

* T: Traced or stopped

* Z: Zombie

* U: Unrunnable

Breakpoints

The commonly used breakpoint commands are bp, bc, bd, be, and bl.

1. The bp command takes an address/symbol as an argument and applies a breakpoint at the address. It is similar to the bpa command except that it forces the use of a hardware register.

2. The bd command disables a particular breakpoint.

3. The be command enables a breakpoint. The argument to this command is also the breakpoint number.

4. The bl command lists the current set of breakpoints. It includes both the enabled and the disabled breakpoints.

5. The bc command removes a breakpoint from the breakpoint table. It takes either a specific breakpoint number as an argument or it takes *, in which case it will remove all breakpoints.

Kernel GNU debugger, kgdb

The kgdb developed initially as a patch is now included in the official 2.6.26 kernel. This source level debugging tool is much easier to use. It requires two machines to be connected via a serial connection (a RS-232 interface using null modem/a UDP/IP networking protocol).

The gdb is used along with kgdb to trace through kernel code. The gdb runs on the host machine and the kgdb patched kernel runs on the target machine.

GNU debugger, gdb

This is also a powerful stand alone tool used for debugging. But with this tool you can’t set the breakpoint or trace the kernel code. Instead you can use gdb to debug the core files which are dumped when an error occurs.

It leads us with clues on how to troubleshoot the problem.

JTAG- based debuggers

JTAG debuggers use hardware assistance to debug code. This needs a monitor and a front end API to debug the code. It can be used to debug the BIOS, the bootloader, real mode Linux kernel initialization code, and the protected mode kernel.

You can start the debugging mode via command line parameters that you pass to the kernel while booting up or using hardware or software break points. A break point is referred to as the point where the debugger takes in charge when the control is transferred at a point of execution.

If the instruction stops on flash memory and if the corresponding instruction cannot be replaced by the debugger you can use hardware breakpoint. A hardware breakpoint needs processor support. A software breakpoint can be set either using debugger commands or by inserting them into your code. Besides debugging, the second purpose of the JTAG interface is allowing device programmer hardware to transfer data into internal non-volatile device memory.

Conclusion:

After reading through you might be thinking, which one is the best debugger?

The commonly used ones are kdb and kgdb. These tools are aimed at different working environments. The decision depends upon mainly the design.

The kdb home page says like, “This debugger is part of the Linux kernel and provides a means of examining kernel memory and data structures while the system is operational. Additional commands may be easily added to format and display essential system data structures given an identifier or address of the data structure.”

The kgdb home page says like, “The kgdb is a source level debugger for Linux kernel. It is used along with gdb to debug Linux kernel.”

So try the tool you find suitable for your environment, then read more and hack your kernel… 😉

References:

http://oss.sgi.com/projects/kdb/

http://sourceware.org/gdb/

---

About the author:

Abhilash S V has been working with Bobcares for one and half years. He specializes on Linux server administration, a variety of control panels and is an expert in kernel level Linux administration.

 

---