How to make kernel debugging easy?

When we analyze the kernel, there are many times when we want to know the behavior of kernel code. For example, who called the function, what value the variable has, what happens to the call stack, etc.

So I’m going to write down a few things and how to use them.

printk

When we first learn a programming language, we create a program that printing Hello world.

The C language uses printf and the Java language uses System.out.println. These are called STL(Standard Template Library) output. It’s the most basic part and the most powerful. In the kernel, you can print through a function called printk. You can find out more by entering the link.

Integer types

Below is a table of the type and format. C language developer would have seen it often. The kernel’s printf does not support %n. For obvious reasons, floating point formats (%e,%f, %g, %a) are also not recognized. Use of any unsupported specifier or length qualifier results in a WARN and early return from vsnprintf.

printk("test: sector number/total blocks: %llu/%llu\n",
        (unsigned long long)sector, (unsigned long long)blockcount);

If variable is of Type	use printk format specifier
int	%d or %x
unsigned int	%u or %x
long	%ld or %lx
unsigned long	%ld or %lx
long long	%lld or %llx
unsigned long long	%llu or %llx
size_t	%zu or %zx
ssize_t	%zd or %zx
s32	%d or %x
u32	%u or %x
s64	%lld or %llx
u64	%llu or %llx

However, it has become a stronger format than this. You can print pointer Address, functions and symbol/function pointer etc. Debugging is much easier if you know this format.

Pointer Types / Unmodified Addresses

Pointers printed without a specifier extension (i.e unadorned %p) are hashed to give a unique identifier without leaking kernel addresses to user space. On 64 bit machines the first 32 bits are zeroed. If you really want the address use %px.

Please consider whether or not you are leaking sensitive information about the Kernel layout in memory before printing pointers with %px. %px is functionally equivalent to %lx. %px is preferred to %lx because it is more uniquely grep’able. If, in the future, we need to modify the way the Kernel handles printing pointers it will be nice to be able to find the call sites.

%p	abcdef12 or 00000000abcdef12
%px	01234567 or 0123456789abcdef

Symbols/Function Pointers

%pf	versatile_init
%pF	versatile_init+0x0/0x110
%ps	versatile_init
%pS	versatile_init+0x0/0x110
%pSR	versatile_init+0x9/0x110
        (with __builtin_extract_return_addr() translation)
%pB	prev_fn_of_versatile_init+0x88/0x88

The F and f specifiers are for printing function pointers, for example, f->func, &gettimeofday. They have the same result as S and s specifiers. But they do an extra conversion on ia64, ppc64 and parisc64 architectures where the function pointers are actually function descriptors.

The S and s specifiers can be used for printing symbols from direct addresses, for example, __builtin_return_address(0), (void *)regs->ip. They result in the symbol name with (S) or without (s) offsets. If KALLSYMS are disabled then the symbol address is printed instead.

The B specifier results in the symbol name with offsets and should be used when printing stack backtraces. The specifier takes into consideration the effect of compiler optimisations which may occur when tail-calls are used and marked with the noreturn GCC attribute.

printk("Going to call: %pF\n", gettimeofday);
printk("Going to call: %pF\n", p->func);

// #define _RET_IP_  (unsigned long)__builtin_return_address(0)
printk("%s: called from %pS\n", __func__, (void *)_RET_IP_);
printk("%s: called from %pS\n", __func__,
                (void *)__builtin_return_address(0));

printk("Faulted at %pS\n", (void *)regs->ip);
printk(" %s%pB\n", (reliable ? "" : "? "), (void *)*stack);

Kernel Pointers

%pK	01234567 or 0123456789abcdef

For printing kernel pointers which should be hidden from unprivileged users. The behaviour of %pK depends on the kptr_restrict sysctl - see Documentation/sysctl/kernel.txt for more details.

Struct Resources

%pr	[mem 0x60000000-0x6fffffff flags 0x2200] or
	[mem 0x0000000060000000-0x000000006fffffff flags 0x2200]
%pR	[mem 0x60000000-0x6fffffff pref] or
	[mem 0x0000000060000000-0x000000006fffffff pref]

For printing struct resources. The R and r specifiers result in a printed resource with (R) or without (r) a decoded flags member. Passed by reference.

dump_stack

In addition to printk, the kernel provides the ability to show kernel behavior through kernel logs. In other words, if you call the dump_stack() function supported by the kernel, you can view the call-stack as a kernel log. Using the dump_stack() function is simple. Just add the dump_stack() function to the code you want to see the call stack in the kernel log.

To call the dump_stack() function, you need to add the “linux/kernel.h” header file at the top of the code as follows:
#include <linux/kernel.h>

Let’s look at the declaration section of the dump_stack() function.

/**
 * dump_stack - dump the current task information and its stack trace
 *
 * Architectures can override this implementation by implementing its own
 */
asmlinkage __visible void dump_stack(void);

Both the argument and return value types are void. Just add the dump_stack() function anywhere in the kernel source code.

CPU: 1 PID: 1022 Comm: cat Tainted: G         C        4.19.127-v7+ #2
Hardware name: BCM2835
[<80112018>] (unwind_backtrace) from [<8010d364>] (show_stack+0x20/0x24)
[<8010d364>] (show_stack) from [<80851ab4>] (dump_stack+0xd8/0x11c)
[<80851ab4>] (dump_stack) from [<80719280>] (jhs_debug_read+0x18/0x20)
[<80719280>] (jhs_debug_read) from [<803484a8>] (proc_reg_read+0x70/0x98)
[<803484a8>] (proc_reg_read) from [<802d0628>] (__vfs_read+0x48/0x16c)
[<802d0628>] (__vfs_read) from [<802d07e8>] (vfs_read+0x9c/0x164)
[<802d07e8>] (vfs_read) from [<802d0e38>] (ksys_read+0x74/0xe8)
[<802d0e38>] (ksys_read) from [<802d0ec4>] (sys_read+0x18/0x1c)
[<802d0ec4>] (sys_read) from [<80101000>] (ret_fast_syscall+0x0/0x28)

panic

If there is a problem with the kernel, the kernel calls the panic function. And it prints various information such as call-stack and register, and prints reboot or panic message on the screen.

To call the panic() function, you need to add the “linux/kernel.h” header file at the top of the code as follows:
#include <linux/kernel.h>

Let’s look at the declaration section of the panic() function.

/**
 *	panic - halt the system
 *	@fmt: The text string to print
 *
 *	Display a message, then perform cleanups.
 *
 *	This function never returns.
 */
void panic(const char *fmt, ...)

Because kernel internal operation are vast, it can be difficult to see the behavior of the code I added. In that case, put panic() after the code you want to debug, panic the device. And extract the RAM dump or log for analysis.

The annoying thing is that you have to build it with a panic code and work carefully.

JJungs-lee