For some recent depravity I "needed" to build a kernel with ASan for Ubuntu Server. While normally this would've been a relatively straightforward operation, in this case there was a further complication — the server is a Raspberry Pi.
Since documentation on building an Ubuntu kernel for the Raspberry Pi is virtually nonexistent, I'm documenting the process here for any poor souls who might need this in the future.
- Getting the source
- Preparing for the build
- Configuring the kernel
- Patching the bootloader
- Checking that it works
- Appendix: Ubuntu Server RPi boot process
- A Raspberry Pi 4, 4GB RAM, with Ubuntu Server 20.04.4.
- A build (virtual) machine with Ubuntu.
- Because you really don't want to build on the Pi directly.
That's what I worked with. The described methods will probably apply to other kernel versions and/or Raspberry Pi revisions, with or without modifications. Note that the RAM size might also be a factor. ASan requires quite a lot of memory, so systems with little RAM might not be usable with it.
Plea: Make a backup of your system before doing anything. Please.
Getting the source
On the Raspberry Pi, run:
apt-get --download-only source linux-image-$(uname -r)
This will download the source package for the currently-running kernel
to the working directory. The output will contain at least a
.dsc file and a tarball.
If this fails, you might need to uncomment the
deb-src lines in
Transfer the source package files to your build machine.
Preparing for the build
Extract the source package:
dpkg-source -x something-something.dsc
Install build tools:
sudo apt install \ build-essential \ crossbuild-essential-arm64 \ libncurses-dev \ gawk \ flex \ bison \ openssl \ dkms \ libelf-dev \ libudev-dev \ libpci-dev \ libiberty-dev \ autoconf \ git \ bc \ libssl-dev \ make \ libc6-dev
Note: This list may be incomplete. That's what I installed, but the machine wasn't clean to begin with, so perhaps some necessary packages were already installed. Let me know if something is missing here.
Configuring the kernel
In the source directory run:
export $(dpkg-architecture -aarm64) export CROSS_COMPILE=aarch64-linux-gnu-
(You can safely ignore the
Edit the first line in
debian.raspi/changelog and add a custom version tag.
For instance, if that line reads:
linux-raspi (5.4.0-1066.76) focal; urgency=medium
Change it to something like:
linux-raspi (5.4.0-1066.76+kasan) focal; urgency=medium
This will ensure that the kernel we're building is "newer" than the one currently installed.
fakeroot debian/rules clean fakeroot debian/rules editconfigs
The last command will prompt you to edit the kernel configuration. Answer "no" to
the first and "yes" to the second, since we only want the
Do you want to edit config: armhf/config.flavour.raspi? [Y/n] n Do you want to edit config: arm64/config.flavour.raspi? [Y/n] Y
Time to enable KASAN. Go to
Kernel hacking -> Memory Debugging, and enable
KASAN in Generic mode with Inline instrumentation. Also enable
the "Module for testing KASAN for bug detection".
Here you may also want to enable page owner tracking. See here for more information.
Go back to the
Kernel hacking menu and enable stack backtrace support:
Now exit and save the configuration when prompted.
In the source directory, run:
fakeroot debian/rules binary skipdbg=false
This will build the kernel and additional debug binaries. If you don't need them,
After the build finishes, the directory above the source directory will have
.deb packages, looking something like this:
linux-headers-5.4.0-1066-raspi_5.4.0-1066.76+kasan_arm64.deb linux-image-5.4.0-1066-raspi_5.4.0-1066.76+kasan_arm64.deb linux-modules-5.4.0-1066-raspi_5.4.0-1066.76+kasan_arm64.deb linux-raspi-headers-5.4.0-1066_5.4.0-1066.76+kasan_arm64.deb
And if you built debug binaries you'll also have
Copy those over to the server and install with
dpkg -i package.deb.
If you have a newer kernel, you may also want to configure KASAN using the kernel
command-line. See here for more info, and use
cmdline parameter in the Pi's
config.txt to change the command-line.
Patching the bootloader
Now comes the tricky part. The kernel we just built is too big for the bootloader to handle, so we'll need to patch it.
First, get this script:
Determine the size of the kernel you just built:
sudo file -L /boot/vmlinuz
Which should output something like:
/boot/vmlinuz: gzip compressed data, max compression, from Unix, original size modulo 2^32 55808512
./uboot_patch.py \ -i /boot/firmware/uboot_rpi_4.bin -o ./uboot_rpi_4_patched.bin
This will patch the bootloader to support kernels up to 63.5MB in size. If your kernel
is somehow larger, run the script with
-s and specify a larger size.
Copy the patched bootloader to
/boot/firmware, and edit
config.txt to point
to it instead of the original
Checking that it works
Moment of truth.
If everything went according to plan the system should reboot (it will take a little
longer than usual), and in the
dmesg output you'll see:
kasan: KernelAddressSanitizer initialized
If not... I hope you made that backup 😎.
Now to test that KASAN actually works:
sudo insmod /lib/modules/$(uname -r)/kernel/lib/test_kasan.ko
Which should dump something like this to the screen (not over SSH):
================================================================== BUG: KASAN: slab-out-of-bounds in kmalloc_oob_right+0x94/0xb0 [test_kasan] Write of size 1 at addr ffff77250ef6c77b by task insmod/3529 CPU: 2 PID: 3529 Comm: insmod Tainted: G C E 5.4.0-1066-raspi #76+kasan Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT) Call trace: dump_backtrace+0x0/0x2d8 show_stack+0x28/0x38 dump_stack+0x128/0x194 print_address_description.isra.0+0x74/0x354 __kasan_report+0x188/0x1d8 kasan_report+0xc/0x18 __asan_report_store1_noabort+0x1c/0x28 kmalloc_oob_right+0x94/0xb0 [test_kasan] kmalloc_tests_init+0x18/0xd9c [test_kasan] do_one_initcall+0xbc/0x6e8 do_init_module+0x154/0x5c0 load_module+0x2644/0x3468 __do_sys_finit_module+0x120/0x1a8 __arm64_sys_finit_module+0x74/0xa8 el0_svc_common.constprop.0+0x11c/0x488 el0_svc_handler+0x50/0xd0 el0_svc+0x10/0x200 Allocated by task 3529: save_stack+0x24/0xb0 __kasan_kmalloc.isra.0+0xc0/0xe0 kasan_kmalloc+0xc/0x18 kmem_cache_alloc_trace+0x1ac/0x350 kmalloc_oob_right+0x54/0xb0 [test_kasan] kmalloc_tests_init+0x18/0xd9c [test_kasan] do_one_initcall+0xbc/0x6e8 do_init_module+0x154/0x5c0 load_module+0x2644/0x3468 __do_sys_finit_module+0x120/0x1a8 __arm64_sys_finit_module+0x74/0xa8 el0_svc_common.constprop.0+0x11c/0x488 el0_svc_handler+0x50/0xd0 el0_svc+0x10/0x200 Freed by task 2798: save_stack+0x24/0xb0 __kasan_slab_free+0x108/0x180 kasan_slab_free+0x10/0x18 kfree+0xb0/0x390 security_cred_free+0xbc/0x130 put_cred_rcu+0x68/0x2a0 rcu_do_batch+0x2c4/0x610 rcu_core+0x2f4/0xb30 rcu_core_si+0x18/0x20 __do_softirq+0x324/0xc28 The buggy address belongs to the object at ffff77250ef6c700 which belongs to the cache kmalloc-128 of size 128 The buggy address is located 123 bytes inside of 128-byte region [ffff77250ef6c700, ffff77250ef6c780) The buggy address belongs to the page: page:ffffffdc941bdb00 refcount:1 mapcount:0 mapping:ffff772557403c00 index:0xffff77250ef6c200 flags: 0x4000000000000200(slab) raw: 4000000000000200 dead000000000100 dead000000000122 ffff772557403c00 raw: ffff77250ef6c200 000000008010000d 00000001ffffffff 0000000000000000 page dumped because: kasan: bad access detected Memory state around the buggy address: ffff77250ef6c600: 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc fc ffff77250ef6c680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc >ffff77250ef6c700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 ^ ffff77250ef6c780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ffff77250ef6c800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ==================================================================
If after this the system is still somehow running — reboot.
Enjoy your sanitized kernel!
Appendix: Ubuntu Server RPi boot process
The boot partition on the SD card of a Raspberry Pi Ubuntu Server contains a handy
README file explaining the boot process. Briefly:
bootcode.bin- the second stage bootloader loaded by all Pi's with the exception of the Pi 4 (where this is replaced by flash memory).
config.txt- the first configuration file read by the boot process.
start*.elf- the third stage bootloader, which handles device-tree modification and which loads...
uboot*.bin- various U-Boot binaries for different Pi platforms; these are launched as the "kernel" by
boot.scr- the boot script executed by
uboot*.binwhich in turn loads...
vmlinuz- the Linux kernel, executed by
initrd.img- the initramfs, executed by
Of particular interest is the
boot.scr script. It relies on the values of two
environment variables passed from
kernel_addr_r — and performs roughly the following steps:
- Load the kernel image (
vmlinuz) to the address
- Decompress it to
- Load the initramfs (
- Hand over the boot process to the actual kernel.
Unfortunately, the values of these environment variables are hardcoded in the bootloader. This means that the decompressed kernel must fit within the allocated area, or it will overflow into other data necessary for the boot process (e.g. the initramfs).
In some cases U-Boot is able to detect the overflow and will print out an error message.
This is why the bootloader patch is necessary.
The information above was gathered from the following sources:
- Kernel/BuildYourOwnKernel - Ubuntu Wiki
- KernelTeam/ARMKernelCrossCompile - Ubuntu Wiki
- Kernel building for Ubuntu - Raspberry Pi Forums
- Raspberry Pi Documentation - The Linux kernel
And perhaps others that I'm forgetting now.