For some recent depravity I "needed" to build a kernel with ASan for Ubuntu Server. While normally this would've been a relatively straightforward operation, in this case there was a further complication — the server is a Raspberry Pi.

Since documentation on building an Ubuntu kernel for the Raspberry Pi is virtually nonexistent, I'm documenting the process here for any poor souls who might need this in the future.

• Prerequisites
• Getting the source
• Preparing for the build
• Configuring the kernel
• Building
• Installing
• Patching the bootloader
• Checking that it works
• Appendix: Ubuntu Server RPi boot process
• References

Prerequisites

1. A Raspberry Pi 4, 4GB RAM, with Ubuntu Server 20.04.4.
2. A build (virtual) machine with Ubuntu.
   • Because you really don't want to build on the Pi directly.

That's what I worked with. The described methods will probably apply to other kernel versions and/or Raspberry Pi revisions, with or without modifications. Note that the RAM size might also be a factor. ASan requires quite a lot of memory, so systems with little RAM might not be usable with it.

YMMV.

Plea: Make a backup of your system before doing anything. Please.

Getting the source

On the Raspberry Pi, run:

apt-get --download-only source linux-image-$(uname -r)

This will download the source package for the currently-running kernel to the working directory. The output will contain at least a .dsc file and a tarball.

If this fails, you might need to uncomment the deb-src lines in /etc/apt/sources.list.

Transfer the source package files to your build machine.

Preparing for the build

Extract the source package:

dpkg-source -x something-something.dsc

Install build tools:

sudo apt install                \
    build-essential             \
    crossbuild-essential-arm64  \
    libncurses-dev              \
    gawk                        \
    flex                        \
    bison                       \
    openssl                     \
    dkms                        \
    libelf-dev                  \
    libudev-dev                 \
    libpci-dev                  \
    libiberty-dev               \
    autoconf                    \
    git                         \
    bc                          \
    libssl-dev                  \
    make                        \
    libc6-dev

Note: This list may be incomplete. That's what I installed, but the machine wasn't clean to begin with, so perhaps some necessary packages were already installed. Let me know if something is missing here.

Configuring the kernel

In the source directory run:

export $(dpkg-architecture -aarm64)
export CROSS_COMPILE=aarch64-linux-gnu-

(You can safely ignore the dpkg warning.)

Edit the first line in debian.raspi/changelog and add a custom version tag. For instance, if that line reads:

linux-raspi (5.4.0-1066.76) focal; urgency=medium

Change it to something like:

linux-raspi (5.4.0-1066.76+kasan) focal; urgency=medium

This will ensure that the kernel we're building is "newer" than the one currently installed.

Now run:

fakeroot debian/rules clean
fakeroot debian/rules editconfigs

The last command will prompt you to edit the kernel configuration. Answer "no" to the first and "yes" to the second, since we only want the arm64 version:

Do you want to edit config: armhf/config.flavour.raspi? [Y/n] n
Do you want to edit config: arm64/config.flavour.raspi? [Y/n] Y

Time to enable KASAN. Go to Kernel hacking -> Memory Debugging, and enable KASAN in Generic mode with Inline instrumentation. Also enable the "Module for testing KASAN for bug detection".

Here you may also want to enable page owner tracking. See here for more information.

Go back to the Kernel hacking menu and enable stack backtrace support:

Now exit and save the configuration when prompted.

Building

In the source directory, run:

fakeroot debian/rules binary skipdbg=false

This will build the kernel and additional debug binaries. If you don't need them, omit the skipdbg=false argument.

Installing

After the build finishes, the directory above the source directory will have several .deb packages, looking something like this:

linux-headers-5.4.0-1066-raspi_5.4.0-1066.76+kasan_arm64.deb
linux-image-5.4.0-1066-raspi_5.4.0-1066.76+kasan_arm64.deb
linux-modules-5.4.0-1066-raspi_5.4.0-1066.76+kasan_arm64.deb
linux-raspi-headers-5.4.0-1066_5.4.0-1066.76+kasan_arm64.deb

And if you built debug binaries you'll also have linux-image-5.4.0-1066-raspi-dbgsym_5.4.0-1066.76+kasan_arm64.ddeb.

Copy those over to the server and install with dpkg -i package.deb.

If you have a newer kernel, you may also want to configure KASAN using the kernel command-line. See here for more info, and use the cmdline parameter in the Pi's config.txt to change the command-line.

Patching the bootloader

Now comes the tricky part. The kernel we just built is too big for the bootloader to handle, so we'll need to patch it.

First, get this script:

Determine the size of the kernel you just built:

sudo file -L /boot/vmlinuz

Which should output something like:

/boot/vmlinuz: gzip compressed data, max compression, from Unix, original size modulo 2^32 55808512

Now, run:

./uboot_patch.py \
    -i /boot/firmware/uboot_rpi_4.bin
    -o ./uboot_rpi_4_patched.bin

This will patch the bootloader to support kernels up to 63.5MB in size. If your kernel is somehow larger, run the script with -s and specify a larger size.

Copy the patched bootloader to /boot/firmware, and edit config.txt to point to it instead of the original uboot_rpi_4.bin.

Checking that it works

Moment of truth.

sudo reboot

If everything went according to plan the system should reboot (it will take a little longer than usual), and in the dmesg output you'll see:

kasan: KernelAddressSanitizer initialized

If not... I hope you made that backup 😎.

Now to test that KASAN actually works:

sudo insmod /lib/modules/$(uname -r)/kernel/lib/test_kasan.ko

Which should dump something like this to the screen (not over SSH):

==================================================================
BUG: KASAN: slab-out-of-bounds in kmalloc_oob_right+0x94/0xb0 [test_kasan]
Write of size 1 at addr ffff77250ef6c77b by task insmod/3529

CPU: 2 PID: 3529 Comm: insmod Tainted: G         C  E     5.4.0-1066-raspi #76+kasan
Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
Call trace:
 dump_backtrace+0x0/0x2d8
 show_stack+0x28/0x38
 dump_stack+0x128/0x194
 print_address_description.isra.0+0x74/0x354
 __kasan_report+0x188/0x1d8
 kasan_report+0xc/0x18
 __asan_report_store1_noabort+0x1c/0x28
 kmalloc_oob_right+0x94/0xb0 [test_kasan]
 kmalloc_tests_init+0x18/0xd9c [test_kasan]
 do_one_initcall+0xbc/0x6e8
 do_init_module+0x154/0x5c0
 load_module+0x2644/0x3468
 __do_sys_finit_module+0x120/0x1a8
 __arm64_sys_finit_module+0x74/0xa8
 el0_svc_common.constprop.0+0x11c/0x488
 el0_svc_handler+0x50/0xd0
 el0_svc+0x10/0x200

Allocated by task 3529:
 save_stack+0x24/0xb0
 __kasan_kmalloc.isra.0+0xc0/0xe0
 kasan_kmalloc+0xc/0x18
 kmem_cache_alloc_trace+0x1ac/0x350
 kmalloc_oob_right+0x54/0xb0 [test_kasan]
 kmalloc_tests_init+0x18/0xd9c [test_kasan]
 do_one_initcall+0xbc/0x6e8
 do_init_module+0x154/0x5c0
 load_module+0x2644/0x3468
 __do_sys_finit_module+0x120/0x1a8
 __arm64_sys_finit_module+0x74/0xa8
 el0_svc_common.constprop.0+0x11c/0x488
 el0_svc_handler+0x50/0xd0
 el0_svc+0x10/0x200

Freed by task 2798:
 save_stack+0x24/0xb0
 __kasan_slab_free+0x108/0x180
 kasan_slab_free+0x10/0x18
 kfree+0xb0/0x390
 security_cred_free+0xbc/0x130
 put_cred_rcu+0x68/0x2a0
 rcu_do_batch+0x2c4/0x610
 rcu_core+0x2f4/0xb30
 rcu_core_si+0x18/0x20
 __do_softirq+0x324/0xc28

The buggy address belongs to the object at ffff77250ef6c700
 which belongs to the cache kmalloc-128 of size 128
The buggy address is located 123 bytes inside of
 128-byte region [ffff77250ef6c700, ffff77250ef6c780)
The buggy address belongs to the page:
page:ffffffdc941bdb00 refcount:1 mapcount:0 mapping:ffff772557403c00 index:0xffff77250ef6c200
flags: 0x4000000000000200(slab)
raw: 4000000000000200 dead000000000100 dead000000000122 ffff772557403c00
raw: ffff77250ef6c200 000000008010000d 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
 ffff77250ef6c600: 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc fc
 ffff77250ef6c680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>ffff77250ef6c700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03
                                                                ^
 ffff77250ef6c780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
 ffff77250ef6c800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==================================================================

If after this the system is still somehow running — reboot.

Enjoy your sanitized kernel!

Appendix: Ubuntu Server RPi boot process

The boot partition on the SD card of a Raspberry Pi Ubuntu Server contains a handy README file explaining the boot process. Briefly:

1. bootcode.bin - the second stage bootloader loaded by all Pi's with the exception of the Pi 4 (where this is replaced by flash memory).
2. config.txt - the first configuration file read by the boot process.
3. start*.elf - the third stage bootloader, which handles device-tree modification and which loads...
4. uboot*.bin - various U-Boot binaries for different Pi platforms; these are launched as the "kernel" by config.txt.
5. boot.scr - the boot script executed by uboot*.bin which in turn loads...
6. vmlinuz - the Linux kernel, executed by boot.scr.
7. initrd.img - the initramfs, executed by boot.scr.

Of particular interest is the boot.scr script. It relies on the values of two environment variables passed from uboot*.bin — ramdisk_addr_r and kernel_addr_r — and performs roughly the following steps:

1. Load the kernel image (vmlinuz) to the address ramdisk_addr_r.
2. Decompress it to kernel_addr_r.
3. Load the initramfs (initrd.img) to ramdisk_addr_r.
4. Hand over the boot process to the actual kernel.

Unfortunately, the values of these environment variables are hardcoded in the bootloader. This means that the decompressed kernel must fit within the allocated area, or it will overflow into other data necessary for the boot process (e.g. the initramfs).

In some cases U-Boot is able to detect the overflow and will print out an error message.

This is why the bootloader patch is necessary.

References

The information above was gathered from the following sources:

• Kernel/BuildYourOwnKernel - Ubuntu Wiki
• KernelTeam/ARMKernelCrossCompile - Ubuntu Wiki
• Kernel building for Ubuntu - Raspberry Pi Forums
• Raspberry Pi Documentation - The Linux kernel

And perhaps others that I'm forgetting now.