For some recent depravity I "needed" to build a kernel with ASan for Ubuntu Server. While normally this would've been a relatively straightforward operation, in this case there was a further complication — the server is a Raspberry Pi.
Since documentation on building an Ubuntu kernel for the Raspberry Pi is virtually nonexistent, I'm documenting the process here for any poor souls who might need this in the future.
- Prerequisites
- Getting the source
- Preparing for the build
- Configuring the kernel
- Building
- Installing
- Patching the bootloader
- Checking that it works
- Appendix: Ubuntu Server RPi boot process
- References
Prerequisites
- A Raspberry Pi 4, 4GB RAM, with Ubuntu Server 20.04.4.
- A build (virtual) machine with Ubuntu.
- Because you really don't want to build on the Pi directly.
That's what I worked with. The described methods will probably apply to other kernel versions and/or Raspberry Pi revisions, with or without modifications. Note that the RAM size might also be a factor. ASan requires quite a lot of memory, so systems with little RAM might not be usable with it.
YMMV.
Plea: Make a backup of your system before doing anything. Please.
Getting the source
On the Raspberry Pi, run:
apt-get --download-only source linux-image-$(uname -r)
This will download the source package for the currently-running kernel
to the working directory. The output will contain at least a .dsc
file and a tarball.
If this fails, you might need to uncomment the deb-src
lines in
/etc/apt/sources.list
.
Transfer the source package files to your build machine.
Preparing for the build
Extract the source package:
dpkg-source -x something-something.dsc
Install build tools:
sudo apt install \
build-essential \
crossbuild-essential-arm64 \
libncurses-dev \
gawk \
flex \
bison \
openssl \
dkms \
libelf-dev \
libudev-dev \
libpci-dev \
libiberty-dev \
autoconf \
git \
bc \
libssl-dev \
make \
libc6-dev
Note: This list may be incomplete. That's what I installed, but the machine wasn't clean to begin with, so perhaps some necessary packages were already installed. Let me know if something is missing here.
Configuring the kernel
In the source directory run:
export $(dpkg-architecture -aarm64)
export CROSS_COMPILE=aarch64-linux-gnu-
(You can safely ignore the dpkg
warning.)
Edit the first line in debian.raspi/changelog
and add a custom version tag.
For instance, if that line reads:
linux-raspi (5.4.0-1066.76) focal; urgency=medium
Change it to something like:
linux-raspi (5.4.0-1066.76+kasan) focal; urgency=medium
This will ensure that the kernel we're building is "newer" than the one currently installed.
Now run:
fakeroot debian/rules clean
fakeroot debian/rules editconfigs
The last command will prompt you to edit the kernel configuration. Answer "no" to
the first and "yes" to the second, since we only want the arm64
version:
Do you want to edit config: armhf/config.flavour.raspi? [Y/n] n
Do you want to edit config: arm64/config.flavour.raspi? [Y/n] Y
Time to enable KASAN. Go to Kernel hacking -> Memory Debugging
, and enable
KASAN in Generic mode with Inline instrumentation. Also enable
the "Module for testing KASAN for bug detection".
Here you may also want to enable page owner tracking. See here for more information.
Go back to the Kernel hacking
menu and enable stack backtrace support:
Now exit and save the configuration when prompted.
Building
In the source directory, run:
fakeroot debian/rules binary skipdbg=false
This will build the kernel and additional debug binaries. If you don't need them,
omit the skipdbg=false
argument.
Installing
After the build finishes, the directory above the source directory will have
several .deb
packages, looking something like this:
linux-headers-5.4.0-1066-raspi_5.4.0-1066.76+kasan_arm64.deb
linux-image-5.4.0-1066-raspi_5.4.0-1066.76+kasan_arm64.deb
linux-modules-5.4.0-1066-raspi_5.4.0-1066.76+kasan_arm64.deb
linux-raspi-headers-5.4.0-1066_5.4.0-1066.76+kasan_arm64.deb
And if you built debug binaries you'll also have
linux-image-5.4.0-1066-raspi-dbgsym_5.4.0-1066.76+kasan_arm64.ddeb
.
Copy those over to the server and install with dpkg -i package.deb
.
If you have a newer kernel, you may also want to configure KASAN using the kernel
command-line. See here for more info, and use
the cmdline
parameter in the Pi's config.txt
to change the command-line.
Patching the bootloader
Now comes the tricky part. The kernel we just built is too big for the bootloader to handle, so we'll need to patch it.
First, get this script:
Determine the size of the kernel you just built:
sudo file -L /boot/vmlinuz
Which should output something like:
/boot/vmlinuz: gzip compressed data, max compression, from Unix, original size modulo 2^32 55808512
Now, run:
./uboot_patch.py \
-i /boot/firmware/uboot_rpi_4.bin
-o ./uboot_rpi_4_patched.bin
This will patch the bootloader to support kernels up to 63.5MB in size. If your kernel
is somehow larger, run the script with -s
and specify a larger size.
Copy the patched bootloader to /boot/firmware
, and edit config.txt
to point
to it instead of the original uboot_rpi_4.bin
.
Checking that it works
Moment of truth.
sudo reboot
If everything went according to plan the system should reboot (it will take a little
longer than usual), and in the dmesg
output you'll see:
kasan: KernelAddressSanitizer initialized
If not... I hope you made that backup 😎.
Now to test that KASAN actually works:
sudo insmod /lib/modules/$(uname -r)/kernel/lib/test_kasan.ko
Which should dump something like this to the screen (not over SSH):
==================================================================
BUG: KASAN: slab-out-of-bounds in kmalloc_oob_right+0x94/0xb0 [test_kasan]
Write of size 1 at addr ffff77250ef6c77b by task insmod/3529
CPU: 2 PID: 3529 Comm: insmod Tainted: G C E 5.4.0-1066-raspi #76+kasan
Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
Call trace:
dump_backtrace+0x0/0x2d8
show_stack+0x28/0x38
dump_stack+0x128/0x194
print_address_description.isra.0+0x74/0x354
__kasan_report+0x188/0x1d8
kasan_report+0xc/0x18
__asan_report_store1_noabort+0x1c/0x28
kmalloc_oob_right+0x94/0xb0 [test_kasan]
kmalloc_tests_init+0x18/0xd9c [test_kasan]
do_one_initcall+0xbc/0x6e8
do_init_module+0x154/0x5c0
load_module+0x2644/0x3468
__do_sys_finit_module+0x120/0x1a8
__arm64_sys_finit_module+0x74/0xa8
el0_svc_common.constprop.0+0x11c/0x488
el0_svc_handler+0x50/0xd0
el0_svc+0x10/0x200
Allocated by task 3529:
save_stack+0x24/0xb0
__kasan_kmalloc.isra.0+0xc0/0xe0
kasan_kmalloc+0xc/0x18
kmem_cache_alloc_trace+0x1ac/0x350
kmalloc_oob_right+0x54/0xb0 [test_kasan]
kmalloc_tests_init+0x18/0xd9c [test_kasan]
do_one_initcall+0xbc/0x6e8
do_init_module+0x154/0x5c0
load_module+0x2644/0x3468
__do_sys_finit_module+0x120/0x1a8
__arm64_sys_finit_module+0x74/0xa8
el0_svc_common.constprop.0+0x11c/0x488
el0_svc_handler+0x50/0xd0
el0_svc+0x10/0x200
Freed by task 2798:
save_stack+0x24/0xb0
__kasan_slab_free+0x108/0x180
kasan_slab_free+0x10/0x18
kfree+0xb0/0x390
security_cred_free+0xbc/0x130
put_cred_rcu+0x68/0x2a0
rcu_do_batch+0x2c4/0x610
rcu_core+0x2f4/0xb30
rcu_core_si+0x18/0x20
__do_softirq+0x324/0xc28
The buggy address belongs to the object at ffff77250ef6c700
which belongs to the cache kmalloc-128 of size 128
The buggy address is located 123 bytes inside of
128-byte region [ffff77250ef6c700, ffff77250ef6c780)
The buggy address belongs to the page:
page:ffffffdc941bdb00 refcount:1 mapcount:0 mapping:ffff772557403c00 index:0xffff77250ef6c200
flags: 0x4000000000000200(slab)
raw: 4000000000000200 dead000000000100 dead000000000122 ffff772557403c00
raw: ffff77250ef6c200 000000008010000d 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
Memory state around the buggy address:
ffff77250ef6c600: 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff77250ef6c680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>ffff77250ef6c700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03
^
ffff77250ef6c780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff77250ef6c800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==================================================================
If after this the system is still somehow running — reboot.
Enjoy your sanitized kernel!
Appendix: Ubuntu Server RPi boot process
The boot partition on the SD card of a Raspberry Pi Ubuntu Server contains a handy
README
file explaining the boot process. Briefly:
bootcode.bin
- the second stage bootloader loaded by all Pi's with the exception of the Pi 4 (where this is replaced by flash memory).config.txt
- the first configuration file read by the boot process.start*.elf
- the third stage bootloader, which handles device-tree modification and which loads...uboot*.bin
- various U-Boot binaries for different Pi platforms; these are launched as the "kernel" byconfig.txt
.boot.scr
- the boot script executed byuboot*.bin
which in turn loads...vmlinuz
- the Linux kernel, executed byboot.scr
.initrd.img
- the initramfs, executed byboot.scr
.
Of particular interest is the boot.scr
script. It relies on the values of two
environment variables passed from uboot*.bin
— ramdisk_addr_r
and
kernel_addr_r
— and performs roughly the following steps:
- Load the kernel image (
vmlinuz
) to the addressramdisk_addr_r
. - Decompress it to
kernel_addr_r
. - Load the initramfs (
initrd.img
) toramdisk_addr_r
. - Hand over the boot process to the actual kernel.
Unfortunately, the values of these environment variables are hardcoded in the bootloader. This means that the decompressed kernel must fit within the allocated area, or it will overflow into other data necessary for the boot process (e.g. the initramfs).
In some cases U-Boot is able to detect the overflow and will print out an error message.
This is why the bootloader patch is necessary.
References
The information above was gathered from the following sources:
- Kernel/BuildYourOwnKernel - Ubuntu Wiki
- KernelTeam/ARMKernelCrossCompile - Ubuntu Wiki
- Kernel building for Ubuntu - Raspberry Pi Forums
- Raspberry Pi Documentation - The Linux kernel
And perhaps others that I'm forgetting now.