This is Part 3 of the Embedded Linux series. The previous parts covered building the toolchain and setting up the full boot chain with TF-A and U-Boot. If you are starting here, go through those first:
Embedded Linux Part 1 — How to Build an ARM Cross-Compiling Toolchain with Crosstool-NG
Embedded Linux Part 2 — Booting a Raspberry Pi 4 with TF-A and U-Boot: A Practical Journey
With TF-A and U-Boot running, the full boot chain is in place. The next step is bringing up the Linux kernel itself — from source to first execution on hardware. This post covers fetching the sources, configuring for RPi4, cross-compiling, and reading the first kernel messages.
What you need before starting
- The AArch64 cross-compiler from Part 1 in your PATH
- A working TF-A + U-Boot setup from Part 2
- The SD card prepared in Part 2 (FAT32 boot partition + ext4 rootfs partition)
bcandlibssl-devinstalled on the host:
sudo apt install bc libssl-dev
bc is a Unix calculator tool the kernel Makefile uses internally for version number arithmetic. You never call it yourself, but the build fails immediately without it.
Set the environment variables before anything else:
export PATH=$HOME/x-tools/aarch64-training-linux-musl/bin:$PATH
export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-
ARCH=arm64 tells the kernel Makefile to build for 64-bit ARM instead of the host x86. CROSS_COMPILE=aarch64-linux- tells it which compiler prefix to use. Without both, the build produces an x86 kernel that cannot run on the RPi4.
Always verify before building:
${CROSS_COMPILE}gcc --version
# Must print the AArch64 compiler, not your host gcc
Part A — Getting the kernel sources
A.1 Understanding the two kernel trees
The Linux kernel development model has two trees. The mainline tree is maintained by Linus Torvalds and contains cutting-edge development code. The stable tree is maintained by a separate team and contains backported bug fixes for specific long-term versions.
For embedded systems work, always use a stable version. The mainline tree is cloned first because it is the official reference, then the stable tree is added as a second remote:
mkdir -p $HOME/embedded-linux-rpi4/kernel
cd $HOME/embedded-linux-rpi4/kernel
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
cd linux
git remote add stable \
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux
git fetch stable
The initial clone downloads about 3.2 GB. The stable fetch adds another 5.7 GB covering every stable branch from 2.6.11 to the present. Start it and come back later.
A.2 Checking out the 6.6 stable branch
git checkout stable/linux-6.6.y
make kernelversion
# 6.6.134
The detached HEAD warning from git is normal here. We are not on a named branch but on a specific commit from the stable tree. This is expected and correct.
A.3 Why kernel version 6.6?
Kernel 6.6 is an LTS (Long-Term Support) release, meaning it receives backported security fixes and bug fixes for several years. For embedded products, LTS kernels are the standard choice. The goal is not chasing new features — it is having a stable, well-tested base that will be maintained throughout the product lifetime.
The BCM2711 SoC support in 6.6 is mature. All the drivers needed for this lab — MMC, USB, Ethernet, I2C, UART, GPIO — are present and stable.
Part B — Configuring the kernel
B.1 Generating the default configuration
The ARM64 architecture has a single defconfig that covers all supported platforms:
make ARCH=arm64 CROSS_COMPILE=aarch64-linux- defconfig
This produces a .config file with sensible defaults for ARM64. It enables far more than needed — every platform, every driver family, every filesystem. Use menuconfig to trim it down:
make ARCH=arm64 CROSS_COMPILE=aarch64-linux- menuconfig
B.2 Disabling unnecessary platforms
Navigate to Platform Selection. You will see a long list of ARM SoC families, all enabled by default. Enter Broadcom SoC Support and disable everything except Broadcom BCM2835 family. The BCM2711 on RPi4 is part of this family. Disable iProc, BCA, and Set-Top-Box SoCs — they add compile time without adding anything useful for this board.
B.3 Disabling the display subsystem
Search for DRM using the / key. Navigate to Device Drivers → Graphics support and disable Direct Rendering Manager. Disabling the top-level entry automatically disables all the GPU sub-drivers — Nouveau, AMD, Broadcom V3D, VC4, and dozens of panel drivers. This alone saves 5-10 minutes of compile time for a headless embedded system that has no display.
Save and exit.
Part C — Understanding the Device Tree
C.1 What a Device Tree is and why it matters
On x86, the BIOS or UEFI tells the operating system what hardware exists at boot time. On ARM, there is no such standard firmware. Instead, a Device Tree file describes the hardware in a structured text format.
The Device Tree Source (.dts) for RPi4 describes everything about the BCM2711 SoC and the board: how much RAM exists, where the UART registers are mapped in memory, which GPIO pins connect to what, what I2C buses exist, interrupt numbers, clock frequencies, and the properties of every peripheral.
The build system compiles the .dts file into a binary .dtb file. U-Boot loads this blob into RAM and passes its address to the kernel at boot. The kernel reads it and loads the appropriate driver for each described device. Without the DTB, the kernel has no idea what hardware it is running on.
C.2 The ARM64 boot protocol
Before U-Boot transfers control to the kernel, the ARM64 boot protocol requires specific conditions:
x0 = physical address of the DTB in RAM
x1 = 0 (reserved)
x2 = 0 (reserved)
x3 = 0 (reserved)
MMU = disabled
CPU = running at EL2 or EL1
U-Boot's booti command ensures all of these conditions before jumping to the kernel entry point. The DTB address in x0 is how the kernel finds its hardware description the moment it starts executing. If this address points to garbage, the kernel cannot discover any hardware and will crash very early.
Part D — Building the kernel
D.1 The build command
With configuration saved, the build is a single command:
make -j$(nproc) Image.gz dtbs
-j$(nproc) runs one compilation job per CPU core in parallel. On a 4-core machine this compiles four files simultaneously instead of one at a time, roughly 4x faster.
The two build targets are:
Image.gzproduces the compressed kernel binary. The build compiles all kernel source files, links them intovmlinux, strips it toImage, then gzip-compresses it. U-Boot will decompress it back toImagein RAM before jumping to it.dtbscompiles all Device Tree Source files into Device Tree Blobs. We specifically needbcm2711-rpi-4-b.dtbfor the Raspberry Pi 4B.
Specifying both targets instead of running bare make skips module compilation, saving significant time. The build takes 20-30 minutes on a typical VM with 4 cores. When it finishes:
GZIP arch/arm64/boot/Image.gz
D.2 Verifying the output
ls -lh arch/arm64/boot/Image.gz
ls -lh arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb
Both files must exist before continuing.
Part E — Copying to the SD card
We load the kernel from the SD card rather than TFTP for now. The VM networking setup adds unnecessary complexity at this stage. The kernel and DTB go on partition 2, the ext4 rootfs partition:
sudo mount /dev/sdb2 /mnt
sudo cp arch/arm64/boot/Image.gz /mnt/
sudo cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb /mnt/
ls -lh /mnt/
sudo umount /mnt
sync
Part F — Memory layout and U-Boot variables
F.1 Understanding the memory map
Before loading the kernel, it is important to understand where everything will sit in RAM. Running bdinfo in U-Boot shows the available RAM and reserved regions:
memory[0] 0x00000000 - 0x3b3fffff 948MB main RAM
reserved[0] 0x00000000 - 0x0007ffff first 512KB reserved
reserved[1] 0x39f31000 - 0x3b3fffff U-Boot itself
The safe usable window is 0x00080000 to 0x39f30fff. Within that window, three things are placed at non-overlapping addresses:
SD card partition 2
|
| load mmc 0:2
v
+------------------+
| Image.gz | @ 0x01000000 (~8MB compressed)
+------------------+
| DTB | @ 0x02600000 (~50KB)
+------------------+
|
| booti (decompress)
v
+------------------+
| Kernel (Image) | @ 0x10000000 (up to 32MB decompressed)
+------------------+
|
| jump, x0 = DTB address
v
Linux kernel
Incorrect memory placement often leads to silent crashes with no error output. Always compare chosen addresses against
bdinfobefore loading. A useful habit is to sketch the layout on paper first.
The DTB is placed at 0x02600000, matching RPI3_PRELOADED_DTB_BASE from the TF-A build. TF-A already patched the DTB at that address — keeping it there ensures consistency with the expected memory layout.
F.2 Setting and saving U-Boot variables
Set these variables at the U-Boot prompt so they persist across reboots:
U-Boot> setenv kernel_addr_r 0x01000000
U-Boot> setenv fdt_addr_r 0x02600000
U-Boot> setenv kernel_comp_addr_r 0x10000000
U-Boot> setenv kernel_comp_size 0x2000000
U-Boot> saveenv
What each variable means:
kernel_addr_r: address where U-Boot loads the compressed kernel image from MMC or TFTP.fdt_addr_r: address where U-Boot loads the Device Tree Blob.kernel_comp_addr_r: address where U-Boot decompressesImage.gzbefore jumping to it. Must not overlap with the compressed image or the DTB.kernel_comp_size: maximum size allocated for decompression.0x2000000is 32MB, a safe upper bound.
Without saveenv, all variables disappear on the next power cycle. In a development workflow where the board is rebooted constantly, losing all settings on every power cycle wastes significant time.
Part G — Loading and booting the kernel
G.1 Loading from the SD card
At the U-Boot prompt:
U-Boot> load mmc 0:2 0x01000000 Image.gz
U-Boot> load mmc 0:2 0x02600000 bcm2711-rpi-4-b.dtb
U-Boot> booti 0x01000000 - 0x02600000
load mmc 0:2 means: load from MMC device 0 (the SD card), partition 2 (the ext4 partition). The - in the booti command is a placeholder for the initrd address — there is no initrd at this stage.
booti decompresses Image.gz to kernel_comp_addr_r (0x10000000), then jumps to the kernel entry point, passing the DTB address in register x0 as defined by the ARM64 boot protocol. U-Boot ensures the MMU is disabled and the CPU is at EL2 before the jump.
G.2 Reading the boot output
When the kernel starts, the serial console fills with messages. Here is what the important ones mean:
Booting Linux on physical CPU 0x0000000000 [0x410fd083]
The kernel started on CPU0. 0x410fd083 is the MIDR register value identifying this as a Cortex-A72 revision 3.
Linux version 6.6.134 (tontsa@tontsa-virtual-machine)
(aarch64-linux-gcc (crosstool-NG 1.28.0) 14.3.0)
The compiled kernel, built with the cross-compiler. The hostname and toolchain version appear here as confirmation.
Machine model: Raspberry Pi 4 Model B
The kernel read the DTB and identified the board correctly.
CPU features: detected: Spectre-v2
CPU features: detected: Spectre-v4
CPU features: detected: Spectre-BHB
The kernel applied Spectre mitigations. TF-A BL31 applied these at EL3 during boot — the kernel detects and confirms them here.
CPU: All CPU(s) started at EL2
Confirms the EL transition set up by TF-A. U-Boot ran at EL2 and the kernel entered at EL2 before dropping to EL1 where it runs permanently.
Serial: AMBA PL011 UART driver
fe201000.serial: ttyAMA0 at MMIO 0xfe201000
The PL011 UART driver loaded and registered ttyAMA0 — the same UART being used for this serial console session.
CPU1: failed to come online
CPU2: failed to come online
CPU3: failed to come online
SMP: Total of 1 processors activated.
Only one of four cores started. The kernel uses PSCI to bring up secondary CPUs via SMC calls to TF-A BL31. This failure usually indicates that the PSCI nodes in the DTB are not fully aligned with the TF-A configuration. It is not caused by missing bootargs. This will be investigated in a later lab.
G.3 The expected kernel panic
VFS: Cannot open root device "" or unknown-block(0,0): error -6
Kernel panic: Unable to mount root fs on unknown-block(0,0)
This is not a failure. This is the correct outcome at this stage. The kernel booted successfully, initialised all its subsystems, then looked for a root filesystem and found nothing — because one has not been built yet.
The kernel lists the block device filesystems compiled into it:
List of all bdev filesystems:
ext3
ext4
ext2
squashfs
vfat
When a root filesystem is provided in the next lab, the kernel will mount it and continue booting to a shell.
What comes next
The kernel is running. The missing piece is a root filesystem — the directory tree containing the init process, shell, and basic userspace tools that the kernel hands control to after its own initialisation.
The next lab builds a minimal root filesystem from scratch using BusyBox, sets up TFTP for rapid kernel iteration, mounts the rootfs over NFS, and produces the first interactive shell on the RPi4.
Lab materials adapted from Bootlin's Embedded Linux training (bootlin.com/training/embedded-linux) — CC BY-SA 3.0