This is Part 4 of the Embedded Linux series. The previous parts covered building the toolchain, setting up TF-A and U-Boot, and cross-compiling the Linux kernel. If you are starting here, go through those first:
Embedded Linux Part 1 — How to Build an ARM Cross-Compiling Toolchain with Crosstool-NG
Embedded Linux Part 2 — Booting a Raspberry Pi 4 with TF-A and U-Boot: A Practical Journey
Embedded Linux Part 3 — Cross-Compiling Linux 6.6 for Raspberry Pi 4: From Source to First Boot
With TF-A, U-Boot, and the Linux kernel running on the Raspberry Pi 4, the boot chain is solid. But a kernel with no userspace is just a very expensive panic message. This post covers everything that came next: switching from SD card to TFTP kernel loading, building a minimal root filesystem with BusyBox, mounting it over NFS, fixing every obstacle along the way, and eventually getting a working shell prompt on real hardware.
In Part 3, the kernel was loaded from the SD card. That works for a first boot, but it is not practical for a development workflow where the kernel is rebuilt and reconfigured constantly. Writing a new image to the SD card on every iteration is slow and error-prone. TFTP solves this: the board fetches the latest kernel directly from the host over the network the moment U-Boot starts. No SD card writes, no unmounting, no cables. Just save the file and reboot.
The lab data files used in this article, including the BusyBox configuration and the web interface files, are part of Bootlin's open-source Embedded Linux training and can be downloaded from bootlin.com/training/embedded-linux.
This is not a clean tutorial. It is a real account of what actually happened, including the wrong turns, the silent failures, and the lessons learned from each one.
The goal
The target state at the end of this lab:
U-Boot (TFTP) → Linux kernel → BusyBox shell over NFS
Instead of loading the kernel from the SD card on every iteration, TFTP lets us push a new kernel from the host the moment a build finishes. Instead of writing a root filesystem to the SD card, NFS serves it directly from a directory on the host. Changes to the rootfs appear on the board instantly. This is the standard embedded Linux development workflow.
Part A — Setting up TFTP
A.1 What TFTP is and why we use it
TFTP (Trivial File Transfer Protocol) is the simplest possible file transfer protocol: no authentication, no connection state, just a server that responds to file requests on UDP port 69. U-Boot has a built-in TFTP client. The board requests a file by name, the server sends it, and U-Boot loads it into RAM.
A.2 Installing the TFTP server
sudo apt install tftpd-hpa
sudo mkdir -p /srv/tftp
sudo chown tftp:tftp /srv/tftp
sudo systemctl enable --now tftpd-hpa
Copy the kernel and DTB to the TFTP directory:
cp arch/arm64/boot/Image.gz /srv/tftp/
cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb /srv/tftp/
A.3 Fixing the host networking
Before any of this works, the host VM needs a stable IP address on the interface connected to the RPi4. Our setup uses VMware with two network adapters: ens33 for internet access via NAT, and ens37 bridged directly to the physical Ethernet port where the RPi4 cable connects.
The first problem: ens37 kept losing its IP address after every reboot. NetworkManager was removing manually assigned addresses. The fix was to create a proper NetworkManager connection profile:
sudo nmcli con add \
type ethernet \
ifname ens37 \
ip4 192.168.0.1/24 \
con-name rpi4-link \
ipv4.method manual
sudo nmcli con up rpi4-link
The second problem: VMware's bridged adapter was set to Automatic, which meant it picked a random host network adapter. Sometimes the Wi-Fi, sometimes a VirtualBox virtual adapter, never reliably the physical Ethernet port. The fix was to open VMware Virtual Network Editor as Administrator, find VMnet0, and set it to bridge exclusively to the Realtek PCIe GbE Family Controller, the physical Ethernet port with the cable to the RPi4.
A.4 Testing TFTP from U-Boot
U-Boot> setenv ipaddr 192.168.0.100
U-Boot> setenv serverip 192.168.0.1
U-Boot> ping 192.168.0.1
host 192.168.0.1 is alive
Once ping works, TFTP boot works:
U-Boot> setenv kernel_comp_addr_r 0x10000000
U-Boot> setenv kernel_comp_size 0x2000000
U-Boot> tftp 0x01000000 Image.gz
U-Boot> tftp 0x02600000 bcm2711-rpi-4-b.dtb
U-Boot> booti 0x01000000 - 0x02600000
Part B — SD card boot as a fallback
B.1 Why the fallback matters
Before TFTP was working reliably, the kernel and DTB were loaded from the SD card. This is a critical fallback to understand, because it revealed the real root cause of several problems that would otherwise have been impossible to diagnose.
Loading from SD card uses U-Boot's load mmc command instead of tftp:
U-Boot> load mmc 0:2 0x01000000 Image.gz
U-Boot> load mmc 0:2 0x02600000 bcm2711-rpi-4-b.dtb
U-Boot> setenv kernel_comp_addr_r 0x10000000
U-Boot> setenv kernel_comp_size 0x2000000
U-Boot> booti 0x01000000 - 0x02600000
mmc 0:2 means MMC device 0 (the SD card), partition 2 (the ext4 rootfs partition).
B.2 Using it as a diagnostic tool
Early in the TFTP bring-up, loading the kernel via TFTP produced a silent hang after Starting kernel .... Loading the exact same kernel binary from the SD card produced full output. This difference was the key diagnostic: the kernel binary was not the problem. Something in the boot path differed between the two methods.
The actual cause turned out to be a misconfigured serial console. But the SD card fallback was what proved the kernel itself was fine and directed the investigation toward the boot environment rather than the kernel build.
The lesson is practical: always keep the ability to load from SD card. When TFTP boot misbehaves, one load mmc command tells you immediately whether the problem is in the kernel or in the TFTP/network path.
B.3 Automating boot with saveenv and bootcmd
With TFTP working, the next logical step is to automate the entire boot sequence so the board boots without any manual input. U-Boot provides two mechanisms for this: saveenv persists the environment variables to uboot.env on the FAT boot partition so they survive power cycles, and bootcmd defines the sequence of commands U-Boot runs automatically after the countdown expires.
Save the memory layout variables:
U-Boot> setenv kernel_addr_r 0x01000000
U-Boot> setenv fdt_addr_r 0x02600000
U-Boot> setenv kernel_comp_addr_r 0x10000000
U-Boot> setenv kernel_comp_size 0x2000000
U-Boot> saveenv
Saving Environment to FAT... OK
Then set bootcmd to fetch the kernel and DTB via TFTP and boot:
U-Boot> setenv bootcmd 'setenv kernel_comp_addr_r 0x10000000; setenv kernel_comp_size 0x2000000; tftp 0x01000000 Image.gz; tftp 0x02600000 bcm2711-rpi-4-b.dtb; booti 0x01000000 - 0x02600000'
U-Boot> saveenv
After the autoboot countdown expires, U-Boot fetches the latest kernel from the TFTP server and boots automatically. From this point on, the development cycle is: rebuild the kernel on the host, copy Image.gz to /srv/tftp/, and power cycle the board. Nothing else is needed.
Part C — The console mystery
C.1 Silence after Starting kernel
Loading the kernel via TFTP worked. But after Starting kernel ..., silence. No output at all. The same kernel loaded from the SD card produced full output. Loading via TFTP produced nothing.
The investigation took a long time. The eventual finding: the kernel was booting and outputting normally, but to the wrong console. When no bootargs were set, the kernel defaulted to ttyS1, the BCM2835 auxiliary mini UART. When console=ttyAMA0,115200 was added to bootargs, the kernel waited for the PL011 UART driver to register before producing output. That driver takes 16 seconds to initialize on RPi4 due to the firmware mailbox dependency.
C.2 The fix
The solution was to use console=ttyS1,115200:
U-Boot> setenv bootargs console=ttyS1,115200 root=/dev/nfs ...
This single change unlocked full kernel output. Every previous hang that looked like a crash was actually the kernel running fine but printing to a console we were not listening to.
Part D — Setting up NFS
D.1 What NFS provides
NFS (Network File System) allows the RPi4 to mount a directory from the host as its root filesystem. Every file read and write goes over the network. For development this is invaluable: modify a file on the host and the change is immediately visible on the board without touching the SD card.
D.2 Configuring the NFS server
sudo apt install nfs-kernel-server
mkdir -p /home/tontsa/embedded-linux-rpi4/nfsroot
Add the export to /etc/exports:
/home/tontsa/embedded-linux-rpi4/nfsroot 192.168.0.100(rw,no_root_squash,no_subtree_check)
Apply and verify:
sudo exportfs -ar
showmount -e localhost
D.3 Configuring U-Boot bootargs for NFS
U-Boot> setenv bootargs console=ttyS1,115200 root=/dev/nfs ip=192.168.0.100:::::eth0 nfsroot=192.168.0.1:/home/tontsa/embedded-linux-rpi4/nfsroot,nfsvers=3,tcp rw
The ip= parameter tells the kernel to configure eth0 with a static IP before attempting the NFS mount. The nfsvers=3,tcp specifies NFS version 3 over TCP.
Part E — Kernel driver problems
E.1 The Genet Ethernet driver problem
After setting up NFS, the kernel still could not mount the root filesystem. The logs showed VFS: Unable to mount root fs via NFS at 125 seconds, the NFS timeout. There was no bcmgenet driver output, no eth0, no IP assignment. The Ethernet driver simply was not loading.
The root cause was discovered by checking the kernel config:
grep "CONFIG_BCMGENET" .config
CONFIG_BCMGENET=m
During an earlier interactive make oldconfig session, the Genet driver had been silently changed from built-in (y) to module (m). Since there was no rootfs yet to load modules from, the driver never loaded and the network never came up. This is a classic chicken-and-egg problem: the rootfs is on NFS, the NFS needs the network, the network driver is in the rootfs.
The fix: set CONFIG_BCMGENET=y in menuconfig so the driver is compiled directly into the kernel image.
E.2 The WiFi ADMA flood
Before discovering the Genet issue, another problem consumed hours of debugging: the boot log was flooded with hundreds of mmc1: ADMA error messages from timestamp 16 seconds to 127 seconds:
mmc1: error -5 whilst initialising SD card
mmc1: ADMA error: 0x02000000
mmc1 is the SDIO slot used by the BCM43455 WiFi chip. With no WiFi driver present, the SDIO controller kept probing and failing in a tight loop. These errors consumed all the available probe time, preventing the Ethernet driver from initializing even when it was built-in.
The fix: disable CONFIG_BRCMFMAC in menuconfig, the Broadcom FullMAC WiFi driver. Without the driver, the SDIO bus probes mmc1 once, finds nothing to attach to, and moves on silently. The flood disappears and the Ethernet driver initializes cleanly.
Part F — Building BusyBox
F.1 What BusyBox is
BusyBox is a single binary that implements over 300 Unix utilities: sh, ls, cat, mount, ps, httpd, and many more. It is the standard userspace for minimal embedded Linux systems.
F.2 Getting the sources
mkdir -p $HOME/embedded-linux-rpi4/tinysystem
cd $HOME/embedded-linux-rpi4/tinysystem
git clone https://git.busybox.net/busybox
cd busybox
git checkout 1_37_stable
F.3 Configuration
Copy the Bootlin-provided config and open menuconfig:
cp $HOME/embedded-linux-rpi4/data/busybox-1.37.config .config
make ARCH=arm64 CROSS_COMPILE=aarch64-linux- menuconfig
Two critical settings in Settings:
- Build static binary: enabled for the first build. This compiles the C library directly into BusyBox, producing a single self-contained binary with no shared library dependencies. Essential when the rootfs has no shared libraries yet.
- Installation prefix: set to the nfsroot directory so
make installputs everything in the right place.
F.4 The GCC 14 patch
The lab provided a patch for a BusyBox 1.37 bug that causes menuconfig to fail with GCC 14. In our case git am failed to apply the patch cleanly, but menuconfig opened fine anyway. The bug was not present with our specific GCC version. When in doubt, try menuconfig directly before applying workarounds.
F.5 The SHA hardware acceleration build error
The first build attempt failed with:
libbb/hash_md5_sha.c: error: 'sha1_process_block64_shaNI' undeclared
BusyBox 1.37 tries to use SHA hardware acceleration intrinsics that are not available when cross-compiling for ARM64 from an x86 host. The compiler on the build machine has no knowledge of ARM SHA extension instructions, so the reference to sha1_process_block64_shaNI goes unresolved.
The fix is to disable SHA hardware acceleration in menuconfig. Search with / for SHA1_HWACCEL and SHA256_HWACCEL, navigate to each one and press N to disable them. With hardware acceleration disabled, BusyBox falls back to a portable software implementation that compiles cleanly on any host architecture.
This is a good example of why menuconfig is preferable to direct .config editing for fixes like this. It handles symbol dependencies correctly and gives a visual confirmation that the option is in the expected state before rebuilding.
F.6 Build and install
make -j$(nproc)
make install
The nfsroot now contains bin, sbin, usr, and linuxrc.
Part G — First boot: the kernel panic cascade
The first NFS boot attempt produced the correct sequence of expected failures:
Attempt 1: devtmpfs: error mounting -2
The kernel tried to mount devtmpfs on /dev but the directory did not exist. Fix: mkdir /home/tontsa/embedded-linux-rpi4/nfsroot/dev.
Attempt 2: Kernel panic: No working init found
The kernel mounted NFS successfully but found an empty rootfs with no init process. This was the correct state before BusyBox was installed.
Attempt 3 (after BusyBox install): Same panic.
NFS mount was still failing due to the Genet driver being a module. Fixed by setting CONFIG_BCMGENET=y.
After fixing the Genet driver, the boot log finally showed what we had been waiting for:
bcmgenet fd580000.ethernet: GENET 5.0 EPHY: 0x0000
bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full
IP-Config: Complete:
device=eth0, ipaddr=192.168.0.100
VFS: Mounted root (nfs filesystem) on device 0:21.
Run /sbin/init as init process
Press Enter at the prompt:
~ #
A BusyBox shell running on a Raspberry Pi 4 with a Linux 6.6.134 kernel built entirely from source.
Part H — Setting up the init system
H.1 Creating inittab
BusyBox init reads /etc/inittab to know what to run at boot. Without it, it prints warnings and falls back to a basic shell. Create a proper inittab:
::sysinit:/etc/init.d/rcS
ttyS1::askfirst:/bin/sh
::ctrlaltdel:/sbin/reboot
::shutdown:/bin/umount -a -r
Note: the Bootlin lab specifies ttyS2 for the shell. On RPi4 with a mainline kernel, the correct device is ttyS1, the BCM2835 auxiliary UART. Using ttyS2 produces a can't access tty; job control turned off warning because that device does not exist.
H.2 Creating the startup script
The startup script mounts the virtual filesystems:
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
Without /proc mounted, commands like ps and halt fail. With it mounted, ps shows the running processes and halt cleanly unmounts filesystems before powering down.
Part I — Switching to shared libraries
I.1 The dynamic linker
With the dynamic linker ld-musl-aarch64.so.1 copied to /lib on the target, dynamically linked programs run correctly. The test:
# On host
aarch64-linux-gcc -o hello hello.c
cp hello /home/tontsa/embedded-linux-rpi4/nfsroot/
# On target
./hello
Hello world!
I.2 Rebuilding BusyBox with shared libraries
BusyBox was then rebuilt without the static binary option. The binary shrank from 1.2MB to 1.0MB. With musl libc the reduction is modest because musl is compact. With glibc the difference would be more dramatic.
Part J — The web interface
J.1 Starting the HTTP server
BusyBox includes a minimal HTTP server. Copy the provided web files to the nfsroot:
cp -r /home/tontsa/embedded-linux-rpi4/data/www \
/home/tontsa/embedded-linux-rpi4/nfsroot/www
Start the server on the target:
/usr/sbin/httpd -h /www/
It backgrounds itself immediately. Opening http://192.168.0.100/index.html in the host browser loads a page served directly from the RPi4. The page shows device information links, Processor and Uptime, implemented as CGI shell scripts that read from /proc and return live system data on each request.
This is the essence of embedded web interfaces: a few shell scripts, a single-binary HTTP server, and the kernel's own virtual filesystem providing all the data. No framework, no runtime, no dependencies beyond BusyBox itself.
J.2 Adding httpd to the startup script
Update /etc/init.d/rcS so the web server starts automatically on every boot:
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev
/usr/sbin/httpd -h /www/
Part K — Initramfs: embedding the rootfs inside the kernel
K.1 How initramfs works
The final exercise takes rootfs portability to its logical conclusion: embedding the entire nfsroot directory directly inside the kernel image. The result is a single Image.gz file that contains both the kernel and the complete userspace. No NFS, no SD card partition 2, no network. Just one file.
When the kernel starts, before it tries to mount any block device or network filesystem, it checks whether an initramfs is embedded in the image. If one is present, it extracts it into a RAM-backed tmpfs and runs /init from there. The root= parameter in bootargs is ignored entirely.
Normal NFS boot:
kernel starts → brings up eth0 → mounts NFS → runs /sbin/init
Initramfs boot:
kernel starts → extracts embedded rootfs into RAM → runs /init
K.2 Preparation
Two things are needed before embedding. First, the kernel looks for /init at the root of the initramfs. BusyBox init lives at /sbin/init, so a symlink is needed:
cd /home/tontsa/embedded-linux-rpi4/nfsroot
ln -s sbin/init init
Second, when booting from initramfs the kernel cannot auto-mount devtmpfs. The rcS script must mount it explicitly, which was already added in Part J.
K.3 Kernel configuration
In menuconfig, navigate to General setup and find Initramfs source file(s). Set it to the full path of the nfsroot directory:
/home/tontsa/embedded-linux-rpi4/nfsroot
The kernel build system walks that directory, packages every file into a cpio archive, compresses it, and appends it to the kernel image. The resulting Image.gz is larger than before, since it now contains the entire userspace, but it is completely self-contained.
K.4 Rebuild and deploy
make -j$(nproc) Image.gz
cp arch/arm64/boot/Image.gz /srv/tftp/
On the next boot, U-Boot fetches the new kernel via TFTP and the board boots to a shell with no NFS mount, no network dependency, and no host VM involvement beyond serving the TFTP file. The web server starts automatically, /proc and /sys are mounted, and the system is fully operational.
A system that boots without intervention
At the end of this lab, the RPi4 boots to a fully functional Linux system without any human interaction. The sequence from power-on to running shell takes about 25 seconds and requires nothing from the operator:
- GPU ROM loads U-Boot from the FAT partition
- U-Boot counts down 2 seconds, runs
bootcmdautomatically bootcmdfetchesImage.gzandbcm2711-rpi-4-b.dtbfrom the TFTP server- The kernel boots, extracts the embedded initramfs into RAM
- BusyBox init runs
/etc/init.d/rcS /proc,/sys, and/devare mounted- The HTTP server starts in the background
- The shell prompt appears on the serial console
Every piece of this chain was built from source and understood from the inside. The bootloader, the kernel, the userspace utilities, the init system, the web server: nothing was installed from a package or flashed from a pre-built image.
That is the point of the Bootlin lab approach. The goal is not just to have a working system. It is to have a working system where you understand every decision that was made along the way.
What this lab teaches
Every problem in this lab had a common theme: the build system made a quiet change during an interactive configuration session, and the symptom appeared far downstream with no obvious connection to the cause.
BCMGENET=m instead of y produced NFS mount failures. A disabled serial driver produced a silent hang after Starting kernel .... A missing dynamic linker produced a misleading not found error for a file that clearly existed.
The debugging approach that worked every time: read the boot log from the beginning, not just the error at the end. The error is often just a consequence of something that failed twenty lines earlier.
The other lesson: grep CONFIG_ .config is faster than menuconfig when you know what you are looking for. Every unexpected behavior in embedded Linux ultimately traces back to a configuration option.
What was built
✓ TFTP kernel loading with automatic bootcmd
✓ NFS root filesystem for rapid development
✓ BusyBox userspace (static, then shared libraries)
✓ Proper init system with inittab and rcS
✓ Virtual filesystems mounted at boot
✓ Dynamic linking with musl libc
✓ BusyBox web interface with CGI pages
✓ Initramfs: rootfs embedded inside the kernel image
✓ Fully autonomous boot from power-on to shell
The next lab moves the root filesystem onto the SD card as a proper ext4 partition, making the system independent of both NFS and TFTP.
Lab materials adapted from Bootlin's Embedded Linux training (bootlin.com/training/embedded-linux), CC BY-SA 3.0