Mauro Morales

software developer

Tag: Booting

  • How does a Raspberry Pi5 boot an image?

    When the Raspberry Pi5 is turned on, it will check on which device it is configured to boot. By default, this is the SD card, but you can change it to boot from an NVMe or USB drive while still fallback to SD. In my case, I’m using a USB SSD. Let’s take a look at how the disk is partitioned.

    For this article, I will be making reference to the Ubuntu 24.04 server image because its configuration is easier to understand than the Raspbian one, which uses implicit defaults. I mounted the image as a loop device, hence the dev/loop44 in the examples, but if you burned it to an SSD or SD card, you could get the same results from /dev/sdX and /dev/mmcblkY.

    root@zeno:~# lsblk -f /dev/loop44
    NAME       FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
    loop44
    ├─loop44p1 vfat   FAT32 system-boot F526-0340                             419.3M    17% /media/mauro/system-boot
    └─loop44p2 ext4   1.0   writable    1305c13b-200a-49e8-8083-80cd01552617  781.9M    66% /media/mauro/writable

    From the labels, we can assume the system-boot partition will be the one booting the system, but how does the system know this is the case. From the documentation, I was able to find this:

    Partition numbers start at 1 and the MBR partitions are 1 to 4. Specifying partition 0 means boot from the default partition which is the first bootable FAT partition.

    Bootable partitions must be formatted as FAT12, FAT16 or FAT32 and contain a start.elf file (or config.txt file on Raspberry Pi 5) in order to be classed as be bootable by the bootloader

    Looking at the output of the previous command, only the system-boot partition has the right format, so let’s look into that one first.

    # ls -1 /media/mauro/system-boot/
    README
    bcm2710-rpi-2-b.dtb
    bcm2710-rpi-3-b-plus.dtb
    bcm2710-rpi-3-b.dtb
    bcm2710-rpi-cm3.dtb
    bcm2710-rpi-zero-2-w.dtb
    bcm2710-rpi-zero-2.dtb
    bcm2711-rpi-4-b.dtb
    bcm2711-rpi-400.dtb
    bcm2711-rpi-cm4.dtb
    bcm2711-rpi-cm4s.dtb
    bcm2712-rpi-5-b.dtb
    bcm2712-rpi-cm5-cm4io.dtb
    bcm2712-rpi-cm5-cm5io.dtb
    bcm2712d0-rpi-5-b.dtb
    boot.scr
    bootcode.bin
    cmdline.txt
    config.txt
    fixup.dat
    fixup4.dat
    fixup4cd.dat
    fixup4db.dat
    fixup4x.dat
    fixup_cd.dat
    fixup_db.dat
    fixup_x.dat
    hat_map.dtb
    initrd.img
    meta-data
    network-config
    overlays
    start.elf
    start4.elf
    start4cd.elf
    start4db.elf
    start4x.elf
    start_cd.elf
    start_db.elf
    start_x.elf
    uboot_rpi_3.bin
    uboot_rpi_4.bin
    uboot_rpi_arm64.bin
    user-data
    vmlinuz
    

    We can see that the expected config.txt. Let’s take a look at its contents.

    root@zeno:~# cat /media/mauro/system-boot/config.txt
    [all]
    kernel=vmlinuz
    cmdline=cmdline.txt
    initramfs initrd.img followkernel
    
    [pi4]
    max_framebuffers=2
    arm_boost=1
    
    [all]
    # Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
    # parameters related to the base device-tree they must appear *before* any
    # other dtoverlay= specification
    dtparam=audio=on
    dtparam=i2c_arm=on
    dtparam=spi=on
    
    # Comment out the following line if the edges of the desktop appear outside
    # the edges of your display
    disable_overscan=1
    
    # If you have issues with audio, you may try uncommenting the following line
    # which forces the HDMI output into HDMI mode instead of DVI (which doesn't
    # support audio output)
    #hdmi_drive=2
    
    # Enable the serial pins
    enable_uart=1
    
    # Autoload overlays for any recognized cameras or displays that are attached
    # to the CSI/DSI ports. Please note this is for libcamera support, *not* for
    # the legacy camera stack
    camera_auto_detect=1
    display_auto_detect=1
    
    # Config settings specific to arm64
    arm_64bit=1
    dtoverlay=dwc2
    
    # Enable the KMS ("full" KMS) graphics overlay, leaving GPU memory as the
    # default (the kernel is in control of graphics memory with full KMS)
    dtoverlay=vc4-kms-v3d
    disable_fw_kms_setup=1
    
    [pi3+]
    # Use a smaller contiguous memory area, specifically on the 3A+ to avoid an
    # OOM oops on boot. The 3B+ is also affected by this section, but it shouldn't
    # cause any issues on that board
    dtoverlay=vc4-kms-v3d,cma-128
    
    [pi02]
    # The Zero 2W is another 512MB board which is occasionally affected by the same
    # OOM oops on boot.
    dtoverlay=vc4-kms-v3d,cma-128
    
    [all]
    
    [cm4]
    # Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
    # such a board)
    dtoverlay=dwc2,dr_mode=host
    
    [all]

    I’m only interested in the first 5 lines.

    • [all]: makes reference to the board, or in this case, all boards
    • kernel: defines the kernel file to be read, in this case vmlinuz which was present on the files list
    • cmdline: defines the file with the cmdline used to boot the kernel. In this case cmdline.txt which is also there
    • initramfs: defines the initrd file to be loaded, in this case initrd.img, also there. And the followkernel stanza loads the initrd file in memory right after the kernel. Pay attention that this instruction, different from all others, doesn’t use the assignment =.

    Now we can take a look into cmdline.txt

    # cat /media/mauro/system-boot/cmdline.txt
    console=serial0,115200 multipath=off dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc
    

    This tells us that the root of the system is a partition with the label writable. This matches with the output of our very first command. Listing everything in writable we find:

    root@zeno:~# ls -1 /media/mauro/writable/
    bin
    bin.usr-is-merged
    boot
    dev
    etc
    home
    lib
    lib.usr-is-merged
    lost+found
    media
    mnt
    opt
    proc
    root
    run
    sbin
    sbin.usr-is-merged
    snap
    srv
    sys
    tmp
    usr
    var

    This looks like a common root directory for an Ubuntu system, so I will not go deeper into it.

    From a Linux installation on a PC the bootloader, but it turns out that for the Pi 5 it is already part of the EEPROM. So we can trust that it’s present and following the instructions from config.txt.

    An important part of this process is the Device Tree (.dtb files), which is also read by the bootloader. The Device Tree describes the hardware present on the board, ensuring that the kernel knows how to interact with all connected peripherals.

    To summarize it all. When the Raspberry Pi 5 powers up, the EEPROM will look for the first bootable partition, where it will read the config.txt file. The configuration file tells the bootloader which kernel, initramfs and cmdline params to load. After that, it’s the kernel’s job to decide how to proceed, in this case after the kernel and initramfs are running in memory, it will pivot to the system living in the writable partition. Last and out of the scope of this article, the init system will finalize the system.

    Raspbian

    Keep in mind that the Raspbian image doesn’t define all these details, since it uses defaults:

    The Raspberry Pi 5 firmware defaults to loading kernel_2712.img because this image contains optimizations specific to Raspberry Pi 5 (e.g. 16K page-size). If this file is not present, then the common 64-bit kernel (kernel8.img) will be loaded instead.

    And I assume that for initramfs, if not defined, it will also look within the directory and load either initramfs8 or initramfs_2712 by default since those files are present in the raspbian image.