Jetson Nano – Boot from USB

With the introduction of JetPack 4.5, it is now possible to boot your Jetson Nano from a USB drive! Looky here:

Background

The NVIDIA Jetson Nano Developer Kits (A02, B01, and 2GB) boot and run from a micro-SD card. It is now possible to set the Nano up to boot and run from a USB drive.

There are a couple of reasons to boot from USB. First, it is more reliable over the long term than using the micro SD card. SD cards aren’t really designed to handle the files of an operating system, and tend to be less reliable than USB drives in general.

The second reason is that USB is much faster than the SD card speed, usually between 4 to 10 times depending on the application. This makes things much snappier!

USB drives are available in various flavors. You’ll see them as SSD, which stands for Solid State Drive, HDD, which stands for Hard Disk Drive, and flash drive which are sometimes referred to as thumb drives or USB sticks.

For this application, HDD is the most reliable long term, SSD is an excellent alternative and is faster. HDD tends to be slower than SSD and tends to use more power, but offers more storage for the same price. Some HDD drives can be powered externally. Flash drives aren’t as suitable for this application for much the same reason as SD cards, but some people still use them.

People also like the extra storage that many USB drives afford. This makes it easy to store lots of programs and large amounts of data, such as video files and large data models.

Here are a few drives we’ve had success with (affiliate links):

Samsung T7 500 GB USB SSD: https://amzn.to/3vdp62b

Samsung T5 500 GB USB SSD: https://amzn.to/2PtE7bK​

Western Digital 2TB External Hard Drive: https://amzn.to/3t7A7jH

These are all available in a variety of sizes, some careful hunting will find a drive which meets your needs.

Disk Formatting and Partitions

Here are some terms that you will see when we talk about drives on Linux. Partitions, GPT and Ext4. Sometimes the terms are jumbled together, making them hard to understand.

Physical devices consist of a collection of sectors. These are the physical bits were data are stored. Software creates something known as a Partition, which maps and groups sectors together. In that way, each region of the device can be managed independently.

Partition descriptions are stored in a Partition Table. A physical drive can have one or more partitions. 

Because a  Partition is treated as a separate logical volume, drives with multiple partitions appear as though they are multiple physical drives. 

Partitions are handy for storing things like file systems, swap disks and so on. 

You have heard the term ‘format the drive’ before. When we talk about formatting a drive for Linux, this is a two step process. First we format the drive to define the partition layout, and then we format partitions for a specific use.

Linux uses GPT for the layout of a partition table on a physical storage device using globally unique identifiers, or GUIDs. It is the Linux preferred partitioning system for drives larger than 2TB. This results in a table at a known location which tells the system where partitions are physically located on the drive. In other words, a map!

Once the drive is partitioned, we are ready to format a partition for the Linux file system. Different partitions can have different format types. 

This is how you might setup a dual boot Windows and Linux machine on a PC, for example. One partition for the Windows filesystem, and another for the Linux filesystem.

For our Jetson, we want to format a partition as Ext4.  Ext4 is a format for a journaling file system for Linux. We will put the rootfs of the Jetson on this partition. Rootfs is simply the root file system, we usually think about this as all of the files on the system. 

QSPI-NOR?

In previous versions of JetPack, the system boots from information stored on specific partitions on the SD card. For JetPack 4.5+, this changes. 

On the Jetson module, there is flash memory. Typically you will see this referred to as QSPI-NOR. NOR is a type of flash memory. QSPI stands for Quad Serial Protocol Interface, a highway for transferring data to and from the Jetson Tegra chip to the NOR flash memory.

When JetPack goes through its setup process, it transfers information to boot the system from the SD card to the QSPI-NOR. Consequently, the Jetson can use this information to boot the system without having to have a SD card present.

There are a couple of more things to know. If there is no SD card, then the Jetson will look to the USB drive for a rootfs. There are other devices from which the Jetson will be able to boot if they are setup correctly, such as a NVMe drive. In our case, we will just concern ourselves with the USB drive.

There is a special file in the /boot directory which tells Linux where the root file system is located. We will need to modify this file, named extlinux.conf to point to the new location of the root.

You can follow the instructions from NVIDIA in case you want to create your bootable USB drive on your host PC. Here we are going to create the drive from the Jetson itself.

Install

You will need to do an initial setup of the Jetson with JetPack 4.5+ in order to load updated firmware into the Jetson Module QSPI-NOR flash memory. Follow the ‘Getting Started’ instructions on the JetPack site: https://developer.nvidia.com/embedded/jetpack

The JetPack archives are located here: https://developer.nvidia.com/embedded/jetpack-archive

During the initial setup of L4T 32.5+, the firmware for the Jetson Nano developer kits relocates the boot firmware from the micro SD card to the Jetson module integrated QSPI-NOR flash memory. This also changes the layout of the SD card. This layout is now analagous to the BIOS in a PC.

There is a repository on the JetsonHacksNano account on Github, bootFromUSB which contains convenience scripts to help with this task.

Clone the repository, and switch to that repositories’ directory.

$ git clone https://github.com/jetsonhacksnano/bootFromUSB
$ cd bootFromUSB

Step 1

Prepare a USB drive (preferably USB 3.0+, SSD, HDD) by formatting the disk with the GPT partitioning scheme. This will erase the data that is on the drive, be warned. Then create a partition, and set the format type to Ext4. In the video, we use the ‘Disks’ application. See the video for a walk through. It is easier if you only plug in one USB drive during this procedure. When finished, the disk should show as /dev/sda1 or similar. Note: Make sure that the partition is Ext4, as NTSF will appear to copy correctly but cause issues later on. Typically it is easiest to set the volume label for later use during this process.

Step 2

Copy the application area of the micro SD card to the USB drive. The script copyRootToUSB.sh copies the contents of the entire system micro SD card to the USB drive. Naturally, the USB drive storage should be larger than the micro SD card. Note: Make sure that the USB drive is mounted before running the script. In order to copyRootToUSB:

usage: ./copyRootToUSB.sh [OPTIONS]

  -d | --directory     Directory path to parent of kernel

  -v | --volume_label  Label of Volume to lookup

  -p | --path          Device Path to USB drive (e.g. /dev/sda1)

  -h | --help  This message

In the video, we:

$ ./copyRootToUSB.sh -p /dev/sda1

Step 3

Modify the /boot/extlinux/extlinux.conf file on the USB drive. An entry should be added to point to the new rootfs (typically this is /dev/sda1). There is a sample configuration file: sample-extlinux.conf in the repository.

Modify the /boot/extlinux/extlinux.conf file located on the USB drive. This is in a system protected area, so you will need privileges to change the file, ie ‘sudo gedit’. Make a copy of the ‘PRIMARY’ entry and rename it sdcard.

In the PRIMARY entry change the location of the root to point to the USB drive, ie change ‘root=/dev/mmcblk0p1’ which is the address of the SD card. Provided in this repository is a sample configuration file: sample-extlinux.conf as an example.

While using root=/dev/sda1 in the extlinux.conf works, it can be a good idea to use the PARTUUID of the disk to identify the disk location. Because USB devices are not guaranteed to enumerate in the same order every time, it is possible that that /dev/sda1 points to a different device. This may happen if an extra flash drive is plugged into the Jetson along with the USB boot drive, for example.

The UUID of the disk in the GPT partition table is called the PARTUUID. This is a low level descriptor. Note that there is another identifier, referred to as UUID, which is given by the Linux file system. Use the PARTUUID for this application, as UUID has been reported to cause issues at the present time in this use case.

There is a convenience file: partUUID.sh which will determine the PARTUUID of a given device. This is useful in determining the PARTUUID of the USB drive. Note: If the PARTUUID returned is not similar in length to the sample-extlinux.conf example (32a76e0a-9aa7-4744-9954-dfe6f353c6a7), then it is likely that the device is not formatted correctly.

$ ./partUUID.sh

While this defaults to sda1 (/dev/sda1), you can also determine other drive PARTUUIDs. The /dev/ is assumed, use the -d flag. For example:

$ ./partUUID.sh -d sdb1

After saving the extlinux.conf file, you are ready to test things out. Shutdown the Jetson, and remove the SD card. Boot the Jetson, and the Jetson will then boot from the USB drive. There will be a slight pause as it looks for which drive to boot from. Make sure that your USB drive is plugged in when you attempt to boot the machine, as this is not hot swappable.

If you encounter issues, you can always boot from the SD card again.

Then you are all set! Enjoy.

Notes

Once you flash your Jetson with JetPack 4.5+, the QSPI-NOR changes and may not properly boot a disk that was created using JetPack 4.4 and lower. In order to restore the QSPI-NOR you will have to run the SDK Manager from a pre-JetPack 4.5 release on a host PC. The SDK Manager will refresh the QSPI-NOR for the earlier version, and should be able to work with the earlier card.

There is an issue with JetPack 4.5 for which you will need a workaround. This was fixed in JetPack 4.5.1. The workaround is in the releases of the bootFromUSB repository, but you should use JetPack 4.5.1 if possible.

Release Notes

March, 2021

  • JetPack 4.5.1
  • L4T 32.5.1
  • Samsung T5 250GB
  • Tested on Jetson Nano

13 Comments

  1. Thank you Jim, and Happy St Patrick’s Day!

    Ive been waiting for this!

    Boot from USB worked a treat only problem is that the jetson cant see the SD card like at the end of the video. Can you advise me on what I could do to fix this?
    It doesn’t even show up in the disks application

  2. Hi Jim. I followed your tutorial to boot from my SSD and it worked quite well. Just one comment and a question for you:

    1) Comment: I had to power my SSD externally because the time between power being delivered to USB was sometimes too long, so that the Jetson either reverted back to SD Card or got stuck in the boot sequence. It may therefore be beneficial to mention this in your tutorial.

    2) Question:After moving my root to SSD, jetson-io doesn’t work anymore. I get the error “findfs partlabel=app could not resolve”. I understand this is because my SSD partition is not called APP. If you have an idea how to easily fix this, would be great.

    Thank you for the great work!

    Cheers
    Florian

    • I am glad you found the article useful.
      You can set the name of the partition using the Disks application. Select the partition that you would like to name (partition 1 in our case). Select the gear icon, and select ‘Edit Filesystem…’ from the menu. You will then be prompted to change the name of the partition, change it to APP. Select the ‘Change’ button to save the change.

      As an alternative, you can use the CLI parted command. For example, open a Terminal. If your partition is /dev/sda1 as shown in the video, then:

      $ sudo parted /dev/sda name 1 APP
      ## Verify
      $ blkid /dev/sda1

      where 1 represents the partition number. The blkid command should list /dev/sda1 with the name APP.
      Thanks for reading!

  3. Nvidia docs seem to indicate I can do this with my TX1. Since I can’t remove the eMMC do I just delete the /boot/extlinux/extlinux.conf file from the eMMC device to get it to continue searching for a boot device and finding the USB next in sequence?

Leave a Reply

Your email address will not be published.


*