Jetson Nano – Run From USB Drive

Here’s a fast way to set up your Jetson Nano to run from a USB drive. Looky here:

Background

For external storage, the Jetson Nano uses a Micro SD Card. The SD card holds the operating system, applications and any data in use. In the overall scheme of things, this device provides relatively slow access. Also, even though Micro SD cards are much better now than they have been in the past, the cards have a reputation of low reliability in long term or heavy use.

Most desktop/laptop computers use a different type of external storage, such as a Hard Disk Drive (HDD) or Solid Sstate Disk (SSD). Typically these are built into the computer, though you can add an external one also.

In fact, we can add one of those types of drives to the Jetson Nano through the USB 3.0 port! We will cover setting up our USB drive so that it can be the “root file system” or rootfs for the Nano. The end result is that the system will be much snappier in response because the disk access is much faster.

Typically most larger computers boot directly to the disk drive. However, this is not possible using the current configuration of the boot loader on the Jetson Nano.

Remember that the term ‘boot’ is shorthand for the slang term ‘bootstrapping’, that is, pulling ones self up by their own boot straps. The Jetson uses a two step boot process. The basic idea is that the boot loader loads a memory image with minimal support for key attached peripherals, and then loads over itself with the Linux kernel specified. It’s really clever and quite tricky.

By default, the bootloader loads the Linux kernel from the SD card and then configures the rootfs to point to files on the SD card. Here we will load the Linux kernel from the SD card, but we will Pivot the Root to point the rootfs to files on the USB drive.

Why is this difficult?

During the boot process, the Jetson bootloaders use an initrd (initial ramdisk) to load a temporary file system into memory to load the Linux kernel. The initrd is a minimal file system, with just enough for loading the kernel. What we would like to do is load the kernel, and then switch over to the USB drive.

Here’s the problem. While the actual USB drivers are built in to the Linux kernel, the firmware for the USB devices themselves are not. By the time the root file system gets mounted in the boot process, the USB controller is not yet initialized. You can’t talk to the USB drive, so the system falls back to the SD card as the rootfs. The system then loads the USB firmware to make the USB drive available.

In a previous article, we recompiled the Linux kernel to include the USB firmware to circumvent this issue. That is a fairly lengthy process that takes ~45 minutes. However, one of our readers (a special thanks to George!) pointed out a much easier way in a post they wrote on the Jetson Nano Developer Forum!

The idea is to include the USB firmware in the initrd so that the USB drive is available early on in the boot cycle. It takes less than a minute to build a new initrd with the change, and we’re off and running.

Note: In the video, we refer to the initramfs. As in most things technical, the initrd and initramfs are not strictly equivalent, but for the purposes of this discussion we treat them the same.

Install

You should do this on a freshly flashed Micro SD card. You can use a 16GB card for this process. In the video, we use a Samsung T5 500 GB USB SSD. On the JetsonHacksNano account on Github, there is a repository named rootOnUSB. Clone the repository, and then switch to the repositories directory:

$ git clone https://github.com/JetsonHacksNano/rootOnUSB

$ cd rootOnUSB

This is a 4 step process.

Step 1

Build the initramfs with USB support, so that USB is available early in the boot process. A convenience script named addUSBToInitramfs.sh provides this functionality.

$ ./addUSBToInitramfs.sh

Step 2

Prepare a USB drive (preferably USB 3.0, SSD, HDD, or SATA->USB) by formatting the disk as ext4 with a partition. In the video, we use the ‘Disks’ application. It is easier if you only plug in one USB drive during this procedure. When finished, the disk should show as /dev/sda1 or similar. Note: Make sure that the partition is ext4, as NTSF will appear to copy correctly but cause issues later on. Typically it is easiest to set the volume label for later use during this process.

Step 3

Copy the application area of the micro SD card to the USB drive. copyRootToUSB.sh copies the contents of the entire system micro SD card to the USB drive. Naturally, the USB drive storage should be larger than the micro SD card. Note: Make sure that the USB drive is mounted before running the script. In order to copyRootToUSB:

usage: ./copyRootToUSB.sh [OPTIONS]

  -d | --directory     Directory path to parent of kernel

  -v | --volume_label  Label of Volume to lookup

  -p | --path          Device Path to USB drive (e.g. /dev/sda1)

  -h | --help  This message

In the video, we:

$ ./copyRootToUSB.sh -p /dev/sda1

Remember to mount the USB drive before attempting to copy.

Step 4

Modify the /boot/extlinux/extlinux.conf file. An entry should be added to point to the new rootfs (typically this is /dev/sda1). There is a sample configuration file: sample-extlinux.conf

You should make a backup of the original extlinux.conf file. Also, when you edit the file you should make a backup of the original configuration and relabel the backup. This will allow you to access an alternate boot method from the serial console in case something goes sideways.

Then you should changed the INITRD line to:

INITRD /boot/initrd-xusb.img

So that the system uses the initramfs that we built that includes the USB firmware. Then set the root to the USB drive.

Here are some examples. You can set the drive by the UUID of the disk drive, the volume label of the drive, or the device path:

APPEND ${cbootargs} root=UUID=0e437280-bea0-42a2-967f-a240dd3075eb rootwait rootfstype=ext4
APPEND ${cbootargs} root=LABEL=JetsonNanoSSD500 rootwait rootfstype=ext4
APPEND ${cbootargs} root=/dev/sda1 rootwait rootfstype=ext4

The first entry is most specific, the last most generic. Note that you are not guaranteed that a USB device is enumerated in a certain order and will always have the same device path. That is, if you leave another USB drive plugged in along with your root disk when you boot the Jetson, the root disk may have a different path than originally, such as /dev/sdb1.

Also, there is a convenience file: diskUUID.sh which will determine the UUID of a given device. This is useful for example to determine the UUID of the USB drive. Note: If the UUID returned is not similar in length to the above example, then it is likely that the device is not formatted as ext4.

$ ./diskUUID.sh

While this defaults to sda1 (/dev/sda1), you can also determine other drive UUIDs. The /dev/ is assumed, use the -d flag. For example:

$ ./diskUUID.sh -d sdb1

You may find this information useful for setting up the extlinux.conf file

Reboot and Enjoy!

After editing the /boot/extlinux/extlinux.conf file, you are ready to reboot the system. Remember that the system boots from the micro SD card, so it must be present. The extlinux.conf that the system reads during the boot procedure is on the SD card. Even though there is an extlinux.conf file on the USB drive, the one on the SD card is the one that is read during the boot process. Modifying the one on the USB drive will have not effect.

Benchmarks

If you use the Micro SD card, you typically get average read rates ~ 87 MB/s and access times of ~ 0.52 msec. With a SSD, you get average read rates of ~367 MB/s and access times of 0.29 msec. About a 4X speedup! A HDD drive performs about the same as the Micro SSD, but tends to be much more reliable over time.

Special Thanks

Thank you George for pointing this out!

Notes

  • Even though the Jetson Nano has 4 USB 3.0 connectors, they all go through one USB hub. Therefore, you end up sharing the USB bandwidth with all the other USB peripherals. Bandwidth can fill up quick!
  • Mostly recently tested on Jetson Nano, L4T 32.2.1 [JetPack 4.2.2]
  • In the video, tested on Jetson Nano, L4T 32.2.1 [JetPack 4.2.2]

5 Comments

  1. In attempting to use the addUSBToInitramfs.sh script, on a fresh Ubuntu 18.04 minimal install, I’m experiencing the following:

    Adding USB to initramfs
    Warning: couldn’t identify filesystem type for fsck hook, ignoring.
    I: The initramfs will attempt to resume from /dev/zram3
    I: (UUID=3bfa4a08-9aa0-4f47-9811-a18035593b13)
    I: Set the RESUME variable to override this.
    /sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_EGL.conf: No such file or directory
    /sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_GL.conf: No such file or directory

    Which is strange since both of the files exist, although they were initially:
    /etc/ld.so.conf.d/aarch64-linux-gnu-[GL|EGL].conf

    So both files existed, but the script used an underscore rather than a hypen, so I ‘ln -s’ the two files swapping the _ for a – and they’re still not being found. Strange.

    Any thoughts?

    Thanks – Jeff

  2. What happens if you run a apt-get update, apt-get upgrade or apt-get dist-upgrade after doing this USB mod? Does the Micro SD Card get updated, the USB drive, both? Thanks.

    • The drive that is the rootfs is updated. Typically people do not dist-upgrade on the Jetson, as L4T is specific to the machine, and the upgrade from the Ubuntu repository is generic.

Leave a Reply

Your email address will not be published.


*