Jetson Nano – Run on USB Drive

Using a USB Drive as the root file system on a Jetson Nano Developer Kit provides many advantages, including faster disk access and more storage. Looky here:

Background

For external storage, the Jetson Nano uses a Micro SD Card. The SD card holds the operating system, applications and any data in use. In the overall scheme of things, this device provides relatively slow access. Also, even though Micro SD cards are much better now than they have been in the past, the cards have a reputation of low reliability in long term or heavy use.

Most desktop/laptop computers use a different type of external storage, such as a Hard Disk Drive (HDD) or Solid Sstate Disk (SSD). Typically these are built into the computer, though you can add an external one also.

In fact, we can add one of those types of drives to the Jetson Nano through the USB 3.0 port! We will cover setting up our USB drive so that it can act as the “root file system” for the Nano. Conceptually, this means that the Nano is running off of the USB drive.

Typically most larger computers boot directly to the disk drive. However, this is not possible using the current configuration boot loader on the Jetson Nano. The boot loader understands USB 2.0, the Nano speaks USB 3.0.

Remember that the term ‘boot’ is shorthand for the slang term ‘bootstrapping’, that is, pulling ones self up by their own boot straps. The Jetson uses a two step boot process. The basic idea is that the boot loader loads a memory image with minimal support for key attached peripherals, and then loads over itself with the Linux kernel specified. It’s really clever and quite tricky.

The solution here is to tell the boot loader to use the USB drive as the new root file system. But in order to be able to do that, we have to embed the USB 3 driver directly into the Linux kernel itself (the kernel file is called ‘Image’). Therefore we end up building a new kernel to include the USB 3 driver, and replacing the old one

Installation

Note: Before we get started, a major shout out goes out to Syonyk’s Project Blog for deciphering how to place the Tegra USB firmware in to the kernel. A very nice write up is available there. Thank you!

USB Drives

In the video, we took a look at the following drives:

  • Samsung T5 500 GB USB SSD: https://amzn.to/2PtE7bK
  • Samsung 860 Pro 256GB SATA III: https://amzn.to/2UEWQCb
  • SATA SSD to USB cable: https://amzn.to/2UEB1md
  • Western Digital 2TB External Hard Drive: https://amzn.to/2GH50WY

Install

You should do this on a freshly flashed Micro SD card. On the JetsonHacksNano account on Github, there is a repository named rootOnUSB. Clone the repository, and then switch to the repositories directory:

$ git clone https://github.com/JetsonHacksNano/rootOnUSB

$ cd rootOnUSB

You are now ready to build the kernel:

$ ./buildKernel.sh

It takes ~ 45 minutes to download and build the kernel. You should reboot the machine to make sure the Image build is correct. You are now ready to copy the root file system from the Micro SD card to the USB drive.

First, prepare a USB drive by formatting it Ext4 with at least one partition. In the video, we assign a volume label. If you need instructions for formatting, check out the video where we do so using the ‘Disks’ application.

Make sure that the USB drive is mounted (you can open a file browser on it to make sure it is mounted). To copy the root:

$ ./copyRootToUSB.sh -v <Volume Label>

OR

$ ./copyRootToUSB.sh -d <Directory Path>

An example of the first and the latter:

$ ./copyRootToUSB.sh -v NanoSSD500

OR

$ ./copyRootToUSB.sh -d /media/jetsonhacks/NanoSSD500

It will take several minutes to copy over all the files. Note: Removable media like USB drives are typically in the /media/<user name> directory.

The final step is to modify /boot/extlinux/extlinux.conf to tell the Nano to switch over to the USB drive. Make a backup of the file, and then duplicate the primary entry. Rename the label of the original entry, and then set the primary entry to point to the USB device. Typically the USB drive will be /dev/sda1 (1 is the partition). The important line needs to be:

APPEND ${cbootargs} rootfstype=ext4 root=/dev/sda1 rw rootwait

There is a file named sample-extlinux.conf which is a sample of the a modified extlinux.conf file might look like. Notice that the label on the second entry is emmc, which represents the Micro SD device address. If you use the serial console for debugging, you will see two entries that you can boot, “primary” and “emmc”. The first is the USB drive, the second the Micro SD card. This can come in handy if the machine decides not to boot after you make your changes.

Benchmarks

If you use the Micro SD card, you typically get average read rates ~ 87 MB/s and access times of ~ 0.52 msec. With a SSD, you get average read rates of ~367 MB/s and access times of 0.29 msec. About a 4X speedup! A HDD drive performs about the same as the Micro SSD, but tends to be much more reliable over time.

Notes

  • Even though the Jetson Nano has 4 USB 3.0 connectors, they all go through one USB hub. Therefore, you end up sharing the USB bandwidth with all the other USB peripherals. Bandwidth can fill up quick!
  • If you use swap memory on a SSD best practice is to use a swap file, not a swap partition. The advantage over a swap partition is that the swap file will move around the disk as it is written and deleted. That way the wear leveling algorithms can help manage the loads. A partition is always in the same place.
  • Tested on Jetson Nano, L4T 32.0.1 [JetPack 4.2]

12 Comments

  1. Very interesting, thank you for this article!

    Is it possible to do the same with high speed usb flash drive instead of SSD? Would there be any speed advantages over SD card?

    • It is faster. It is not recommended to use a flash drive for a long period of time, as they do not have the needed wear leveling algorithms aboard. They exhibit much the same issues as SD cards under extended use. Thanks for reading!

  2. Hi. Thank you for sharing this tutorial first. I just followed the instruction and there is an issue comes up. When I tried to build the Kernel. it stop at “Firmware blobs root directory (extra_firmware_dir) [firmware] (NEW)”. Should I type some dir in there? Thank you!

  3. Hi. is there a minimum requirement for space of the SD card which is used for building kernel? Is 16G SD card ok for this?

  4. You may want to emphasize more on your statement, “You should do this on a freshly flashed Micro SD card.” After waiting for nearly 2 hours for the compile to finish, I realized that it was waiting for input. The previous kernel was modified and it was waiting for my decision to input or continue. It displayed the change that was made and stop the process. No input question or indicator. Not even a blinking cursor. I hit the enter key and it continued.

    Thanks for the article! Its still chugging along as I write this.

Leave a Reply

Your email address will not be published.


*