Jetson Nano – Run From USB Drive

Here’s a fast way to set up your Jetson Nano to run from a USB drive. Looky here:

Background

For external storage, the Jetson Nano uses a Micro SD Card. The SD card holds the operating system, applications and any data in use. In the overall scheme of things, this device provides relatively slow access. Also, even though Micro SD cards are much better now than they have been in the past, the cards have a reputation of low reliability in long term or heavy use.

Most desktop/laptop computers use a different type of external storage, such as a Hard Disk Drive (HDD) or Solid Sstate Disk (SSD). Typically these are built into the computer, though you can add an external one also.

In fact, we can add one of those types of drives to the Jetson Nano through the USB 3.0 port! We will cover setting up our USB drive so that it can be the “root file system” or rootfs for the Nano. The end result is that the system will be much snappier in response because the disk access is much faster.

Typically most larger computers boot directly to the disk drive. However, this is not possible using the current configuration of the boot loader on the Jetson Nano.

Remember that the term ‘boot’ is shorthand for the slang term ‘bootstrapping’, that is, pulling ones self up by their own boot straps. The Jetson uses a two step boot process. The basic idea is that the boot loader loads a memory image with minimal support for key attached peripherals, and then loads over itself with the Linux kernel specified. It’s really clever and quite tricky.

By default, the bootloader loads the Linux kernel from the SD card and then configures the rootfs to point to files on the SD card. Here we will load the Linux kernel from the SD card, but we will Pivot the Root to point the rootfs to files on the USB drive.

Why is this difficult?

During the boot process, the Jetson bootloaders use an initrd (initial ramdisk) to load a temporary file system into memory to load the Linux kernel. The initrd is a minimal file system, with just enough for loading the kernel. What we would like to do is load the kernel, and then switch over to the USB drive.

Here’s the problem. While the actual USB drivers are built in to the Linux kernel, the firmware for the USB devices themselves are not. By the time the root file system gets mounted in the boot process, the USB controller is not yet initialized. You can’t talk to the USB drive, so the system falls back to the SD card as the rootfs. The system then loads the USB firmware to make the USB drive available.

In a previous article, we recompiled the Linux kernel to include the USB firmware to circumvent this issue. That is a fairly lengthy process that takes ~45 minutes. However, one of our readers (a special thanks to George!) pointed out a much easier way in a post they wrote on the Jetson Nano Developer Forum!

The idea is to include the USB firmware in the initrd so that the USB drive is available early on in the boot cycle. It takes less than a minute to build a new initrd with the change, and we’re off and running.

Note: In the video, we refer to the initramfs. As in most things technical, the initrd and initramfs are not strictly equivalent, but for the purposes of this discussion we treat them the same.

Install

You should do this on a freshly flashed Micro SD card. You can use a 16GB card for this process. In the video, we use a Samsung T5 500 GB USB SSD. On the JetsonHacksNano account on Github, there is a repository named rootOnUSB. Clone the repository, and then switch to the repositories directory:

$ git clone https://github.com/JetsonHacksNano/rootOnUSB

$ cd rootOnUSB

This is a 4 step process.

Step 1

Build the initramfs with USB support, so that USB is available early in the boot process. A convenience script named addUSBToInitramfs.sh provides this functionality.

$ ./addUSBToInitramfs.sh

Step 2

Prepare a USB drive (preferably USB 3.0, SSD, HDD, or SATA->USB) by formatting the disk as ext4 with a partition. In the video, we use the ‘Disks’ application. It is easier if you only plug in one USB drive during this procedure. When finished, the disk should show as /dev/sda1 or similar. Note: Make sure that the partition is ext4, as NTSF will appear to copy correctly but cause issues later on. Typically it is easiest to set the volume label for later use during this process.

Step 3

Copy the application area of the micro SD card to the USB drive. copyRootToUSB.sh copies the contents of the entire system micro SD card to the USB drive. Naturally, the USB drive storage should be larger than the micro SD card. Note: Make sure that the USB drive is mounted before running the script. In order to copyRootToUSB:

usage: ./copyRootToUSB.sh [OPTIONS]

  -d | --directory     Directory path to parent of kernel

  -v | --volume_label  Label of Volume to lookup

  -p | --path          Device Path to USB drive (e.g. /dev/sda1)

  -h | --help  This message

In the video, we:

$ ./copyRootToUSB.sh -p /dev/sda1

Remember to mount the USB drive before attempting to copy.

Step 4

Modify the /boot/extlinux/extlinux.conf file. An entry should be added to point to the new rootfs (typically this is /dev/sda1). There is a sample configuration file: sample-extlinux.conf

You should make a backup of the original extlinux.conf file. Also, when you edit the file you should make a backup of the original configuration and relabel the backup. This will allow you to access an alternate boot method from the serial console in case something goes sideways.

Then you should changed the INITRD line to:

INITRD /boot/initrd-xusb.img

So that the system uses the initramfs that we built that includes the USB firmware. Then set the root to the USB drive.

Here are some examples. You can set the drive by the UUID of the disk drive, the volume label of the drive, or the device path:

APPEND ${cbootargs} root=UUID=0e437280-bea0-42a2-967f-a240dd3075eb rootwait rootfstype=ext4
APPEND ${cbootargs} root=LABEL=JetsonNanoSSD500 rootwait rootfstype=ext4
APPEND ${cbootargs} root=/dev/sda1 rootwait rootfstype=ext4

The first entry is most specific, the last most generic. Note that you are not guaranteed that a USB device is enumerated in a certain order and will always have the same device path. That is, if you leave another USB drive plugged in along with your root disk when you boot the Jetson, the root disk may have a different path than originally, such as /dev/sdb1.

Also, there is a convenience file: diskUUID.sh which will determine the UUID of a given device. This is useful for example to determine the UUID of the USB drive. Note: If the UUID returned is not similar in length to the above example, then it is likely that the device is not formatted as ext4.

$ ./diskUUID.sh

While this defaults to sda1 (/dev/sda1), you can also determine other drive UUIDs. The /dev/ is assumed, use the -d flag. For example:

$ ./diskUUID.sh -d sdb1

You may find this information useful for setting up the extlinux.conf file

Reboot and Enjoy!

After editing the /boot/extlinux/extlinux.conf file, you are ready to reboot the system. Remember that the system boots from the micro SD card, so it must be present. The extlinux.conf that the system reads during the boot procedure is on the SD card. Even though there is an extlinux.conf file on the USB drive, the one on the SD card is the one that is read during the boot process. Modifying the one on the USB drive will have not effect.

Benchmarks

If you use the Micro SD card, you typically get average read rates ~ 87 MB/s and access times of ~ 0.52 msec. With a SSD, you get average read rates of ~367 MB/s and access times of 0.29 msec. About a 4X speedup! A HDD drive performs about the same as the Micro SSD, but tends to be much more reliable over time.

Special Thanks

Thank you George for pointing this out!

Notes

  • Even though the Jetson Nano has 4 USB 3.0 connectors, they all go through one USB hub. Therefore, you end up sharing the USB bandwidth with all the other USB peripherals. Bandwidth can fill up quick!
  • Mostly recently tested on Jetson Nano, L4T 32.2.1 [JetPack 4.2.2]
  • In the video, tested on Jetson Nano, L4T 32.2.1 [JetPack 4.2.2]

34 Comments

  1. In attempting to use the addUSBToInitramfs.sh script, on a fresh Ubuntu 18.04 minimal install, I’m experiencing the following:

    Adding USB to initramfs
    Warning: couldn’t identify filesystem type for fsck hook, ignoring.
    I: The initramfs will attempt to resume from /dev/zram3
    I: (UUID=3bfa4a08-9aa0-4f47-9811-a18035593b13)
    I: Set the RESUME variable to override this.
    /sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_EGL.conf: No such file or directory
    /sbin/ldconfig.real: Warning: ignoring configuration file that cannot be opened: /etc/ld.so.conf.d/aarch64-linux-gnu_GL.conf: No such file or directory

    Which is strange since both of the files exist, although they were initially:
    /etc/ld.so.conf.d/aarch64-linux-gnu-[GL|EGL].conf

    So both files existed, but the script used an underscore rather than a hypen, so I ‘ln -s’ the two files swapping the _ for a – and they’re still not being found. Strange.

    Any thoughts?

    Thanks – Jeff

  2. What happens if you run a apt-get update, apt-get upgrade or apt-get dist-upgrade after doing this USB mod? Does the Micro SD Card get updated, the USB drive, both? Thanks.

    • The drive that is the rootfs is updated. Typically people do not dist-upgrade on the Jetson, as L4T is specific to the machine, and the upgrade from the Ubuntu repository is generic.

  3. How would I get the OS from the USB drive to use the GPU drivers with acceleration?

    Ideally I would like to use a completely different distribution like Alpine Linux. I was able to boot with Alpine’s mini rootfs which is really amazing. It only uses 10 MB!

    I tried Nvidia’s official sample rootfs and was not able to get acceleration working. How can I fix this? I believe if I get it working on Ubuntu I can use that to get it working on other systems like Arch or Alpine.

  4. Sorry to disturb but i would like to ask if i boot from a external USB device, would i still be able to have access to the Jetson Nano’s GPIO ports?

  5. Hi, I am currently working on a project that requires ubuntu 16.04 to run a SDK. I was wondering if it is possible to boot ubuntu 16.04 on the jetsonano using this method.
    Thanks with regards

  6. Hi, I followed your instruction and succeeded in booting from usb ssd. Thanks a lot.
    Just one question, I noticed that the sd card is still needed for booting. Can I clone the sd card as a backup and use the clone for booting in case the original is corrupted?

    • You do not have to, but I usually do in case I want to add an extra partition later. This can be for a swap partition. However generally I just keep one around in case I want to easily wipe a partition, but have some things saved elsewhere. Thanks for reading!

  7. Hi Jim,
    thanks for this tutorial. I successfully managed to switch root to an usb ssd. But what I have noticed is that on my jetson nano still the usb ssd is displayed as mounted in the ubuntu application launcher and in the files application. In your video first, before switching the root to usb ssd, also your usb ssd was displayed there. And after successfully switching only the sdcard is displayed there and usb ssd disappeared. At my jetson nano now both sdcard and usb ssd are displayed. Do you have a clue why? I have to admit that I’m kinda linux novice.

    • My guess is that you did not update the extlinux.conf file correctly. You must change the extlinux.conf file on the SD card, the one on the USB drive is ignored during the boot process.

      • Hi Jim,
        thanks for your reply. I figured out that R32.2.1 and R32.3.1 behave differently. I only have the issue with R32.3.1. Doing the same steps with R32.2.1 (as you did) works fine.
        In regard to your guess, I did change the extlinux.conf, my home folder is now 920GB large, so it worked, it would say. But the SSD is in the new release also visible in /media, what it wasn’t in the old release.

  8. Hi all, a couple of questions:
    I followed these instructions and all went well, I now have my Nano booting from a USB connected SSD. The thing is that when I plug another device (specifically an rplidar) into another USB port, it is recognized as TTYUSB0 which is already mapped to sda1, the SSD. Of course it’s rejected by the OS. Have I done something wrong or is there some special medicine I need to apply to override the overlap?

    Also, is it safe to remove the SD card after booting from the SSD?

    thanks,
    jeff

    • Do you mean that the USB is plugged in, the Nano is running, and then you plug in the RPLidar it gets assigned to ttyUSB0?
      Not sure why you want to remove the SD card. Did you try it to see what happens?

  9. Yes, here is what I get, first without the rplidar, then with it plugged in….

    jeffib@jeff-nano01:~$ lsusb
    Bus 002 Device 002: ID 0bda:0411 Realtek Semiconductor Corp.
    Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 003: ID 8087:0a2b Intel Corp.
    Bus 001 Device 007: ID 2109:0715 VIA Labs, Inc.
    Bus 001 Device 005: ID 1c4f:0002 SiGma Micro Keyboard TRACER Gamma Ivory
    Bus 001 Device 004: ID 046d:c52b Logitech, Inc. Unifying Receiver
    Bus 001 Device 002: ID 0bda:5411 Realtek Semiconductor Corp.
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    jeffib@jeff-nano01:~$ ls -l /dev | grep ttyUSB

    jeffib@jeff-nano01:~$ lsusb
    Bus 002 Device 002: ID 0bda:0411 Realtek Semiconductor Corp.
    Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 003: ID 8087:0a2b Intel Corp.
    Bus 001 Device 007: ID 2109:0715 VIA Labs, Inc.
    Bus 001 Device 010: ID 10c4:ea60 Cygnal Integrated Products, Inc. CP210x UART Bridge / myAVR mySmartUSB light
    Bus 001 Device 005: ID 1c4f:0002 SiGma Micro Keyboard TRACER Gamma Ivory
    Bus 001 Device 004: ID 046d:c52b Logitech, Inc. Unifying Receiver
    Bus 001 Device 002: ID 0bda:5411 Realtek Semiconductor Corp.
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    jeffib@jeff-nano01:~$ ls -l /dev | grep ttyUSB
    lrwxrwxrwx 1 root root 7 Jan 15 08:20 gps0 -> ttyUSB0
    crw-rw—- 1 root dialout 188, 0 Jan 15 08:20 ttyUSB0
    jeffib@jeff-nano01:~$

    *************

    I have not tried pulling the sd card but I’ll give it a try….

    Thanks,
    Jeff

    • I do not know what you are expecting here.
      You appear to have several different USB devices plugged in. I don’t see an external USB hub.
      A USB drive mounts as /dev/sd* it does not mount as ttyUSB*. Unless you are using a powered USB hub, I would be suspicious of not having enough power to drive the peripherals.
      You would be better off looking at the dmesg log to see what happens when you plug in the lidar. You can use $dmesg –follow to watch.

Leave a Reply

Your email address will not be published.


*