Jetson Nano – Run on USB Drive

NOTE: THIS ARTICLE HAS BEEN UPDATED WITH AN EASIER METHOD. PLEASE SEE: Jetson Nano – Run From USB Drive.

Using a USB Drive as the root file system on a Jetson Nano Developer Kit provides many advantages, including faster disk access and more storage. Looky here:

Background

For external storage, the Jetson Nano uses a Micro SD Card. The SD card holds the operating system, applications and any data in use. In the overall scheme of things, this device provides relatively slow access. Also, even though Micro SD cards are much better now than they have been in the past, the cards have a reputation of low reliability in long term or heavy use.

Most desktop/laptop computers use a different type of external storage, such as a Hard Disk Drive (HDD) or Solid Sstate Disk (SSD). Typically these are built into the computer, though you can add an external one also.

In fact, we can add one of those types of drives to the Jetson Nano through the USB 3.0 port! We will cover setting up our USB drive so that it can act as the “root file system” for the Nano. Conceptually, this means that the Nano is running off of the USB drive.

Typically most larger computers boot directly to the disk drive. However, this is not possible using the current configuration boot loader on the Jetson Nano. The boot loader understands USB 2.0, the Nano speaks USB 3.0.

Remember that the term ‘boot’ is shorthand for the slang term ‘bootstrapping’, that is, pulling ones self up by their own boot straps. The Jetson uses a two step boot process. The basic idea is that the boot loader loads a memory image with minimal support for key attached peripherals, and then loads over itself with the Linux kernel specified. It’s really clever and quite tricky.

The solution here is to tell the boot loader to use the USB drive as the new root file system. But in order to be able to do that, we have to embed the USB 3 driver directly into the Linux kernel itself (the kernel file is called ‘Image’). Therefore we end up building a new kernel to include the USB 3 driver, and replacing the old one

Installation

Note: Before we get started, a major shout out goes out to Syonyk’s Project Blog for deciphering how to place the Tegra USB firmware in to the kernel. A very nice write up is available there. Thank you!

USB Drives

In the video, we took a look at the following drives:

  • Samsung T5 500 GB USB SSD: https://amzn.to/2PtE7bK
  • Samsung 860 Pro 256GB SATA III: https://amzn.to/2UEWQCb
  • SATA SSD to USB cable: https://amzn.to/2UEB1md
  • Western Digital 2TB External Hard Drive: https://amzn.to/2GH50WY

Install

You should do this on a freshly flashed Micro SD card. On the JetsonHacksNano account on Github, there is a repository named rootOnUSB. Clone the repository, and then switch to the repositories directory:

$ git clone https://github.com/JetsonHacksNano/rootOnUSB

$ cd rootOnUSB

You are now ready to build the kernel:

$ ./buildKernel.sh

Note: Depending on which version of L4T you are running, you may have to “git checkout” the appropriate release to match the release you are running.

It takes ~ 45 minutes to download and build the kernel. You should reboot the machine to make sure the Image build is correct. You are now ready to copy the root file system from the Micro SD card to the USB drive.

First, prepare a USB drive by formatting it Ext4 with at least one partition. In the video, we assign a volume label. If you need instructions for formatting, check out the video where we do so using the ‘Disks’ application.

Make sure that the USB drive is mounted (you can open a file browser on it to make sure it is mounted). To copy the root:

$ ./copyRootToUSB.sh -v <Volume Label>

OR

$ ./copyRootToUSB.sh -d <Directory Path>

An example of the first and the latter:

$ ./copyRootToUSB.sh -v NanoSSD500

OR

$ ./copyRootToUSB.sh -d /media/jetsonhacks/NanoSSD500

It will take several minutes to copy over all the files. Note: Removable media like USB drives are typically in the /media/<user name> directory.

The final step is to modify /boot/extlinux/extlinux.conf to tell the Nano to switch over to the USB drive. Make a backup of the file, and then duplicate the primary entry. Rename the label of the original entry, and then set the primary entry to point to the USB device. Typically the USB drive will be /dev/sda1 (1 is the partition). The important line needs to be:

APPEND ${cbootargs} rootfstype=ext4 root=/dev/sda1 rw rootwait

There is a file named sample-extlinux.conf which is a sample of the a modified extlinux.conf file might look like. Notice that the label on the second entry is emmc, which represents the Micro SD device address. If you use the serial console for debugging, you will see two entries that you can boot, “primary” and “emmc”. The first is the USB drive, the second the Micro SD card. This can come in handy if the machine decides not to boot after you make your changes.

Benchmarks

If you use the Micro SD card, you typically get average read rates ~ 87 MB/s and access times of ~ 0.52 msec. With a SSD, you get average read rates of ~367 MB/s and access times of 0.29 msec. About a 4X speedup! A HDD drive performs about the same as the Micro SSD, but tends to be much more reliable over time.

Notes

  • Even though the Jetson Nano has 4 USB 3.0 connectors, they all go through one USB hub. Therefore, you end up sharing the USB bandwidth with all the other USB peripherals. Bandwidth can fill up quick!
  • If you use swap memory on a SSD best practice is to use a swap file, not a swap partition. The advantage over a swap partition is that the swap file will move around the disk as it is written and deleted. That way the wear leveling algorithms can help manage the loads. A partition is always in the same place.
  • Mostly recently tested on Jetson Nano, L4T 32.2 [JetPack 4.2.1]
  • In the video, tested on Jetson Nano, L4T 32.0.1 [JetPack 4.2]

68 Comments

  1. Very interesting, thank you for this article!

    Is it possible to do the same with high speed usb flash drive instead of SSD? Would there be any speed advantages over SD card?

    • It is faster. It is not recommended to use a flash drive for a long period of time, as they do not have the needed wear leveling algorithms aboard. They exhibit much the same issues as SD cards under extended use. Thanks for reading!

  2. Hi. Thank you for sharing this tutorial first. I just followed the instruction and there is an issue comes up. When I tried to build the Kernel. it stop at “Firmware blobs root directory (extra_firmware_dir) [firmware] (NEW)”. Should I type some dir in there? Thank you!

  3. Hi. is there a minimum requirement for space of the SD card which is used for building kernel? Is 16G SD card ok for this?

  4. You may want to emphasize more on your statement, “You should do this on a freshly flashed Micro SD card.” After waiting for nearly 2 hours for the compile to finish, I realized that it was waiting for input. The previous kernel was modified and it was waiting for my decision to input or continue. It displayed the change that was made and stop the process. No input question or indicator. Not even a blinking cursor. I hit the enter key and it continued.

    Thanks for the article! Its still chugging along as I write this.

  5. Once a new kernel is built and booting to SSD is working and one decides to apply other changes to the kernel, what is the process the repeat these steps? The new kernel will be built on the SSD and does one just need to copy the newly created image to the SD card?

    • That’s correct. There are a couple of caveats:
      The new Image needs to be placed onto the SD card /boot directory and
      any modules that you modified/created are on the USB drive. Backup as needed.
      Thanks for reading!

      • We are now at Jetpack 4.2.2 and I made the SSD a while back with Jetpack 4.2.

        I would like to figure out how I can update the SSD to 4.2.2 without needing to reinstall and recompile all the programs I have on the SSD. After updating the SD card to 4.2.2 my SSD will not boot anymore and if I copy the SD card (with 4.2.2) to the SSD I believe it will loose the installed programs or am I mistaken? I have a backup of SD card but I dont have a backup of the SSD to test this.

        Is it correct to say that the USB boot modification does not allow for future firmware upgrades?

  6. Your comment in the notes of “A partition is always in the same place.” is not entirely correct. SSD devices (not USB thumb drives or flash cards) do their own internal wear leveling using over-provisioning. While a partition is logically in the same space as you write to the device the physical location moves around based on the device’s wear leveling algorithm. If you are going to use a SSD I would recommend purchasing an enterprise SSD rather than a consumer SSD as an enterprise SSD contains more over-provisioning, faster speeds, and a longer lifetime.

  7. Hi, i tried running the above but kept hitting –

    rsync: write failed on “/media/user1/myvolumename/usr/bin/lshw”: Read-only file system (30)

    I even tried using sudo but the same error. Any advice?

    • Lookup dmesg any UAS error, my usb3.1 m2 ssd had these troubles too: after copying gigs of data (like the root fs) the ssd issued an UAS error, and was remounted as read-only.
      A usb_storage_qirck=xxxx.yyyy.u in the extlinux boot, and/or in /etc/modprobe.d/ … blacklist
      had solved my problem (or the linux usb3.0 troubles)…

      • @Teik Hooi Beh and others having that problem:
        “Read-only file system” nearly always is an indicator of the target file system having an error. When the error is sensed, the file system in question is re-mounted read-only. As long as the file system is not root, you can unmount it, and check/repair it.
        Assume we are talking about /dev/sda1:
        umount /dev/sda1
        fsck /dev/sda1
        If you have already pivoted to the drive, you can’t simply umount it. The easiest fix is a re-boot. During the re-boot, the system notices an error, and automatically does an fsck. As a last resort, you can plug the drive into another linux system, and do an fsck from there. While doing that, I noticed that the ext4 file system placed onto the ssd by the Jetson was incompatible with a kernel 5.1 on another machine. I reformatted the SSD to the ext4 of the kernel 5.1 machine, and never had a problem since.

        @Marcos: “A usb_storage_qirck=xxxx.yyyy.u in the extlinux boot, and/or in /etc/modprobe.d/ … blacklist
        had solved my problem” ???? – Could you please be a bit more specific?

        • If the culprit are UAS errors in your log, it could be a known problem with usb3.0 on linux, with usb3.x adapter that have problem with UAS, but work flawlessly on usb2.
          The solution is to use: lsusb to find vendor ID and productID, both are 4 digit codes (XXXX:YYYY), these codes are needed to tell the kernel module loader that a specific kernel module for this device is blacklisted.
          In our case we want that the problematic USB adapter does NOT load the UAS kernel module.
          By adding the “usb-storage.quirks=XXXX:YYYY:u” to the /boot/exxlinux/extlinux.conf, the kernel will not load it at the startup.
          For temporary usb drives that will be automounted:
          Edit or create the file /etc/modprobe.d/uas.conf that contain “options usb-storage quirks=XXXX:YYYY:u” , where XXXX:YYYY is your specific USB adapter vendorID, productID.

  8. on the final reboot i get the error
    tegra-xusb 700900000.xusb: cannot find firmware…retry after 1 second
    tegra-xusb 700900000.xusb: Direct firmware load for tegra21x_xusb_firmware failed with error -2
    tegra-xusb 700900000.xusb: Falling back to user helper

    those three lines repeat over time

    • This could be a result of several things. The most likely causes tend to be an incorrect extlinux.conf file, followed by not having the USB drive connected.

    • was able to resolve the problem by starting fresh with a brand new, larger micro sd card. I was using 32GB but the drive was old and very slow and may have not been a good card. After attempting again I noticed a warning about being out of memory and I think I just missed it the first time. The new sd card resolved the problem and was much faster as well. Thanks for these great tutorials!

  9. Thank you for your great instructional and easy to follow videos and all the work you put into them. Single question, after following your instructions and changing the root to the SSD I now want to incorporate swapfile, since the default is /mnt do I need to change the usage instructions to another directory? Thank you in advance

    • I’m glad you find the articles useful. I don’t know how to answer your question. The rootfs is copied over to the USB drive. Everything pretty much stays the same. I don’t know what usage instructions are. Thanks for reading!

  10. Nice recipe, thank you! I have expanded on it a bit to improve on the kerbnel, mainloy to make it work with the CFS firewall.

    Configure configureKernel.sh in scripts as follows:

    #!/bin/bash
    # Copyright (c) 2016-19 Jetsonhacks
    # MIT License
    # Configure kernel to include tegra usb firmware
    # SOURCE_TARGET and KERNEL_RELEASE should be set in caller
    cd “$SOURCE_TARGET”kernel/kernel-$KERNEL_RELEASE
    cp /lib/firmware/tegra21x_xusb_firmware ./firmware/
    # CONFIG_EXTRA_FIRMWARE=”tegra21x_xusb_firmware”
    FIRMWARE_NAME=”tegra21x_xusb_firmware”
    bash scripts/config –file .config \
    –set-val CONFIG_NETFILTER_XT_MATCH_MULTIPORT m \
    –set-val CONFIG_NETFILTER_XT_MATCH_RECENT m \
    –set-val CONFIG_NETFILTER_XT_MATCH_OWNER m \
    –set-val CONFIG_NF_NAT_REDIRECT m \
    –set-val CONFIG_IP_NF_TARGET_REDIRECT m \
    –set-val CONFIG_NETFILTER_XT_TARGET_REDIRECT m \
    –set-val CONFIG_CGROUP_HUGETLB y \
    –set-val CONFIG_CFQ_GROUP_IOSCHED y \
    –set-val CONFIG_INET_ESP m \
    –set-val CONFIG_IP_VS_PROTO_TCP y \
    –set-val CONFIG_IP_VS_PROTO_UDP y \
    –set-val CONFIG_NET_L3_MASTER_DEV y \
    –set-val CONFIG_DM_THIN_PROVISIONING m \
    –set-val CONFIG_DM_BUFIO m \
    –set-val CONFIG_DM_BIO_PRISON m \
    –set-val CONFIG_DM_PERSISTENT_DATA m \
    –set-val CONFIG_IPVLAN m \
    –set-str CONFIG_EXTRA_FIRMWARE “$FIRMWARE_NAME” \
    –set-str CONFIG_EXTRA_FIRMWARE_DIR “firmware”

    Also, add ./makeMudules.sh to the end of buildKernel.sh

    # Make the kernel
    echo “Making kernel”
    cd $BUILD_REPOSITORY
    ./makeKernel.sh
    ./makeModules.sh
    if [ $? -eq 0 ]; then
    # Put a copy of the current Image here
    cp /boot/Image $INSTALL_DIR/Image.orig
    ./copyImage.sh
    echo “New Image created and placed in /boot”
    fi

    That step would be very important for any other changes that affect modules

  11. Also, the first thing I would do is connect a decent 5V/5A power supply via the barrel jack. The Jetson already taxes even the best 2.5A supply connected via micro USB. Connecting an SSD makes the situation worse. If you can’t find a suitable 5A PSU, use an old/retired PC power supply. The soldering is left as an exercise to the student.

  12. Thank you for this great tutorial.
    Unfortunately I got the error when executing ./copyRootToUSB.sh that the I hav a read onlz file sysytem.
    E.g. mkdir “/media/tom/jetsonSsd/var/metrics” failed: Read-only file system (30)

    So can you help me to tell what I am doing wrong?
    Thank you.
    Tom

    • Unless you have set a file system to read only, or mounted it as read-only, “read-only- file system” means that the machine has detected an error of the file system, and has set the respective partition to read-only to protect it from developing more errors.

      If the ssd has the error, you need to to umount the ssd and fsck the file system on the ssd.

      • Thank you for your help.
        I tried as follows:
        sudo ./copyRootToUSB.sh -v jetsonSsd
        Destination Target:
        Device Path: /dev/sda1
        Destination Target: /mnt/jetsonSsd
        Target: /mnt/jetsonSsd
        E: Konnte Sperre /var/lib/dpkg/lock-frontend nicht bekommen – open (11: Resource temporarily unavailable)
        E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
        9,165,199,647 98% 24.71MB/s 0:05:53 (xfr#84184, ir-chk=1110/129090)
        rsync: write failed on “/mnt/jetsonSsd/usr/src/kernel/kernel-4.9/vmlinux.o”: Read-only file system (30)
        rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.2]

        Afterwards I made an fschk and got an OK status.

        When I use NTFS the file system, copy works but execution not…

          • Yes, I just wanted to say that the SSD is working. With NTFS the script execution works, with ext4 file system not. So I am wondering why.

            • You need to create a partition of the ssd, and format it to ext4. It won’t work with ntfs.
              Let’s say the ssd is /dev/sda. Create a /dev/sda1 partition, and format it to ext4. Mount /dev/sda1, and try creating a few files on it. If you succeed, you are good to go. If you don’t succeed, you need to rectify this situation before proceeding.
              Recompiling the kernel and pivoting the root is not for the faint of heart, and it is not an exercise I recommend when learning.

              • Thanks.
                Now I created the ext4 partition again, and now the script copied all. And afterwards bootup worked :-).
                I do not know why it did not work for me at the beginning. I tried it before I created the NTFS around 4 times with ext4. Ok, but now it worked.

                Thanks for your help.
                BR

  13. I goofed this up by botching the modification to /boot/extlinux/extlinux.conf I’d formatted the drive with 2 partitions and so I needed sda2 rather than sda1. What bugs me is that if I plug a USB SD adapter into a USB slot, This can become Sdb2:

    Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 33553920 bytes
    Disklabel type: gpt
    Disk identifier: 88841F98-3EC8-4741-95F4-B36210B18889

    Device Start End Sectors Size Type
    /dev/sda1 40 409639 409600 200M EFI System
    /dev/sda2 411648 976773119 976361472 465.6G Linux filesystem

    vs:

    Disk /dev/sdb: 465.8 GiB, 500107862016 bytes, 976773168 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 33553920 bytes
    Disklabel type: gpt
    Disk identifier: 88841F98-3EC8-4741-95F4-B36210B18889

    Device Start End Sectors Size Type
    /dev/sdb1 40 409639 409600 200M EFI System
    /dev/sdb2 411648 976773119 976361472 465.6G Linux filesystem

    Is there a way to express it by label rather than something that is dependent on what is plugged in??

  14. These instructions do not work with the release of jetpack 4.2. I upgraded to the new jetpack and when i tried to rootOn USB, I got this error

    jetson1@jetson1-desktop:~/rootOnUSB$ ./buildKernel.sh
    Jetson Model: NVIDIA Jetson Nano Developer Kit
    Jetson L4T:
    ==== Jetson Board Mismatch! =============
    Currently this script works for the jetson-nano.
    This processor appears to be a NVIDIA Jetson Nano Developer Kit, which does not have a corresponding script

    Exiting
    jetson1@jetson1-desktop:~/rootOnUSB$

    Anybody have a solution?

    • I forgot to mention that I am working with the “Jetson nano developers kit” and this rootOnUSB worked on it just fine until i upgraded to the new jet pack 4.2 that was released on July 19 2019. And it should be mentioned that this does not work with the new jetpack 4.2 next to the video title.

  15. Hi, the L4T R32.2 has released now, but the script can not work on this release, will it be any update of this script to support the R32.2?

  16. Thank you for your amazing post. I’ve had a problem when i plug my external HDD to my jetson. After i plugged in my external hdd, jetson crash and dead. Before that, i use power source with 5V and 2A. Is that bcs my power source?

    • Thank you for the kind words. The Jetson by itself uses almost 2A. If you add more peripherals, there is not enough power. In earlier articles, we covered adding a 4A, 5V power supply. Thanks for reading!

  17. Easiest way to do :
    1) burn the jetson image on your ssd usb drive
    2) change the /boot/extlinux/exlinux.conf to add rootfstype=ext4 root=/dev/sda1 rw rootwait
    3) do a reboot
    The jetson will use the usb drive instead of the sdcard to boot up.

    Enjoy!

  18. Hi,
    I’d like to know where should I modify the /boot/extlinux/extlinux.conf file ? On the micro SD card or on the SSD Drive ? Also should I keep the micro SD card in the socket while booting with the SSD drive, please ?

    Thanks and best regards,
    Pascal

Leave a Reply

Your email address will not be published.


*