Native Boot for Jetson Xaviers

With the advent of JetPack 4.6 (L4T 32.6.1) it is now possible to boot the Jetson AGX Xavier and Jetson Xavier NX from external storage. Looky here:

Background

In the JetPack 4.6 release Cboot now has NVMe driver support for the Jetson Xavier NX and Jetson AGX Xavier series. This enablesnables loading kernel, kernel-dtb and initrd from the root file system on NVMe.

There is also a new flashing tool which can flash internal or external media connected to the Jetson Xaviers. The new tool uses initial RAM disk for flashing and is up to 1.5x faster when flashing compared to the previous method!

If you have read the highlights from the press release, you already know all of that. Let’s talk about the new flashing tool in a little more detail.

For reference, the JetsonHacks scripts are in the JetsonHacks repository on Github: bootFromExternalStorage

You’ll need a NVMe drive in your Jetson to follow this tutorial. Only Gen3 drives have been tested, there appear to be issues with Gen4. These drives are available in different sizes of course, but 500GB seems like a good price/capacity. Here’s some suggestions (Amazon affiliate links):

This process does not require that the NVIDIA SDK Manager GUI application be present. These are all command line scripts that can be run in a Terminal. For many people, this better fits their workflow.

Flashing with initrd

The new flashing method uses initrd (initial RAM Disk) to flash both internal media and external media connected to a Jetson device. This method uses initrd and USB device mode. Initrd flash supports several workflows such as:

  • Flashing internal storage devices
  • Flashing external storage devices such as NVMe SSD and USB drives
  • Enabling A/B rootfs on external storage devices
  • Enabling disk encryptions on external storage devices
  • Flashing individual partitions
  • Flashing fused Jetson devices
  • Flashing a Massflash blob to normal and fused Jetson devices
  • Generating separate images for external and internal storage devices, then flashing the combined images

Note: Jetson AGX Xavier series devices use boot firmware that is stored only on internal eMMC memory. Consequently this type of device cannot boot from USB or NVMe SSD until its internal eMMC has been flashed.

As usual, tools this flexible have a little bit of a learning curve. There are a group of scripts in the Linux_for_Tegra/tools/kernel_flash directory which handle the above mentioned tasks. It is worth reading the README_initrd_flash.txt file to get a feel for what it takes accomplish the task you are trying to accomplish.

The flashing process will reformat the external drive on the Jetson. Make sure that you backup any data you want to keep.
Before flashing a Jetson Xaver NX Developer Kit, remove the SD card if present.

Preparing the Host

You will need to install dependencies on the x86 host to support flashing the Jetson. NVIDIA officially supports Ubuntu 16.04 and 18.04 for the flashing scripts. Then you will minimally need three archives:

  • Jetson Xavier Board Support Package (BSP)
  • Sample Root File System (rootfs)
  • Secure Boot

Here are Direct downloadable links to JetPack 4.6 and L4T debian packages for Jetson

There are helper scripts in the JetsonHacks bootFromExternalStorage repository. The time for the entire process of downloading and flashing the Jetson will depend on the speed of your Internet connection and host computer. Here it was ~ 1 hour. To download the helper scripts and run them:

$ git clone https://github.com/jetsonhacks/bootFromExternalStorage.git

$ cd bootFromExternalStorage

$ ./install_dependencies.sh

$ ./get_jetson_files.sh

There are a couple of tricks here and there in the scripts to put everything in the right place.

Flashing the Jetson

Once the archives are expanded and put in the correct place, put the Jetson into Force Recovery mode. This is different on each Xavier. The video shows a Xaiver NX. You can use the command lsusb to determine if the host can ‘see’ the Xavier. There should be an entry with the NVidia Corp tag when the Xavier is connected to the host in Force Recovery mode. Default is to flash to the NVMe SSD. You will find this works best if you actually have a NVMe SSD installed (64GB minimum; more is better) before attempting to flash.

You can use the convenience script in the repository to flash the Jetson:

$ flash_jetson_external_storage.sh

It takes ~ 20 minutes to install everything in this manner. After flashing, the Jetson will be in ‘oem-config’ mode, ready to be setup. At this point, we’re done on the host side.

Note on Flashing

The script flash_jetson_external_storage.sh wraps a NVIDIA script which calls the initrd flashing script. After everything is setup, this boils down to:

$ ./nvsdkmanager_flash.sh –storage nvme0n1p1

For a Jetson Xavier NX, this flashes the internal QSPI-NOR memory and puts the rootfs and other partitions on the NVMe SSD. The Jetson AGX Xavier does not have QSPI memory, so the equivalent on eMMC is flashed instead.

Setting up the Jetson

You can choose to setup the Jetson in headless mode or desktop mode. In the video, we chose desktop mode. In either case, the next step is to install the NVIDIA JetPack packages on the Jetson. Switch from the host to the Jetson. Finish the Jetson configuration, then:

$ git clone https://github.com/jetsonhacks/bootFromExternalStorage.git

$ cd bootFromExternalStorage

./install_jetson_default_packages.sh

The default packages are from the meta-package nvidia-jetpack and includes all of our Jetson friends like:

  • CUDA
  • cuDNN
  • TensorRT
  • VisionWorks
  • VPI
  • OpenCV

plus a couple of other of packages to replicate the same experience as the default Xavier NX SD card.

Conclusion

With the convenience scripts, setting up your Jetson Xavier to boot from external storage should be straightforward. This is the NVIDIA ‘approved’ way, so Over The Air (OTA) updates should work going forward too.

Note that NVIDIA is working on the SDK Manager GUI application to replicate this type of functionality. That’s certainly worth checking out!

Notes

The JetPack tools from NVIDIA support Ubuntu 16.04 and 18.04 on the host machine. There are work arounds in these scripts for Ubuntu 20.04. However these are not well tested yet.
This method will work with the Jetson Xavier AGX Developer Kit. However, there appears to be an issue of oem-config not being called on first boot of the Jetson. The work around is to setup a default user. On the host, switch to the Linux_for_Tegra folder and execute:

# this command create username ubuntu and password ubuntu
$ sudo ./tools/l4t_create_default_user.sh -u ubuntu -p ubuntu </blockquote>

Then reflash the Jetson.

Initial Release

  • JetPack 4.6, L4T 32.6.1
  • Tested on Jetson Xavier NX, NVMe SSD
  • Host: Ubuntu 16.04 (should work with Ubuntu 18.04 also)
  • Experimental Ubuntu 20.04 support added

28 Comments

  1. Great news! Just a quick question: does it require a clean NVME? What if I already have data on my NVME drive? Would it be wiped out when I run the flashing script?

  2. Thanks!! Was finally able to get this to work from the latest version of Proxmox VE (Debian Buster). Sadly my only option for an X86 Linux distro at the moment.

    BUT, it needed:
    sudo (yep, it’s Proxmox…apt install sudo)
    uuidgen (apt install uuid-runtime)
    strings (apt install binutils)
    Python 2 (!!! 2002 called and wants its snake back….apt install python)
    parted (apt install parted)

    Python and parted were late (i.e., painful) failures. parted was especially tricky since it failed silently (i.e., it said SUCCESS!! but failed to boot for pretty obvious reasons).

    But it’s running happily now. Thanks for all you do for the Jetson community. So I guess the real question is, between USB booting on the Nano and SSD booting on the Xavier, what the heck am I supposed to do with all of my Micro SD cards now?!? You stinker…

    • Wow, that sounds like a lot of work! I am glad you were able to flash your Jetson.
      You can do like I do. Keep your Micro SD cards next to your SD cards and small, unusable thumb drives. Those are right next to the R/W DVDs. Those are right next to the R/W CD ROMS. So that I can easily find them, I put those next to the disk cartridges and 3.5″ floppies. Of course, it makes sense to store those next the the 5 1/4″ floppies, and the 8″ floppies. On the shelf below, keep the paper tape, punch cards and program listings because you will find that you won’t access those as much anymore.
      Thanks for reading!

  3. Wonderful tutorial. I used Ubuntu 20.04 on a parallels VM installed on a Mac to implement this. Two issues got me. I had to install Qemu and Python 2.7 as additional dependencies. I also had to manually set the read / write status of the folder to get the script to work.

    Glad to have a method for direct to NVMe SSD.

    Cheers

    • Thank you for the kind words, and thank you for your feedback.

      As you know, NVIDIA does not officially support Ubuntu 20.04 for these scripts, which leads to the issues you encountered.

      In your ‘install_dependencies.sh’ file, do you have the line:

      $ sudo apt install simg2img qemu-user-static

      I added this which I had hoped would solve the qemu issue.

      I’ve been working on a Python workaround. The file /usr/bin/python is a symbolic link to the Python environment. The NVIDIA scripts use this. For 20.04, they decided to leave the link empty, as the directive is to explicitly call python3.
      If you use Python3, I have been told you get warnings, but it still works.

      The idea is to set the symbolic link to Python3 if it is not set, flash, and then put it back to its original state.

  4. It worked for me in Ubuntu 20.03 LTS Physical Machine Mode
    One of the note is it require 8G of extra space on Host machine to download the needed files

    • get_jetson_files.sh downloads the BSP and rootfs in a compressed form. Then it converts them into a disk image. The disk image represents a base Ubuntu 18.04 OS.

      flash_jetson_external_storage.sh – Copies the disk image to the Jetson, and updates the flash memory to handle booting the Jetson. After this step, the Jetson boots into Ubuntu.

      install_jetson_default_packages.sh – Installs the JetPack libraries on to the Jetson. These libraries are Jetson specific, and provide CUDA and machine learning and vision processing.

      Hope this helps.

  5. hi, thx for this video. I ran the installation and succeeded.

    But.. I can’t install nano, htop, etc.

    Which repositories i need to add?

  6. Hi,I followed the video and succed,but when I move the SSD to another jetson NX board it cannot startup, only black screen flash slowly. is this normal? When i move hard disk from one PC to another same type PC, it always can startup.thanks

  7. Should this work with a TX2 SOM? Does not seem to:
    [ 0.1461 ] Boot Rom communication
    [ 0.1499 ] tegrarcm_v2 –chip 0x19 0 –rcm rcm_list_signed.xml –skipuid
    [ 0.1522 ] RCM version 0X16
    [ 0.1544 ] Boot Rom communication failed
    [ 5.1913 ]
    Error: Return value 3
    Command tegrarcm_v2 –chip 0x19 0 –rcm rcm_list_signed.xml –skipuid
    Reading board information failed.
    Error: Failed to generate images for external device
    Cleaning up…

  8. So.. I did all the steps and nothing changed on my jeston..
    After “successful” flashing when I boot jetson it just starts like nothing changed
    I have all my data and setting still the same, SSD not formatted
    I even tried “sudo ./tools/l4t_create_default_user.sh -u ubuntu -p ubuntu” command

    I have ubuntu 16.04 x86 on host, jetson xavier nx (mic-710 aix) and Kingston SA400M8/240 SSD

    My flashing logs can be found there: https://disk.yandex.ru/d/tf_B5NhWkbLXHQ

    Please tell me what I’m missing there
    Thx for your work

  9. when i’m doing the fist jetson flashing:
    $ flash_jetson_external_storage.sh
    I’m getting a timeout on step 3: start the flashing process

    any idea how to succeed?

Leave a Reply

Your email address will not be published.


*