Jetson Xavier NX – Run from SSD

It is a simple task to a sea of gigabyte goodness to the NVIDIA Jetson Xavier NX by adding a NVMe SSD. This is the best performance upgrade you can make for your Jetson. Looky here:

Background

Even though they share the same form factor, the Jetson Xavier NX Developer Kit has a nice addition in comparison to the Jetson Nano. A M.2 Key M slot. With the M.2 Key M slot, we can easily add a Solid State Drive (SSD). The M.2 Key M slot uses the Non-Volatile Memory Express (NVMe) protocol that runs over PCIe. 

There are two types of SSDs that use Key M. When you install a SSD card in the slot, the card needs to be the NVMe PCIe type.

The SSD is typically 5-20x faster than the flash memory in a SD card, so anytime you do disk access things speed up considerably.

This process works on the Jetson AGX Xavier too. See Notes below.

Note: Before upgrading using Over The Air (OTA) updates, read the notes below in Upgrading.

Materials and Tools

In the video, we install a Western Digital 500GB NVMe SSD. There are several different sizes and brands of these types of devices, we also get good results with the Samsung variety. We also use our trusty iFixit Pro Tech Toolkit which contains a variety of useful tools for just this purpose.

Hardware Installation

The M.2 Key M connector is on the underside of the carrier board. A retaining screw is at the end of the M.2 slot. Remove the retaining screw. From the factory, the screw may have some blue thread locker on it, which may require a little elbow grease to start the removal process.

Remove Retaining Screw

Next install the the SSD card into the Key M connector, and then use the same retaining screw to hold it in place.

Install SSD and secure with retaining screw

Disk Configuration

After the hardware install, it is time to configure the disk under Ubuntu. Hook the Xavier back up to a keyboard, mouse and monitor. In the video above, we use mostly GUI tools, please refer to the video for a walk through.

The basic steps are to format the disk, and then create partition(s). In the video, we just allocate most of the disk space to one big ol’ partition. However you may want to be a little fancier. Being Linux, there are tools for doing this through the command line, as all the pros will tell you.

At this point, you can now begin using the SSD. It will show up on your desktop as an unmounted drive.

Running from SSD

We go a little further here. After the Jetson boots the image from the SD card, we switch the rootfs to point to the SSD. In effect, the system will now run from the SSD, the SD card is only there to boot the system.

You should do this process directly after creating a new SD card.

On the JetsonHacks account on Github, there is a repository rootOnNVMe. Clone the repository:

$ git clone https://github.com/jetsonhacks/rootOnNVMe

and switch over to that repository’s directory:

$ cd rootOnNVMe

Next, copy the rootfs of the eMMC/SD card to the SSD

$ ./copy-rootfs-ssd.sh

Finally, we will add a service which will run a script when the system starts up. The script will “pivot the root” to the SSD so that the system will run from the SSD.

$ ./setup-service.sh

There’s a chapter in the video ‘Tech Talk Time’ that walks through how the script works. After setting up the service, reboot for the changes to take effect.

The performance of your system will feel much snappier after this simple modification.

Boot Notes

These script changes the rootfs to the SSD after the kernel image is loaded from the eMMC/SD card. For the Xavier NX, you will still need to have the SD card installed for booting. As of this writing, the default configuration of the Jetson NX does not allow direct booting from the NVMe.

Upgrading

Note: (7/9/2020) There are currently issues with the Over The Air (OTA) updates with the JetPack 4.4 release. Please see the article: JetPack 4.4 Over The Air (OTA) Update issues. NVIDIA is working on the issues, but currently we do not recommend upgrading to JetPack 4.4 production release via the OTA update.

Once this service is installed, the rootfs will be on the SSD. If you upgrade to a newer version of L4T using OTA updates (using the NVIDIA .deb repository), you will need to also apply those changes to the SD card that you are booting from.

Typically this involves copying the /boot* directory and /lib/modules/<kernel name>/ from the SSD to the SD card. If they are different, then modules load will be ‘tainted’, that is, the modules version will not match the kernel version.

Special Shout Out!

This is taken from the NVIDIA Jetson AGX Xavier forum https://forums.developer.nvidia.com/t/how-to-boot-from-nvme-ssd/65147/22, written by user crazy_yorik (https://forums.developer.nvidia.com/u/crazy_yorick). Thank you crazy_yorik!

Notes

  • Initial Release, May 2020
  • JetPack 4.4 DP
  • Tested on Jetson Xavier NX, Jetson AGX Xavier

Installation of the SSD is different on the AGX Xavier. See: Install NVMe SSD on NVIDIA Jetson Developer Kit. After formatting the drive, software installation is the same as shown in the article.

27 Comments

  1. Hi, I immediately went out and bought a SSD for my NX and I am very happy with the snappy results. I worked from the video that I found to be fantastic with humor too.

    My only concern is that if and when I flash a new micro SD card, will I be able to handle it? Right now we are on Jetpack 4.4, and there will certainly be a 4.5 in the near future. I have read crazy_yorik’s forum comments, but that is beyond me, I was good with your video and hope that you will lead us through any updates without starting all over again.

  2. It depends on how you decide to upgrade your Jetson. If you do an over the air update, then you will need to copy the new kernel image and /boot directory over to the SD card. Typically you would also copy over the modules too. Thanks for reading!

  3. I have a question. I have been reading your articles on changing the swap file on the nano and was trying to use resizeSwapMemory article to reduce the size of my swap file. I also used this article to run the NX from an SSD which is working great.

    However, is this creating any issues with the swap file? I notice that I have a 8GB swfile on the micro SD card and a 4GB swapfile in the SSD. Activity Monitor sees nearly 12 GB of swap (8GB swap file and 4GB of zram). I would like to be using a swap file on the faster ssd vs. the sd.

  4. Hi! I bought a WD Black 500GB Nvme SSD but it’s not showing up on the Xavier NX. Could my NVME slot or SSD be faulty? Any possible solutions or ways of investigating?

  5. Since we are dealing with the same hardware, it could be in the process and hopefully not the hardware. Did you follow the video step-by-step? I do not remember all the details, but the video went through the formatting of the disk and then I followed the process step by step. Watch the video again and see if you missed something.

  6. Thanks for the excellent guidance – very much appreciated!

    So if a new version of jetpack comes out and we have already setup booting to SSD, how do you recommend updating the whole system?

    Thanks!

  7. This process no longer works. I ran the scripts, but it failed to identify the new NVME disk, so I changed it from /dev/nvmeonn1p1 to /dev/nvme0n1 and mounted it to /mnt seems to work fine and browsing the folders shows a copy of the FS.
    I then run the startup scripts, but nothing is changed, the system boots from and loads from the SDHC and /mnt isn’t even mounting.. where else can I look?

      • The script assumes /dev/nvmeon1p1 and the Samsung Evo 970 Plus seemed to install at /dev/nvme0n1. So I mounted /dev/nvme0n1 to /mnt and re-ran the scripts. When the device reboots, it boots from the SD card and loads from the SD card. /mnt shows up, but it isn’t mounted until I access it. Are there items in the second script that hard code /dev/nvmeon1p1? It seems that everything points to /mnt, but it doesn’t work. How can I force it, or do this manually without the script?

      • I got it working. I just changed all the instances of /nvme0n1p1 to /nvme0n1. It’s in a few locations including a couple files in the data folder.

          • Oh right. I think by the time I turn 50 I will start paying attention to details. Thanks. Got it working correctly with the newest jet pack. Your contributions are incredible and valuable and much appreciated. Don’t let internet jerks like me discourage you.

            • Thanks for the note. I don’t think this is trolling, I know you were probably a little frustrated. Unfortunately there are some things you need to learn the hard, expensive way if you know what I mean.

              If you want to see an inappropriate response, look down a couple of comments and look at Alain’s comment. Note that the main issue encountered was something wrong with the NVIDIA update process, unrelated to the anything in this article. Fortunately when I was a young lad I was taught never to take criticism from someone from whom I would not take advice. Turns out that’s all you need to know when dealing with people on the inter-webs.

              As with most computer stuff, missing one of the steps can ruin everything afterwards. BTW, I think once you’re 50 you pay even less attention to details unless that’s your personality type to begin with. Thanks for reading!

  8. Thank you for this article, could you please give us more details on the UPGRADING part ? I already bricked a jetson Xavier nx (“cpu bootloader is not running on device”) and I suspect this was caused by the Nvidia OTA bootloader running on a device that booted from the SSD. I updated everything on the micro sd card, then ran your script to boot from SSD, but today I am seeing on the update list an OTA bootloader update waiting, very afraid to run it …

  9. Well, with new jetpack 4.4 update, the Xavier NX does not boot anymore.

    I think you proposed this method to quickly. With major update, everything goes down.

    Many thanks for this poor method. Now, i have to restart from zero. Great. Really great.

    • Too soon? This was published in May. You don’t state how you upgraded, or what steps you took for taking into account that the rootfs is on the NVMe drive.
      I do not know why you believe you need to restart from zero, but good luck on your project.

    • That isn’t the fault of this process to move root to .m2 its the upgrade its self that was broke. Blame NVIDIA, not these guys. ( and that upgrade bug is supposed to be fixed by now )

  10. Is seems this procedure works, but gets broken if the user takes a JetPack updates which come out periodically. Is there ways to either avoid this breaking and still run the OS off NVMe. Or is there a way to quickly fix the issue? And if so, what are the steps?

    • I’m working on the articles for the upgrade steps. However, upgrading is a two step process. First, upgrade the system (running from the SSD). Before rebooting, copy over the /boot directory to the SD card. Something like:

      $ sudo mkdir /mnt/sd/
      $ sudo mount /dev/mmcblk0p1 /mnt/sd/
      $ sudo rsync -a /boot/ /mnt/sd/boot/

      I don’t think so, but you may have to copy over the modules


      $ sudo rsync -a /lib/modules/4.9.140-tegra/ /mnt/sd/lib/modules/4.9.140-tegra/

      I have not tested this enough to make sure it is bullet proof, but others have had good success with it.

  11. I just copied boot and modules over to the SD *before* a reboot after upgrades. Worked just fine. After, might not be a good idea. May not boot, depending on how much of a change.

  12. I tried your commands after the OTA upgrade and then rebooted with success. I was a bit like a dog watching TV, but so far it works. Thanks. I look forward to your new article on the subject just to see if we can make any tweaks to the system. Any chance of booting from the SSD in the future>

  13. Hi, there was another OTA upgrade for Ubuntu and NVIDIA upgrades. This time I just let the upgrade take place and rebooted with no issues showing up yet. Should we still be using the following commands? I have not done this yet.

    $ sudo mkdir /mnt/sd/
    $ sudo mount /dev/mmcblk0p1 /mnt/sd/
    $ sudo rsync -a /boot/ /mnt/sd/boot/

Leave a Reply

Your email address will not be published.


*