Intel RealSense D435i on NVIDIA Jetson AGX Xavier

Intel is now shipping the RealSense D435i Depth Camera. The camera includes a built in IMU. In this article, we cover interfacing with the Jetson AGX Xavier. Looky here:

Background

In previous articles, we went through how to install the Intel RealSense library (called librealsense 2) on the Jetson TX1 and Jetson TX2. Since that time, we have seen the introduction of the RealSense D435i camera and Jetson AGX Xavier.

The software drivers to interface with the D435i have seen a few updates (such as better CUDA support) since that time. However, the biggest challenge in installing full librealsense 2 support on the Xavier is that additional kernel modules must be built and installed. Because some of the affected modules are built in to the kernel itself, the kernel image itself should be built.

The Jetson AGX Xavier is an embedded system. In this implementation, the Linux kernel is signed and resides in a partition on disk. This is for security reasons, as you are aware if you have been following the computer news for the last few months. This complicates building the kernel on the Jetson Xavier itself, as the signing application only runs on a PC. The JetPack installer contains the signing application.

The NVIDIA approved method is to cross compile the kernel and modules on the PC, and then flash them on the Jetson. In this article, we do something exactly unlike that.

Librealsense 2 Installation

Note: This installation is for intermediate and advanced developers. You should be familiar with kernel modules, and the kernel Image. It is strongly suggested that you do this on a fresh install, immediately after flashing the Jetson. You should make backups of anything value, by you know that already.

WARNING: This installation method assumes that you are running a stock kernel.

On the JetsonHacks Github account, there is a repository named buildLibrealsense2Xavier. To download the repository:

$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2Xavier
$ cd buildLibrealsense2Xavier

Build Kernel and Modules

The first step is to build the needed modules and a new kernel. Also, in order to have the Xavier understand the different video formats, there are some patches to apply to the module source code.

There are no librealsense 2 patches available to directly patch the Xavier, because it is running kernel version 4.9. This is between the different kernel versions which have patch revisions. For this reason, the patches sub-directory contains artisan patches, exquisitely crafted to upgrade the Jetson Xavier to run the librealsense 2 libraries.

Some of the patches actually change header files which touch some modules which are built in to the kernel. That is why the kernel Image needs to be updated. If the kernel Image is not updated, the logs get cluttered with a large number of warnings about incorrect video formats being present.

In order to patch and rebuild the kernel Image, modules and then install the modules:

$ ./buildPatchedKernel.sh

At the end of the process, you should have an ‘Image’ file in the image sub-directory.

Note: There is a convenience script removeAllKernelSources.sh which will remove all of the kernel sources that the buildPatchedKernel script downloaded, along with all of the build remnants. If you do not intend to build other kernel modules, or modify the kernel Image itself, you may find this a useful tool.

Flashing the Kernel Image

Once the script is finished, you can now flash the new Image on to the Jetson. Use a file transfer utility such as scp or ftp to transfer the new Image file to the host PC with the JetPack installer. Sneakernet works too.

Now shutdown the Xavier. Connect the Jetson to the host PC via the USB cable in the same manner as the original JetPack flash. Then place the Xavier into Force Recovery Mode. This is also the same procedure used when flashing JetPack.

Go to the host PC. Make sure you make a backup of the original Xavier kernel Image. The Image should be in the host JetPack directory, in the Xavier/Linux_for_Tegra/kernel directory. With the backup secure, you can then copy the new Image that was built on the Jetson in its place.

Then go up one level to the Linux_for_Tegra directory. You are then ready to flash the Image:

$ sudo ./flash.sh -k kernel jetson-xavier mmcblk0p1


This will flash only the kernel Image to the Xavier, replacing the previous version. This should not effect the rootfs. The Xavier will reboot once the kernel Image has been transferred. Go back to the Xavier.

Build librealsense 2

The machine has just rebooted. Open a terminal and go back to the repository directory.

$ cd buildLibrealsense2Xavier

Make sure there are no RealSense cameras attached to the Xavier. Now build librealsense 2, and install the libraries and applications:

$ ./installLibrealsense.sh

The script compiles the library, examples and tools:

  • The library is installed in /usr/local/lib
  • The header files are in /usr/local/include
  • The examples and tools are located in /usr/local/bin

The script also sets up a udev rule so that the RealSense camera is available in user space.

Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then execute the tools and examples, such as:

$ cd /usr/local/bin
$ ./realsense-viewer

Conclusion

This installer is still a work in progress. There are a couple of issues that need some work. However, enough people have asked about this that it seems worthwhile to get something out there for people to start working with.

Notes

  • In the video, installation was performed on a Jetson AGX Xavier running L4T 31.1 (JetPack 4.1.1 Developers Preview).
  • librealsense v2.17.1
  • buildLibrealsense2Xavier v0.8

16 Comments

  1. Are you able to get the realsense camera to work when connected directly to the Xavier’s USB ports? I got errors like “Incomplete video frame detected” and “USB SCP overflow” when I use the dev kit’s built in USB ports. If I use a PCIe USB card it works fine. Trying to figure out if I got a bad unit or if they are all like this.

    Thanks for the great post, the kernel build script works perfectly.

    • I have the same issue on this Xavier as you encountered. The whole kernel build process is a little convoluted here, but it feels simpler to me than setting up the cross compile environment and running it on the PC. Which PCIe USB card are you using? Did you have to compile a kernel module to recognize it?

      I’m guessing that this has to do with the USB firmware, I’m a little surprised that the USB-C ports act differently than the USB 3.1. I tried a USB-C hub, but that did not appear to help matters; it acts the same as when the camera is plugged directly in to the USB-C port.

      I’m guessing that it has something to do with the RealSense Camera acting like a block-transfer device. There’s probably some timing or handshake issue down at the firmware level. Fortunately I don’t have access to that code or the tools needed, because that’s real work to debug!

      Thanks for reading!

    • Have the same issue. Thought that it’s been caused by this kernel’s patch, but after rolling back to original kernel image, found the same errors, whether connecting to Type-C or Type-A USB port, or USB hub.
      Though I hadn’t had so many errors before. So, probably, the reason is the latest firmware (05.11) and/or librealsense version (2.19).

  2. I used the HighPoint Technologies RocketU 1322A. (marked RU1322A on pcb) It cost me around 50 USD and it worked without any special kernel module. The card does stick out the top rather precariously, ok for desktop use but not mechanically robust enough for a robot.

  3. Please.. JETSON TX2 + Jetpack 3.3 + D435i install guide..
    I followed your install guide for installing a realsense with Jetson tx2 with Jetpack 3.2 but I failed.. I reset my jetson tx2 with Jetpack 3.2 before following your install guide. And I tried with Jetpack 3.3 as well. But the camera always not detected when I activate the realsense-viewer. The camera was connected with usb 3.0 port.
    Do you think it is happened due to my camera is D435i, not D435?

  4. Hi. Has anyone attempted to use more than one D435 cameras (synced) for wider field of view with Xavier? How does Xavier compare with other embedded computers for robotic applications (with ROS)?

  5. How long does it take to compile the kernel on Xavier (after running ./jetsonClocks.sh)

    I noticed that the default kernel of Jetpack 4.1.1 has PREEMPT enabled, does your custom kernel also have this enabled?

    nvidia@jetson-0423018054325:~/dev/rt-tests-1.3$ uname -a
    Linux jetson-0423018054325 4.9.108-tegra #1 SMP PREEMPT Wed Oct 31 15:17:21 PDT 2018 aarch64 aarch64 aarch64 GNU/Linux

    • It takes ~ 19 minutes.
      The kernel build uses the current running configuration:

      $ zcat /proc/config.gz > .config

      The .config file sets all of the kernel and module information. I haven’t looked specifically at PREEMPT, but would be very surprised if it was different.

  6. Small issue in the description above: it says twice
    cd buildLibrealsense2Xavier
    cd buildLibrealsense2Xavier

    But you want
    cd buildLibrealsense2Xavier
    ./installLibrealsense.sh

  7. Good job, thanks!
    One thing to note is that librealsense/CMake/cuda_config.cmake seems to be better modified with Xavier’s arch ‘compute_72’ instead of ‘compute_53’ and ‘compute_62’ currently used, which probably stand for TX1/2.

Leave a Reply

Your email address will not be published.


*