Intel is now shipping the RealSense D435i Depth Camera. The camera includes a built in IMU. In this article, we cover interfacing with the Jetson AGX Xavier. Looky here:
In previous articles, we went through how to install the Intel RealSense library (called librealsense 2) on the Jetson TX1 and Jetson TX2. Since that time, we have seen the introduction of the RealSense D435i camera and Jetson AGX Xavier.
The software drivers to interface with the D435i have seen a few updates (such as better CUDA support) since that time. However, the biggest challenge in installing full librealsense 2 support on the Xavier is that additional kernel modules must be built and installed. Because some of the affected modules are built in to the kernel itself, the kernel image itself should be built.
The Jetson AGX Xavier is an embedded system. In this implementation, the Linux kernel is signed and resides in a partition on disk. This is for security reasons, as you are aware if you have been following the computer news for the last few months. This complicates building the kernel on the Jetson Xavier itself, as the signing application only runs on a PC. The JetPack installer contains the signing application.
The NVIDIA approved method is to cross compile the kernel and modules on the PC, and then flash them on the Jetson. In this article, we do something exactly unlike that.
Librealsense 2 Installation
Note: This installation is for intermediate and advanced developers. You should be familiar with kernel modules, and the kernel Image. It is strongly suggested that you do this on a fresh install, immediately after flashing the Jetson. You should make backups of anything value, by you know that already.
WARNING: This installation method assumes that you are running a stock kernel.
On the JetsonHacks Github account, there is a repository named buildLibrealsense2Xavier. To download the repository:
$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2Xavier
$ cd buildLibrealsense2Xavier
Build Kernel and Modules
The first step is to build the needed modules and a new kernel. Also, in order to have the Xavier understand the different video formats, there are some patches to apply to the module source code.
There are no librealsense 2 patches available to directly patch the Xavier, because it is running kernel version 4.9. This is between the different kernel versions which have patch revisions. For this reason, the patches sub-directory contains artisan patches, exquisitely crafted to upgrade the Jetson Xavier to run the librealsense 2 libraries.
Some of the patches actually change header files which touch some modules which are built in to the kernel. That is why the kernel Image needs to be updated. If the kernel Image is not updated, the logs get cluttered with a large number of warnings about incorrect video formats being present.
In order to patch and rebuild the kernel Image, modules and then install the modules:
At the end of the process, you should have an ‘Image’ file in the image sub-directory.
Note: There is a convenience script removeAllKernelSources.sh which will remove all of the kernel sources that the buildPatchedKernel script downloaded, along with all of the build remnants. If you do not intend to build other kernel modules, or modify the kernel Image itself, you may find this a useful tool.
Flashing the Kernel Image
Once the script is finished, you can now flash the new Image on to the Jetson. Use a file transfer utility such as scp or ftp to transfer the new Image file to the host PC with the JetPack installer. Sneakernet works too.
Now shutdown the Xavier. Connect the Jetson to the host PC via the USB cable in the same manner as the original JetPack flash. Then place the Xavier into Force Recovery Mode. This is also the same procedure used when flashing JetPack.
Go to the host PC. Make sure you make a backup of the original Xavier kernel Image. The Image should be in the host JetPack directory, in the Xavier/Linux_for_Tegra/kernel directory. With the backup secure, you can then copy the new Image that was built on the Jetson in its place.
Then go up one level to the Linux_for_Tegra directory. You are then ready to flash the Image:
$ sudo ./flash.sh -k kernel jetson-xavier mmcblk0p1
This will flash only the kernel Image to the Xavier, replacing the previous version. This should not effect the rootfs. The Xavier will reboot once the kernel Image has been transferred. Go back to the Xavier.
Build librealsense 2
The machine has just rebooted. Open a terminal and go back to the repository directory.
$ cd buildLibrealsense2Xavier
Make sure there are no RealSense cameras attached to the Xavier. Now build librealsense 2, and install the libraries and applications:
The script compiles the library, examples and tools:
- The library is installed in /usr/local/lib
- The header files are in /usr/local/include
- The examples and tools are located in /usr/local/bin
The script also sets up a udev rule so that the RealSense camera is available in user space.
Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then execute the tools and examples, such as:
$ cd /usr/local/bin
This installer is still a work in progress. There are a couple of issues that need some work. However, enough people have asked about this that it seems worthwhile to get something out there for people to start working with.
- In the video, installation was performed on a Jetson AGX Xavier running L4T 31.1 (JetPack 4.1.1 Developers Preview).
- librealsense v2.17.1
- buildLibrealsense2Xavier v0.8