Librealsense Update – NVIDIA Jetson TX Dev Kits

Recently the Intel Librealsense Development Team added CUDA support to the Librealsense SDK. Here’s how to install it on the Jetson TX Dev kits. Looky here:

Background

As you may recall, we added CUDA support to librealsense in a previous article, Now with CUDA! Intel RealSense D400 cameras – NVIDIA Jetson TX. Recently I received a nice note from the Librealsense Development Team stating that they had used the code as a basis for adding CUDA support to librealsense (starting in version 2.13.0).

Intel took a look at the performance (from https://github.com/IntelRealSense/librealsense/pull/1866):

Initial CUDA support for conversions and pointcloud generation.

Performance improvement:

As tested on the Jetson TX2:

Description Resolution fps No Cuda, avg. (ms) With Cuda, avg. (ms)
YUY2 into RGB8 1920×1080 30 30 – 32 5-7
YUY2 into RGBA8 1920×1080 30 34 7.5
YUY2 into BGR8 1920×1080 30 35 5 – 7
YUY2 into BGRA8 1920×1080 30 37 7 – 10
YUY2 into Y16 1920×1080 30 10 5 – 8
YSplitting Y8I to Y8, Y8 1280×800 30 9 – 10 3 – 5
Splitting Y12 to Y16, Y16 1280×800 25 22.9 5 – 7
Pointcloud (deproject depth) 640×480 30 8 – 9 3 – 5

In order to activate GPU full power run: sudo ./jetson_clocks.sh

You can see that there is a 4-7x speedup on the CPU only versus GPU code on the Jetson. The term fps means frames per second. Remember that when you are running 30 fps, each frame arrives about every 33ms.

Note: Because space is tight on the Jetson TX1, you should first flash the Jetson TX1 with L4T 28.2 using JetPack 3.2.1, download the buildLibrealsense2TX repository, and then build the patched kernel. Once the kernel is in place, remove the kernel sources to give you enough space to operate.

Installation

On the JetsonHacks Github account, there is a repository buildLibrealsense2TX. To download the repository:

$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2TX
$ cd buildLibrealsense2TX
$ git checkout v0.9

The instructions are much the same as in the previous article. More details are available on the README.md file in the buildLibrealsense2TX directory.

Explain please

First, a word about what we’re doing. There are several RealSense camera formats that the standard UVC (video) module in the kernel does not support. librealsense provides patches to add those video modes to the appropriately. The patches also add support for properly adding timestamps to the incoming video stream. In addition, there are several patches that modify some of the Industrial I/O (IIO) tree. These patches are providing support to supplemental hardware on a RealSense camera, such as a 3D Gyro or 3D Accelerometer.

Note: Some of the patches apply to modules that are built into the kernel itself. Because these modules are required to be in the Image, and not built as external, you will need to compile the kernel itself along with the modules.

If you do not modify the Linux kernel, librealsense will mostly appear to work. You will probably experience issues a little further along when you are trying to get better precision and accuracy from the video and depth streams. As the RealSense D435 camera does not have supplemental hardware, the IIO patches don’t make any difference.

If you’re just looking to play around with the camera, you may be able to get away with not compiling the kernel and skip over it. If you’re more serious, you’ll have to start patchin’ !

Building the Kernel

Note: If you built your kernel as in the previous article, you must rebuild it again!

Building the kernel can be taxing, there are many little things that can go wrong. Plus, you can make your Jetson become brickly if something happens at an untoward momement. If you’re a go for it kind of person:

$ ./buildPatchedKernelTX.sh

If something does go wrong during the build, you may want to try to debug it. As part of its cleanup process, the buildPatchedKernel script erases all of the source files and build files that it has downloaded and built. You can pass nocleanup as a command line flag so it keeps those files around. Hopefully you can fix everything.

$ ./buildPatchedKernelTX.sh –nocleanup

Actually, this script is more useful as a template for rebuilding your own kernel with the librealsense changes. There are two scripts in the ‘scripts’ directory for helping in the kernel build:

patchKernel.sh applies the librealsense kernel patches.

configureKernel.sh Configures the kernel to add the appropriate modules needed by librealsense.

Since you’re a developer, you should be able to figure out what the scripts are doing and modify them to match your needs.

librealsense Installation

make sure that the RealSense camera is not attached to the system. Then install the library:

$ ./installLibrealsense.sh

The script does several things:

  • Installs dependencies
  • Builds CMake
  • Builds librealsense and associated apps

CMake 3.8+

The librealsense build system requires CMake version 3.8 or above. The standard Jetson TX version of CMake is 3.5.1 for L4T 28.2.x. Therefore the script builds CMake (in the home folder). However, the script does not install CMake into the system area, it just uses the new version directly from the build fold. Of course, you can install the new version of CMake on your system if so desired.

After the script compiles the library, the new files are placed in the following directories:

  • The library is installed in /usr/local/lib
  • The header files are in /usr/local/include
  • The examples and tools are located in /usr/local/bin

The script also sets up a udev rule so that the RealSense camera is available in user space.

Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then go and execute the tools and examples. For example:

$ cd /usr/local/bin
$ ./realsense-viewer

Conclusion

It is nice to have the CUDA upgrade for librealsense in the official repository. Because the RealSense SDK is under heavy development, we will have to keep our eye out for improvements in the weeks ahead!

Notes

  • In the video, a Jetson TX2 is shown
  • L4T 28.2.1 installed by JetPack 3.2.1
  • buildLibrealsense2TX is version v0.9

Intel Documentation:

4 Comments

  1. Will this work with the R200 3D camera as well as the D400’s? I ordered a 435 back in January and so far have been unable to obtain one.

    • It depends. When it is freshly flashed, the system is in a known state and the procedure described in the articles works. If you are comfortable building your own kernel, then you probably already know where any possible build issues may lie. For simplicity in these articles, we assume a fresh flash just to eliminate trying to debug different configurations and dependencies. Thanks for reading!

Leave a Reply

Your email address will not be published.


*