Now with CUDA! Intel RealSense D400 cameras – NVIDIA Jetson TX

In our previous article, Intel RealSense D400 librealsense2, we began work on making the the RealSense SDK work more smoothly on the NVIDIA Jetson TX Development kits. We now add CUDA support! Looky here:

Background

As you may recall, there are a few issues with running the RealSense SDK, librealsense 2, on the NVIDIA Jetson TX2. For example, the application “realsense-viewer” pegged a CPU at 100%, which results in poor performance. We also want to add support for the Jetson TX1.

As we had guessed, there was a wild render loop which did not have a sleep call. After fixing that, we add CUDA support. This is a little more of a challenge than one would like. Intel switched over to using CMake in version 2 of librealsense. The first task is to add flags to tell CMake to use CUDA. We chose the appropriately named “USE_CUDA” flag.

Next we add the actual CUDA code itself. Fortunately we have a template to work off. It turns out that Mehran Maghoumi (who is currently working at NVIDIA!) had done some work in his Github repository culibrealsense on adding CUDA support for the image conversion of the native image format (YUY2) to more display friendly RGB. A few nips and tucks later, we now have support for RGB, BGR, RGBA and BGRA in CUDA. BGR is the preferred OpenCV format. The ‘A’ refers to a alpha channel for the image.

There is some speedup using the new conversion code. In a few tests the conversion time of an image went from ~14ms to ~8ms. This is a healthy speedup. The code can probably be improved, but for our purposes it’s a nice speedup for the amount of work involved.

During the development process, we also upgraded to librealsense 2.10.4. The library is under heavy development, unfortunately the challenge is discovering newly “features” which don’t translate well into the Jetson environment. These tend to be things such as code written for Intel x86 processors and the myriad of idiocy that goes along with multi-platform libraries. To be fair, this is an Intel product. It makes sense that there are optimizations for their processors. It just takes a while to figure out what breaks when the optimizations are added.

The kernel patching routines are rewritten to support the Jetson TX1, and to be slightly more intelligent about configuring the kernel.

Note: Because space is tight on the Jetson TX1, you should first flash the Jetson TX1 with L4T 28.2 using JetPack 3.2, download the buildLibrealsense2TX repository, and then build the patched kernel. Once the kernel is in place, remove the kernel sources to give you enough space to operate.

Installation

On the JetsonHacks Github account, there is a repository buildLibrealsense2TX. To download the repository:

$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2TX
$ cd buildLibrealsense2TX
$ git checkout v0.8

The instructions are much the same as in the previous article. More details are available on the README.md file in the buildLibrealsense2TX directory.

Explain please

First, a word about what we’re doing. There are several RealSense camera formats that the standard UVC (video) module in the kernel does not support. librealsense provides patches to add those video modes to the appropriately. The patches also add support for properly adding timestamps to the incoming video stream. In addition, there are several patches that modify some of the Industrial I/O (IIO) tree. These patches are providing support to supplemental hardware on a RealSense camera, such as a 3D Gyro or 3D Accelerometer.

Note: Some of the patches apply to modules that are built into the kernel itself. Because these modules are required to be in the Image, and not built as external, you will need to compile the kernel itself along with the modules.

If you do not modify the Linux kernel, librealsense will mostly appear to work. You will probably experience issues a little further along when you are trying to get better precision and accuracy from the video and depth streams. As the RealSense D435 camera does not have supplemental hardware, the IIO patches don’t make any difference.

If you’re just looking to play around with the camera, you may be able to get away with not compiling the kernel and skip over it. If you’re more serious, you’ll have to start patchin’ !

Building the Kernel

Note: If you built your kernel as in the previous article, you must rebuild it again!

Building the kernel can be taxing, there are many little things that can go wrong. Plus, you can make your Jetson become brickly if something happens at an untoward momement. If you’re a go for it kind of person:

$ ./buildPatchedKernelTX2.sh

If something does go wrong during the build, you may want to try to debug it. As part of its cleanup process, the buildPatchedKernel script erases all of the source files and build files that it has downloaded and built. You can pass nocleanup as a command line flag so it keeps those files around. Hopefully you can fix everything.

$ ./buildPatchedKernelTX2.sh –nocleanup

Actually, this script is more useful as a template for rebuilding your own kernel with the librealsense changes. There are two scripts in the ‘scripts’ directory for helping in the kernel build:

patchKernel.sh applies the librealsense kernel patches.

configureKernel.sh Configures the kernel to add the appropriate modules needed by librealsense.

Since you’re a developer, you should be able to figure out what the scripts are doing and modify them to match your needs.

librealsense Installation

make sure that the RealSense camera is not attached to the system. Then install the library:

$ ./installLibrealsense.sh

Looking inside the script file, you will see that there are a couple of patches for the Jetson. The first patch is a work around for some code related to an Intel specific instruction set, the other is a workaround for the Industrial I/O (IIO) device detection.

The stock librealsense code appears to assume that a IIO device reports with a device and bus number. On the Jetson, the ina3221x Power Monitors do not follow this protocol. The result is that a series of warning are issued continuously as the library scans for HID devices that have been added (plugged in) to the system.

The library is looking for IIO HID devices (the accelerometer and gyroscope on a RealSense camera). The ina3221x is not a HID device, but appears during the IIO scanning. The library scans, but because it does not find a device or bus number for the power monitor, it issues a warning to stderr (the console). The result is that the console gets spammed, which in turn results in a performance penalty.

The workaround patch checks to see if the device detected is the power monitor before issuing a warning.

Similarly, the built in camera module control is via CSI/I2C, not USB as expected by librealsense. Again, a warning is sent to stderr by librealsense. There may be a clever way to determine if the warning is about the on-board Jetson camera, but in this case the patch just comments out the warning.

After applying the patches, the script compiles the library, examples and tools:

  • The library is installed in /usr/local/lib
  • The header files are in /usr/local/include
  • The examples and tools are located in /usr/local/bin

The script also sets up a udev rule so that the RealSense camera is available in user space.

Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then go and execute the tools and examples. For example:

$ cd /usr/local/bin
$ ./realsense-viewer

Conclusion

So there you have it. This is the first pass through of getting the RealSense D400 cameras working with the Jetson TX Dev kits. Because the RealSense SDK is under heavy development, we will have to keep our eye out for improvements in the weeks ahead!

Notes

  • L4T 28.2 installed by JetPack 3.2
  • A Jetson TX2 was used in the video
  • buildLibrealsense2TX is version v0.8

Intel Documentation:

14 Comments

  1. Hello-
    YOur videos/instructions are clear and concise enough for even a newbie like myself.

    I was wondering if anyone has tried using the RS D-400 cameras with a smartphone.
    Whattya think?
    Thanks,
    TM

  2. AWESOME! Hi Jim, do you think the D435 is a candidate for replacing / augmenting the ZED or structure.io cameras on RACECAR/J?

  3. Hello Jim,

    Thank you for your tutorials. It help me so much! I have followed the previous tuturial to install realsense D415 on my Jetson TX2 and patch the kernel.
    But I have just discovered this new tutorial, I’m trying to follow it, but I have an issue during the installLibrealsense process. It takes only a few second and I get the following error:
    CMake Error at /usr/share/cmake-3.5/Modules/FindCUDA.cmake:617 (message):
    Specify CUDA_TOOLKIT_ROOT_DIR
    Call Stack (most recent call first):
    CMakeLists.txt:61 (find_package)
    and the process stop.
    Then, if I run the realsense viewer after running the nvpmodel command I get a CPU to 100% without running the camera…
    Could you help me with this issue, please?

    Thanks
    Rich

  4. I’m using L4T 28.2 and Cuda was normally installed with the Jetpack 3.2, but you are right, it seems that CUDA was not install during the process.

    I will investigate on that way and let I will you know if I found the solution.

    Thanks for your quick answer

    • Hi Sergey,
      Thank you for taking the time to add the changes to your library. I know that the Jetson is not an officially supported platform, and appreciate the effort to add support.

Leave a Reply

Your email address will not be published.


*