Librealsense Update – NVIDIA Jetson TX Dev Kits

Recently the Intel Librealsense Development Team added CUDA support to the Librealsense SDK. Here’s how to install it on the Jetson TX Dev kits. Looky here:

Background

As you may recall, we added CUDA support to librealsense in a previous article, Now with CUDA! Intel RealSense D400 cameras – NVIDIA Jetson TX. Recently I received a nice note from the Librealsense Development Team stating that they had used the code as a basis for adding CUDA support to librealsense (starting in version 2.13.0).

Intel took a look at the performance (from https://github.com/IntelRealSense/librealsense/pull/1866):

Initial CUDA support for conversions and pointcloud generation.

Performance improvement:

As tested on the Jetson TX2:

Description Resolution fps No Cuda, avg. (ms) With Cuda, avg. (ms)
YUY2 into RGB8 1920×1080 30 30 – 32 5-7
YUY2 into RGBA8 1920×1080 30 34 7.5
YUY2 into BGR8 1920×1080 30 35 5 – 7
YUY2 into BGRA8 1920×1080 30 37 7 – 10
YUY2 into Y16 1920×1080 30 10 5 – 8
YSplitting Y8I to Y8, Y8 1280×800 30 9 – 10 3 – 5
Splitting Y12 to Y16, Y16 1280×800 25 22.9 5 – 7
Pointcloud (deproject depth) 640×480 30 8 – 9 3 – 5

In order to activate GPU full power run: sudo ./jetson_clocks.sh

You can see that there is a 4-7x speedup on the CPU only versus GPU code on the Jetson. The term fps means frames per second. Remember that when you are running 30 fps, each frame arrives about every 33ms.

Note: Because space is tight on the Jetson TX1, you should first flash the Jetson TX1 with L4T 28.2 using JetPack 3.2.1, download the buildLibrealsense2TX repository, and then build the patched kernel. Once the kernel is in place, remove the kernel sources to give you enough space to operate.

Installation

On the JetsonHacks Github account, there is a repository buildLibrealsense2TX. To download the repository:

$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2TX
$ cd buildLibrealsense2TX
$ git checkout v0.9

The instructions are much the same as in the previous article. More details are available on the README.md file in the buildLibrealsense2TX directory.

Explain please

First, a word about what we’re doing. There are several RealSense camera formats that the standard UVC (video) module in the kernel does not support. librealsense provides patches to add those video modes to the appropriately. The patches also add support for properly adding timestamps to the incoming video stream. In addition, there are several patches that modify some of the Industrial I/O (IIO) tree. These patches are providing support to supplemental hardware on a RealSense camera, such as a 3D Gyro or 3D Accelerometer.

Note: Some of the patches apply to modules that are built into the kernel itself. Because these modules are required to be in the Image, and not built as external, you will need to compile the kernel itself along with the modules.

If you do not modify the Linux kernel, librealsense will mostly appear to work. You will probably experience issues a little further along when you are trying to get better precision and accuracy from the video and depth streams. As the RealSense D435 camera does not have supplemental hardware, the IIO patches don’t make any difference.

If you’re just looking to play around with the camera, you may be able to get away with not compiling the kernel and skip over it. If you’re more serious, you’ll have to start patchin’ !

Building the Kernel

Note: If you built your kernel as in the previous article, you must rebuild it again!

Building the kernel can be taxing, there are many little things that can go wrong. Plus, you can make your Jetson become brickly if something happens at an untoward momement. If you’re a go for it kind of person:

$ ./buildPatchedKernelTX.sh

If something does go wrong during the build, you may want to try to debug it. As part of its cleanup process, the buildPatchedKernel script erases all of the source files and build files that it has downloaded and built. You can pass nocleanup as a command line flag so it keeps those files around. Hopefully you can fix everything.

$ ./buildPatchedKernelTX.sh –nocleanup

Actually, this script is more useful as a template for rebuilding your own kernel with the librealsense changes. There are two scripts in the ‘scripts’ directory for helping in the kernel build:

patchKernel.sh applies the librealsense kernel patches.

configureKernel.sh Configures the kernel to add the appropriate modules needed by librealsense.

Since you’re a developer, you should be able to figure out what the scripts are doing and modify them to match your needs.

librealsense Installation

make sure that the RealSense camera is not attached to the system. Then install the library:

$ ./installLibrealsense.sh

The script does several things:

  • Installs dependencies
  • Builds CMake
  • Builds librealsense and associated apps

CMake 3.8+

The librealsense build system requires CMake version 3.8 or above. The standard Jetson TX version of CMake is 3.5.1 for L4T 28.2.x. Therefore the script builds CMake (in the home folder). However, the script does not install CMake into the system area, it just uses the new version directly from the build fold. Of course, you can install the new version of CMake on your system if so desired.

After the script compiles the library, the new files are placed in the following directories:

  • The library is installed in /usr/local/lib
  • The header files are in /usr/local/include
  • The examples and tools are located in /usr/local/bin

The script also sets up a udev rule so that the RealSense camera is available in user space.

Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then go and execute the tools and examples. For example:

$ cd /usr/local/bin
$ ./realsense-viewer

Conclusion

It is nice to have the CUDA upgrade for librealsense in the official repository. Because the RealSense SDK is under heavy development, we will have to keep our eye out for improvements in the weeks ahead!

Notes

  • In the video, a Jetson TX2 is shown
  • L4T 28.2.1 installed by JetPack 3.2.1
  • buildLibrealsense2TX is version v0.9

Intel Documentation:

18 Comments

  1. Will this work with the R200 3D camera as well as the D400’s? I ordered a 435 back in January and so far have been unable to obtain one.

    • It depends. When it is freshly flashed, the system is in a known state and the procedure described in the articles works. If you are comfortable building your own kernel, then you probably already know where any possible build issues may lie. For simplicity in these articles, we assume a fresh flash just to eliminate trying to debug different configurations and dependencies. Thanks for reading!

  2. Hi,
    I am new to Linux and Jetson platform. I am planning to use TX2 on Auvidea J120 carrier board and attach RealSense D435 to it. Now for enabling USB 3.0 on the carrier board, the vendor has provided some dtb files to flash. From the article above I understand that I need to rebuild kernel as well.

    I went through the Kernel build scripts you provided and notice that it is slightly different from the official docs. In official docs they build kernel, dtb and modules and then copy the kernel image and dtbs to respective folders in Linux_for_Tegra/kernel and Linux_for_Tegra/kernel/dtb (for later flashing via JetPack I suppose). At the end they use make modules_install to install modules in Linux_for_Tegra/rootfs.

    In your scripts you are building on the platform but then you just replace the Image in /boot. What about the dtbs? shouldn’t these be copied somewhere? and I noticed in makekernel.sh at the end you also execute make modules_install. Does it copy the modules to rootfs as well?

    I am planning to cross-compile on host to build the kernel for librealsense. So I am trying to go through your scripts to understand and perform the kernel config, patch and build manually step by step. If you can explain the differences I mentioned above between your post build steps and the official docs it will immensely help. Sorry for the long post.

    Thanks

    • Second question, since I will be replacing some dtbs from the carrier board (Auvudea J120) vendor in the Linux_for_tegra/kernel/dtb folder, now if I build the kernel as well (for realsense) how can I merge the dtbs from the kernel build and ones provided by the carrier board vendor? I am stuck. Sorry if it sounds stupid but like I said this is my very first project in Linux and Jetson so most of the time I am not sure what to do ๐Ÿ™‚

      • You should probably ask for guidance from Auvidea or ask this question on the official NVIDIA Jetson forum where a large number of developers and NVIDIA engineers share their experience. The forum is here: https://devtalk.nvidia.com/default/board/188/jetson-tx2/

        To be clear, this is not a good project for an introduction to Linux or the Jetson ๐Ÿ˜‰ You can probably use the D435 without recompiling the kernel until you hit some of the bottlenecks outlined in the article.

        • I have tried Auvidea support, its not good. They don’t respond. NVIDIA forums are great but the opinions are so mixed up that its taking me time to decipher everything, but I am doing it.

          I agree its not a good starting point but I don’t really have an option. It is my job requirement so have to deal with it ๐Ÿ™‚

          Yea I think I will go ahead without recompiling the kernel for now and first get everything running, the carrier board and D435.

          One last question, If I don’t rebuild the kernel, the librealsense would still work with CUDA enhancements? I know it has got nothing to do with the kernel but better to ensure.

          Thanks

          • Your assessment is correct. The kernel modifications have to do with making a wider variety of video modes available in user space, and support for hardware which is not present on the D435.

            • Hi Again,

              I thought to give it a try and build the kernel for learning purposes. I am cross-compiling it on a non-tegra machine. I am following your configure and patch scripts. Here are the steps

              1- Downloaded the kernel source and librealsense source
              2- extracted current .config from TX2, moved it to host and configured it using your configure script
              3- Patched kernel using the patchkernel script
              4- ran make module_prepare, make module, make module_install. It all went fine.
              5- ran make Image and it failed with following message

              LD init/built-in.o
              drivers/built-in.o: In function `nvs_remove’:
              /home/nouman/tegra-build/kernel/kernel-4.4/drivers/misc/nvs/nvs_iio.c:1804: undefined reference to `devm_iio_kfifo_free’
              drivers/built-in.o: In function `nvs_init’:
              /home/nouman/tegra-build/kernel/kernel-4.4/drivers/misc/nvs/nvs_iio.c:1846: undefined reference to `devm_iio_kfifo_allocate’
              /home/nouman/tegra-build/kernel/kernel-4.4/Makefile:959: recipe for target ‘vmlinux’ failed
              make[1]: *** [vmlinux] Error 1
              make[1]: Leaving directory ‘/home/nouman/tegra/kernel’
              Makefile:150: recipe for target ‘sub-make’ failed
              make: *** [sub-make] Error 2

              Now the errors above look like they are related to the modules enables in the .config file earlier. Can you spot any obvious issues with my process? I remember while updating the .config file, there were modules related to IIO KFIFO BUFFER and the errors are to do with undefined references to iio_kfifo_buffer allocate and free. I looked through the source files and I couldn’t find these references defined anywhere. However, I did find them declared in one of the .h files included in main .c file where the error occured.

              Any pointers you can throw at me would be highly appreciated.

              Thanks
              Nouman.

    • Not quite sure I understand the question. The compiled linux kernel is the file “Image”. The Linux_for_Tegra/rootfs is a direct disk image of what will be placed on the Jetson during flashing. Compiling the kernel from source will build a new Image file which replaces the Image file on the disk image (ie Linux_for_Tegra/rootfs/boot/Image). On a running system, this is simply the file /boot/Image.

      modules_install is a script that is part of the kernel build process which basically copies the module files into /lib/modules/(name of the kernel)/kernel You can find these on a host in Linux_for_Tegra/rootfs/lib/modules/(name of the kernel) where name of the kernel is something like ‘4.4.38-tegra’. On a running system, this is simply /lib/modules/(name of the kernel).

      To be clear, JetPack is the name of the program/installer that flashes the disk image to the Jetson. L4T is the name of the operating system, which is an Ubuntu variant.

      The dtbs in the more recent Jetson releases is not stored on the rootfs partition. Instead, it is encoded and placed in a separate device tree partition on the disk. Currently the encoding is done using an x86 binary, so it cannot be done on the Jetson itself. During the flashing process, JetPack copies the compiled device tree files to the device tree partition. There is a command line script utility which you can use on the host to flash only the device tree partition, I believe that it is called “flash” and requires some command line parameters and flags to work correctly.

      Hope this helps.

    • Thank you for the kind words. As far as I know, librealsense is only a driver for the camera, and does not perform the application specific functions that you mention. Thanks for reading!

Leave a Reply

Your email address will not be published.


*