Recently the Intel Librealsense Development Team added CUDA support to the Librealsense SDK. Here’s how to install it on the Jetson TX Dev kits. Looky here:
As you may recall, we added CUDA support to librealsense in a previous article, Now with CUDA! Intel RealSense D400 cameras – NVIDIA Jetson TX. Recently I received a nice note from the Librealsense Development Team stating that they had used the code as a basis for adding CUDA support to librealsense (starting in version 2.13.0).
Intel took a look at the performance (from https://github.com/IntelRealSense/librealsense/pull/1866):
Initial CUDA support for conversions and pointcloud generation.
As tested on the Jetson TX2:
Description Resolution fps No Cuda, avg. (ms) With Cuda, avg. (ms) YUY2 into RGB8 1920×1080 30 30 – 32 5-7 YUY2 into RGBA8 1920×1080 30 34 7.5 YUY2 into BGR8 1920×1080 30 35 5 – 7 YUY2 into BGRA8 1920×1080 30 37 7 – 10 YUY2 into Y16 1920×1080 30 10 5 – 8 YSplitting Y8I to Y8, Y8 1280×800 30 9 – 10 3 – 5 Splitting Y12 to Y16, Y16 1280×800 25 22.9 5 – 7 Pointcloud (deproject depth) 640×480 30 8 – 9 3 – 5
In order to activate GPU full power run: sudo ./jetson_clocks.sh
You can see that there is a 4-7x speedup on the CPU only versus GPU code on the Jetson. The term fps means frames per second. Remember that when you are running 30 fps, each frame arrives about every 33ms.
Note: Because space is tight on the Jetson TX1, you should first flash the Jetson TX1 with L4T 28.2 using JetPack 3.2.1, download the buildLibrealsense2TX repository, and then build the patched kernel. Once the kernel is in place, remove the kernel sources to give you enough space to operate.
On the JetsonHacks Github account, there is a repository buildLibrealsense2TX. To download the repository:
$ cd $HOME
$ git clone https://github.com/jetsonhacks/buildLibrealsense2TX
$ cd buildLibrealsense2TX
$ git checkout v0.9
The instructions are much the same as in the previous article. More details are available on the README.md file in the buildLibrealsense2TX directory.
First, a word about what we’re doing. There are several RealSense camera formats that the standard UVC (video) module in the kernel does not support. librealsense provides patches to add those video modes to the appropriately. The patches also add support for properly adding timestamps to the incoming video stream. In addition, there are several patches that modify some of the Industrial I/O (IIO) tree. These patches are providing support to supplemental hardware on a RealSense camera, such as a 3D Gyro or 3D Accelerometer.
Note: Some of the patches apply to modules that are built into the kernel itself. Because these modules are required to be in the Image, and not built as external, you will need to compile the kernel itself along with the modules.
If you do not modify the Linux kernel, librealsense will mostly appear to work. You will probably experience issues a little further along when you are trying to get better precision and accuracy from the video and depth streams. As the RealSense D435 camera does not have supplemental hardware, the IIO patches don’t make any difference.
If you’re just looking to play around with the camera, you may be able to get away with not compiling the kernel and skip over it. If you’re more serious, you’ll have to start patchin’ !
Building the Kernel
Note: If you built your kernel as in the previous article, you must rebuild it again!
Building the kernel can be taxing, there are many little things that can go wrong. Plus, you can make your Jetson become brickly if something happens at an untoward momement. If you’re a go for it kind of person:
If something does go wrong during the build, you may want to try to debug it. As part of its cleanup process, the buildPatchedKernel script erases all of the source files and build files that it has downloaded and built. You can pass nocleanup as a command line flag so it keeps those files around. Hopefully you can fix everything.
$ ./buildPatchedKernelTX.sh –nocleanup
Actually, this script is more useful as a template for rebuilding your own kernel with the librealsense changes. There are two scripts in the ‘scripts’ directory for helping in the kernel build:
patchKernel.sh applies the librealsense kernel patches.
configureKernel.sh Configures the kernel to add the appropriate modules needed by librealsense.
Since you’re a developer, you should be able to figure out what the scripts are doing and modify them to match your needs.
make sure that the RealSense camera is not attached to the system. Then install the library:
The script does several things:
- Installs dependencies
- Builds CMake
- Builds librealsense and associated apps
The librealsense build system requires CMake version 3.8 or above. The standard Jetson TX version of CMake is 3.5.1 for L4T 28.2.x. Therefore the script builds CMake (in the home folder). However, the script does not install CMake into the system area, it just uses the new version directly from the build fold. Of course, you can install the new version of CMake on your system if so desired.
After the script compiles the library, the new files are placed in the following directories:
- The library is installed in /usr/local/lib
- The header files are in /usr/local/include
- The examples and tools are located in /usr/local/bin
The script also sets up a udev rule so that the RealSense camera is available in user space.
Once the library is installed, plug the camera into the Jetson, or into the Jetson through a powered USB 3.0 hub. You can then go and execute the tools and examples. For example:
$ cd /usr/local/bin
It is nice to have the CUDA upgrade for librealsense in the official repository. Because the RealSense SDK is under heavy development, we will have to keep our eye out for improvements in the weeks ahead!
- In the video, a Jetson TX2 is shown
- L4T 28.2.1 installed by JetPack 3.2.1
- buildLibrealsense2TX is version v0.9