NOTE: There is an updated article for use with L4T 28.1 (JetPack 3.1) and above. Please see: Intel RealSense Camera librealsense – NVIDIA Jetson TX Dev Kits
Intel RealSense cameras can use an open source library called librealsense as a driver for the Jetson TX2 development kit. Looky here:
Note: This article is intended for intermediate users who are comfortable with Linux kernel development, and can read and modify simple shell scripts if needed.
In earlier articles we talked about the Intel RealSense R200 Camera, which is a relatively inexpensive RGBD device in a compact package. The camera uses USB 3.0 to communicate with a computer.
Intel has made available an open source library, librealsense on Github. librealsense is a cross platform library which allows developers to interface with the RealSense family of cameras, including the R200. Support is provided for Windows, Macintosh, and Linux.
There are two major parts to getting the R200 camera to work with the Jetson. First, operating system level files must be modified to recognize the camera video formats. When doing development on Linux based machines you will frequently hear the terms “kernel” and “modules”. The kernel is the code that is the base of the operating system, the interface between hardware and the application code.
A kernel module is code that can be accessed from the kernel on demand, without having to modify the kernel. These modules provide ancillary support for different types of devices and subsystems.
A module is compiled code which is stored as a file separately from the kernel, typically with a .ko extension. The advantage of having a module is that it can be easily changed without having to modify the entire kernel. We will be building a module called uvcvideo to help interface with the RealSense camera. Normally uvcvideo is built-in to the kernel, we will designate it as a module as part of our modification. We will modify uvcvideo to recognize the RealSense camera data formats.
The second part of getting the R200 to work with the Jetson TX2 is to build and install librealsense.
Kernel and Module Building
Note: In the video above, the installation was performed on a newly flashed L4T 27.1 TX2 using JetPack 3.0. The kernel sources were downloaded and built before shooting the video.
In the previous article, we built a new kernel for the Jetson TX2. Please refer to the article for the details. Once completed, you’re ready to install librealsense..
A convenience script has been created to help with this task in the installLibrealsense repository on the JetsonHacks Github account.
$ cd $HOME
$ git clone https://github.com/jetsonhacks/installLibrealsenseTX2.git
$ cd installLibrealsenseTX2
This will build the librealsense library and install it on the system. This will also setup udev rules for the RealSense device so that the permissions will be set correctly and the camera can be accessed from user space.
USB Video Class Module
Note: This step assumes that the kernel sources are located in /usr/src/kernel and that the kernel is to be installed on the board. If this is not your intent, modify the script accordingly. applyUVCPatch.sh has the command to patch the UVC driver with the RealSense camera formats.
The third major step is to build the USB Video Class (UVC) driver as a kernel module. This is can be done using the script:
The buildPatchedKernel script will modify the kernel .config file to indicated that the UVC driver should be built as a module. Next, the script patches the UVC driver to recognize the RealSense camera formats. Finally the script builds the kernel, builds the kernel modules, installs the modules and then copies the kernel image to the /boot directory.
Note: The kernel and modules should have already been compiled once before performing this step.
One more minor point of bookkeeping. In order to save power, the Jetson TX2 will auto-suspend USB ports when not in use for devices like web cams. This confuses the RealSense camera. In order to turn auto-suspend off, run the following script:
Once finished, reboot the machine for the new kernel and modules to be loaded.
Examples are located in the directory /librealsense/build/examples
Note: If you intend on using the RealSense camera with Robot Operating System (ROS), you may find this article useful: Intel RealSense Package for ROS on NVIDIA Jetson TX2
So there you have it. This has been a little bit more involved than some of our other projects here, but if you are interested in this kind of device, well worth it.
There are several notes on this project:
- In the video above, the installation was done on a Jetson TX2 running L4T 27.1 immediately after being flashed by JetPack 3.0
- QtCreator and Qt 5 are installed as dependencies in the librealsense part of the install. There are QtCreator project files located in the librealsense.qt directory. The project files build the library and example files. If you do not use QtCreator, consider modifying the installer script to take QtCreator out.
- The librealsense examples are located in librealsense/build/examples after everything has been built.
- These scripts install librealsense version v1.12.1 (last commit 7332ecadc057552c178addd577d24a2756f8789a)
- The RealSense R200 is the only camera tested at this time.
- Issues have been reported on the Jetson TX2 if the camera is plugged directly into the USB 3.0 port. The workaround is to use a powered hub. It’s unknown at this time if the TX2 suffers from the same issue.
- The RealSense cameras are USB 3.0 devices. If the camera is attached to a USB hub, the USB hub must support USB 3.0
- Intel RealSense Stereoscopic Depth Cameras is a comprehensive overview of the stereoscopic Intel RealSense RGBD imaging systems