NOTE: This article has been updated for use with L4T 28.1+. Please see: Intel RealSense Camera librealsense – NVIDIA Jetson TX Dev Kits
Intel RealSense cameras can use an open source library called librealsense as a driver for the Jetson TX1 development kit. Looky here:
Note: This article is intended for intermediate users who are comfortable with Linux kernel development, and can read and modify simple shell scripts if needed.
In earlier articles we talked about the Intel RealSense R200 Camera, which is a relatively inexpensive RGBD device in a compact package. The camera uses USB 3.0 to communicate with a computer.
Intel has made available an open source library, librealsense on Github. librealsense is a cross platform library which allows developers to interface with the RealSense family of cameras, including the R200. Support is provided for Windows, Macintosh, and Linux.
There are two major parts to getting the R200 camera to work with the Jetson. First, operating system level files must be modified to recognize the camera video formats. When doing development on Linux based machines you will frequently hear the terms “kernel” and “modules”. The kernel is the code that is the base of the operating system, the interface between hardware and the application code.
A kernel module is code that can be accessed from the kernel on demand, without having to modify the kernel. These modules provide ancillary support for different types of devices and subsystems.
A module is compiled code which is stored as a file separately from the kernel, typically with a .ko extension. The advantage of having a module is that it can be easily changed without having to modify the entire kernel. We will be building a module called uvcvideo to help interface with the RealSense camera. Normally uvcvideo is built-in to the kernel, we will designate it as a module as part of our modification. We will modify uvcvideo to recognize the RealSense camera data formats.
The second part of getting the R200 to work with the Jetson TX1 is to build and install librealsense.
Kernel and Module Building
Note: In the video above, the installation was performed on a newly flashed L4T 24.2.1 TX1 using JetPack 2.3
We have covered building the kernel for the Jetson TX1 in a previous article. Here are the major steps involved:
The script files to build the kernel on the Jetson TX1 are available on the JetsonHacks Github account in the buildJetsonTX1 repository.
$ git clone https://github.com/jetsonhacks/buildJetsonTX1Kernel.git
$ cd buildJetsonTX1Kernel
The script getKernelSources.sh gets the kernel sources from the NVIDIA developer website, then unpacks the sources into /usr/src/kernel.
After the sources are installed, the script opens an editor on the kernel configuration file. In the video, the local version of the kernel is set. The stock kernel uses -tegra as its local version identifier. Make sure to save the configuration file when done editing.
Next, patchAndBuildKernel.sh, patches one of the sources files to more easily compile the kernel:
and proceeds to build the kernel and modules using make. The modules are then installed in /lib/modules/3.10.96[local version name]
A convenience script has been created to help with this task in the installLibrealsense repository on the JetsonHacks Github account.
$ cd $HOME
$ git clone https://github.com/jetsonhacks/installLibrealsenseTX1.git
$ cd installLibrealsenseTX1
This will build the librealsense library and install it on the system. This will also setup udev rules for the RealSense device so that the permissions will be set correctly and the camera can be accessed from user space.
USB Video Class Module
Note: This step assumes that the kernel is located in /usr/src/kernel and that the kernel is to be installed on the board. If this is not your intent, modify the script accordingly. applyUVCPatch.sh has the command to patch the UVC driver with the RealSense camera formats.
The third major step is to build the USB Video Class (UVC) driver as a kernel module. This is can be done using the script:
The buildPatchedKernel script will modify the kernel .config file to indicated that the UVC driver should be built as a module. Next, the script patches the UVC driver to recognize the RealSense camera formats. Finally the script builds the kernel, builds the kernel modules, installs the modules and then copies the kernel image to the /boot directory.
Note: The kernel and modules should have already been compiled once before performing this step. Building the kernel from scratch as described in the ‘Kernel and Module Building’ section above pulls a few shenanigans to get things to build properly the first time through.
One more minor point of bookkeeping. In order to save power, the Jetson TX1 will auto-suspend USB ports when not in use for devices like web cams. This confuses the RealSense camera. In order to turn auto-suspend off, run the following script:
Once finished, reboot the machine for the new kernel and modules to be loaded.
So there you have it. This has been a little bit more involved than some of our other projects here, but if you are interested in this kind of device, well worth it.
There are several notes on this project:
- In the video above, the installation was done on a Jetson TX1 running L4T 24.2.1 immediately after being flashed by JetPack 2.3
- These scripts only support the 64-bit L4T series, 24.X
- One difference between L4T 24.2 and L4T 24.2.1 is that a soft link issue with Mesa drivers has been resolved. If you are using L4T 24.2, you may have to:
$ cd /usr/lib/aarch64-linux-gnu
$ sudo rm libGL.so
$ sudo ln -s /usr/lib/aarch64-linux-gnu/tegra/libGL.so libGL.so
- QtCreator and Qt 5 are installed as dependencies in the librealsense part of the install. There are QtCreator project files located in the librealsense.qt directory. The project files build the library and example files. If you do not use QtCreator, consider modifying the installer script to take QtCreator out.
- The librealsense examples are located in librealsense/bin after everything has been built.
- These scripts install librealsense version v1.11.0 (last commit 74ff66da50210e6b9edc3157411bad95c209740f)
- The RealSense R200 is the only camera tested at this time.
- Examples using librealsense are located in librealsense/bin
- Intel RealSense Stereoscopic Depth Cameras is a comprehensive overview of the stereoscopic Intel RealSense RGBD imaging systems