Intel RealSense Camera Installation – NVIDIA Jetson TX1

Intel RealSense cameras can use an open source library called librealsense as a driver for the Jetson TX1 development kit. Looky here:

Background

Note: This article is intended for intermediate users who are comfortable with Linux kernel development, and can read and modify simple shell scripts if needed.

In earlier articles we talked about the Intel RealSense R200 Camera, which is a relatively inexpensive RGBD device in a compact package. The camera uses USB 3.0 to communicate with a computer.

Intel has made available an open source library, librealsense on Github. librealsense is a cross platform library which allows developers to interface with the RealSense family of cameras, including the R200. Support is provided for Windows, Macintosh, and Linux.

There are two major parts to getting the R200 camera to work with the Jetson. First, operating system level files must be modified to recognize the camera video formats. When doing development on Linux based machines you will frequently hear the terms “kernel” and “modules”. The kernel is the code that is the base of the operating system, the interface between hardware and the application code.

A kernel module is code that can be accessed from the kernel on demand, without having to modify the kernel. These modules provide ancillary support for different types of devices and subsystems.

A module is compiled code which is stored as a file separately from the kernel, typically with a .ko extension. The advantage of having a module is that it can be easily changed without having to modify the entire kernel. We will be building a module called uvcvideo to help interface with the RealSense camera. Normally uvcvideo is built-in to the kernel, we will designate it as a module as part of our modification. We will modify uvcvideo to recognize the RealSense camera data formats.

The second part of getting the R200 to work with the Jetson TX1 is to build and install librealsense.

Kernel and Module Building

Note: In the video above, the installation was performed on a newly flashed L4T 24.2.1 TX1 using JetPack 2.3

We have covered building the kernel for the Jetson TX1 in a previous article. Here are the major steps involved:

The script files to build the kernel on the Jetson TX1 are available on the JetsonHacks Github account in the buildJetsonTX1 repository.

$ git clone https://github.com/jetsonhacks/buildJetsonTX1Kernel.git
$ cd buildJetsonTX1Kernel

The script getKernelSources.sh gets the kernel sources from the NVIDIA developer website, then unpacks the sources into /usr/src/kernel.

$ ./getKernelSources.sh

After the sources are installed, the script opens an editor on the kernel configuration file. In the video, the local version of the kernel is set. The stock kernel uses -tegra as its local version identifier. Make sure to save the configuration file when done editing.

Next, patchAndBuildKernel.sh, patches one of the sources files to more easily compile the kernel:

$ ./patchAndBuildKernel.sh

and proceeds to build the kernel and modules using make. The modules are then installed in /lib/modules/3.10.96[local version name]

Install librealsense

A convenience script has been created to help with this task in the installLibrealsense repository on the JetsonHacks Github account.

$ cd $HOME
$ git clone https://github.com/jetsonhacks/installLibrealsenseTX1.git
$ cd installLibrealsenseTX1
$ ./installLibrealsense.sh

This will build the librealsense library and install it on the system. This will also setup udev rules for the RealSense device so that the permissions will be set correctly and the camera can be accessed from user space.

USB Video Class Module

Note: This step assumes that the kernel is located in /usr/src/kernel and that the kernel is to be installed on the board. If this is not your intent, modify the script accordingly. applyUVCPatch.sh has the command to patch the UVC driver with the RealSense camera formats.

The third major step is to build the USB Video Class (UVC) driver as a kernel module. This is can be done using the script:

$ ./buildPatchedKernel.sh

The buildPatchedKernel script will modify the kernel .config file to indicated that the UVC driver should be built as a module. Next, the script patches the UVC driver to recognize the RealSense camera formats. Finally the script builds the kernel, builds the kernel modules, installs the modules and then copies the kernel image to the /boot directory.

Note: The kernel and modules should have already been compiled once before performing this step. Building the kernel from scratch as described in the ‘Kernel and Module Building’ section above pulls a few shenanigans to get things to build properly the first time through.

One more minor point of bookkeeping. In order to save power, the Jetson TX1 will auto-suspend USB ports when not in use for devices like web cams. This confuses the RealSense camera. In order to turn auto-suspend off, run the following script:

$ ./setupTX1.sh

Once finished, reboot the machine for the new kernel and modules to be loaded.

Conclusion

So there you have it. This has been a little bit more involved than some of our other projects here, but if you are interested in this kind of device, well worth it.

Notes

There are several notes on this project:

  • In the video above, the installation was done on a Jetson TX1 running L4T 24.2.1 immediately after being flashed by JetPack 2.3
  • These scripts only support the 64-bit L4T series, 24.X
  • One difference between L4T 24.2 and L4T 24.2.1 is that a soft link issue with Mesa drivers has been resolved. If you are using L4T 24.2, you may have to:

    $ cd /usr/lib/aarch64-linux-gnu
    $ sudo rm libGL.so
    $ sudo ln -s /usr/lib/aarch64-linux-gnu/tegra/libGL.so libGL.so

  • QtCreator and Qt 5 are installed as dependencies in the librealsense part of the install. There are QtCreator project files located in the librealsense.qt directory. The project files build the library and example files. If you do not use QtCreator, consider modifying the installer script to take QtCreator out.
  • The librealsense examples are located in librealsense/bin after everything has been built.
  • These scripts install librealsense version v1.11.0 (last commit 74ff66da50210e6b9edc3157411bad95c209740f)
  • The RealSense R200 is the only camera tested at this time.
  • Examples using librealsense are located in librealsense/bin

24 Comments

  1. Hi,

    We have successfully made the streaming of Intel RealSense SR300 camera in L4T R24.1 by modifying the script and L4T R24.2 using the steps given above.

    Thanks kangalow for the detailed steps provided.

  2. Firstly, thanks for the detail explanation!

    However, when I follow the steps, and connect the Intel RealSense R200 to usb3.0 port of Astro carrier board for jetson TX1. The dmesg shows “can not enable port 2. Maybe USB cable bad?” The board can recognize normal usb webcam, but not the RealSense.

    Since it works well on windows, so I think is not the cable problem. Any suggestions is appreciated!!

  3. Hi,
    Unfortunately I do not have any experience with the Astro Carrier Board. If you have it plugged into the USB 3.0 port, it should enumerate properly assuming that USB 3.0 is enabled. Hope you get it figured out!

    • Thanks for your quick reply!

      I found that in the video the uvcvideo is auto loaded at the start up. However, in my case, I have to run “sudo modprobe uvcvideo” manually after reboot.

      (I am using the method stated in http://qiita.com/furushchev/items/a95acd598b7106fd52ba to boot from SATA SSD. It seems that jetson TX1 will first examine the extconfig inside bootable emmc partition, so that I have to manually copied the generated kernel to /media/ubuntu/xxxxx/boot to boot the correct kernel.)

      I am not sure if it is normal that we should manually loading uvcvideo driver?

      Thanks in advance!!

      • In the method above, the UVC Video driver is marked as an external module and built. After it is built, it is copied to the correct place in the driver file hierarchy. As you note, it is a file called: ‘uvcvideo.ko’. The local version of the OS determines placement, so it’s important that the uvcvideo module be built using the same local version as the kernel that is being used.

        With that said, is it possible that you have ucvideo.ko in the wrong place? I’m not sure where all of the drivers are loaded from when booting from the SSD, but it may be that the uvcvideo.ko may not be seen during the booting process. When booting from the internal flash as shown in the examples, it just loads like all of the other modules.

        • I’m getting the same issue, but I am using the TX1 breakout board. I followed the steps exactly as you laid out here, but the module is not loaded at boot, so the R200 does not activate. I double-checked the module location, and it is in drivers/media/usb/uvc/, which looks right. Is there something obvious that I might have overlooked?

          • Thanks for the response! The camera is indeed plugged in at boot. I wound up trying to use JetPack 2.3 (instead of 2.3.1, the latest) and it appeared that the output of my build was the same as yours. I occasionally get different errors (in addition to the one above) now, but it appears that whether the uvcvideo module loads at boot is not consistent.

            uname -a:
            Linux tegra-ubuntu 3.10.96- #2 SMP PREEMPT Sun Feb 26 02:05:44 UTC 2017 aarch64 aarch64 GNU/Linux

            When the module decides to load, it recognizes the camera as the R200, but fails when it attempts to create a debugfs directory. I verified that the udev was set correctly, adn the rule was added to rules.d properly (though the executable that was added does not exist in /usr/local/bin and adding it does not change anything).

          • It looks like I should have used a different delimiter for the local version, but in the output of uname -a above should be 3.10.96-(local-version).

          • I would advise having a local delimiter of some sort, remember that it must begin with a dash ( ‘-‘ ).

            Have you turned off USB autosuspend? If the USB autosuspend is on, things tend to get tricky.

            What executable are you looking for in /usr/local/bin ?

            Do you get the same results when you do a plug/replug on the R200, and what type of USB hub are you using?

          • Exactly–I used angle brackets to denote the local version, but they got swallowed up by the commenting system. The local version actually looks like 3.10.96-test, or something similar.

            I turned off USB autosuspend using setupTX1.sh, and looking at the file in /boot/, it is set to -1. I have also tried setting the usb_owner_info to 2 instead of 0, but that did not appear to change anything.

            The executable is /usr/local/bin/usb-R200-in, which is what is specified by the rule in 99-realsense. It might be the other one, I’m going from memory on this. This is from the librealsense script install-r200-udev-fix.sh.

            I’m not using a hub–I’ve connected the camera directly into the USB port, and I have a mouse+keyboard combo going into the other USB port.

            When I do a plug/replug, it does do the same thing (can not enable port 2. Maybe USB cable bad), but the error also varies between the debugfs directories, “device not accepting address”, and the one above.

            However, the debugfs error only appears when the uvcvideo module loads at boot, which it does not always do. When the above error occurs, lsmod almost always shows no uvcvideo loaded.

          • Ah, the html thingy. I hate that about comments on here.
            As for the /usr/local/bin executables, I didn’t copy them over and don’t use them. I seem to recall there being issues if they were installed.

            I rebuilt a kernel yesterday following the instructions above. Fortunately it’s been long enough that I’ve forgotten the process and need to do it step by step. It worked fine here, so I don’t know how to fix the issue that you’re having.

            I did you use USB 3.0 hub yesterday, unfortunately I had to reflash the system for another project or I would have tried it straight into the USB plug now. I seem to remember that it worked when plugged directly into the port, but it’s been a while since I actually wrote this article and thankfully I forget such things rather easily.

            At this point I can’t think of anything else that may be causing the issue. Hopefully you can hunt it down on your own and let us know if you find a fix.

          • No worries–thank you for talking it through with me! I’ll try the step-by-step instructions again and see what happens. One last thing that I thought of: do you know of any firmware updates on the R200 itself that might cause incompatibility with the older versions of the kernel (since the steps call for JetPack 2.3 instead of 2.3.1)?

            The only other thing I can think of is that the scripts grab 24.2 even though they are built with 24.2.1 (and the github repo suggests that the commands should be run after flashing with 24.2 instead of 24.2.1 as in the video). Would that be anything to worry about?

        • I have not updated the R200 firmware since I received it, so that may be a difference.

          I should update the kernel script, but since both 24.2 and 24.2.1 report the same kernel name I always just thought that they were the same. I’ve used both .2 and .2.1 and not seen any difference with the 24.2 kernel.

          Unfortunately this whole process is a lot like the children’s show ‘Sesame Street’ where you play “One of these things is not like the other, one of these things just doesn’t belong” Getting this to work in the first place was a multi-week process, I’ve been hesitant to change or update it since then since it’s so much work.

          • Absolutely agree with the analogy! I will keep poking around, and report back if I find anything that could contribute to why it won’t work (with the exact same setup). Thank you for your help!

          • For anyone else who might have this problem: try using a powered USB3 hub. After doing some digging around the internet, some have written that the R200 required the addition of an external hub (with its own power supply). That seemed to do the trick for me!

  4. when I run buildJetsonTX1Kernel/scripts/installKernelSources.sh
    at the end it doesnt open the terminal.
    I have done this before and it worked any idea on how to open this manually?
    thanks heaps for the videos

Leave a Reply

Your email address will not be published.


*