Intel RealSense Camera Installation – NVIDIA Jetson TX2

NOTE: There is an updated article for use with L4T 28.1 (JetPack 3.1) and above. Please see: Intel RealSense Camera librealsense – NVIDIA Jetson TX Dev Kits

Intel RealSense cameras can use an open source library called librealsense as a driver for the Jetson TX2 development kit. Looky here:


tx

Background

Note: This article is intended for intermediate users who are comfortable with Linux kernel development, and can read and modify simple shell scripts if needed.

In earlier articles we talked about the Intel RealSense R200 Camera, which is a relatively inexpensive RGBD device in a compact package. The camera uses USB 3.0 to communicate with a computer.

Intel has made available an open source library, librealsense on Github. librealsense is a cross platform library which allows developers to interface with the RealSense family of cameras, including the R200. Support is provided for Windows, Macintosh, and Linux.

There are two major parts to getting the R200 camera to work with the Jetson. First, operating system level files must be modified to recognize the camera video formats. When doing development on Linux based machines you will frequently hear the terms “kernel” and “modules”. The kernel is the code that is the base of the operating system, the interface between hardware and the application code.

A kernel module is code that can be accessed from the kernel on demand, without having to modify the kernel. These modules provide ancillary support for different types of devices and subsystems.

A module is compiled code which is stored as a file separately from the kernel, typically with a .ko extension. The advantage of having a module is that it can be easily changed without having to modify the entire kernel. We will be building a module called uvcvideo to help interface with the RealSense camera. Normally uvcvideo is built-in to the kernel, we will designate it as a module as part of our modification. We will modify uvcvideo to recognize the RealSense camera data formats.

The second part of getting the R200 to work with the Jetson TX2 is to build and install librealsense.

Kernel and Module Building

Note: In the video above, the installation was performed on a newly flashed L4T 27.1 TX2 using JetPack 3.0. The kernel sources were downloaded and built before shooting the video.

In the previous article, we built a new kernel for the Jetson TX2. Please refer to the article for the details. Once completed, you’re ready to install librealsense..

Install librealsense

A convenience script has been created to help with this task in the installLibrealsense repository on the JetsonHacks Github account.

$ cd $HOME
$ git clone https://github.com/jetsonhacks/installLibrealsenseTX2.git
$ cd installLibrealsenseTX2
$ ./installLibrealsense.sh

This will build the librealsense library and install it on the system. This will also setup udev rules for the RealSense device so that the permissions will be set correctly and the camera can be accessed from user space.

USB Video Class Module

Note: This step assumes that the kernel sources are located in /usr/src/kernel and that the kernel is to be installed on the board. If this is not your intent, modify the script accordingly. applyUVCPatch.sh has the command to patch the UVC driver with the RealSense camera formats.

The third major step is to build the USB Video Class (UVC) driver as a kernel module. This is can be done using the script:

$ ./buildPatchedKernel.sh

The buildPatchedKernel script will modify the kernel .config file to indicated that the UVC driver should be built as a module. Next, the script patches the UVC driver to recognize the RealSense camera formats. Finally the script builds the kernel, builds the kernel modules, installs the modules and then copies the kernel image to the /boot directory.

Note: The kernel and modules should have already been compiled once before performing this step.

One more minor point of bookkeeping. In order to save power, the Jetson TX2 will auto-suspend USB ports when not in use for devices like web cams. This confuses the RealSense camera. In order to turn auto-suspend off, run the following script:

$ ./setupTX1.sh

Once finished, reboot the machine for the new kernel and modules to be loaded.

Examples are located in the directory /librealsense/build/examples

Note: If you intend on using the RealSense camera with Robot Operating System (ROS), you may find this article useful: Intel RealSense Package for ROS on NVIDIA Jetson TX2

Conclusion

So there you have it. This has been a little bit more involved than some of our other projects here, but if you are interested in this kind of device, well worth it.

Notes

There are several notes on this project:

  • In the video above, the installation was done on a Jetson TX2 running L4T 27.1 immediately after being flashed by JetPack 3.0
  • QtCreator and Qt 5 are installed as dependencies in the librealsense part of the install. There are QtCreator project files located in the librealsense.qt directory. The project files build the library and example files. If you do not use QtCreator, consider modifying the installer script to take QtCreator out.
  • The librealsense examples are located in librealsense/build/examples after everything has been built.
  • These scripts install librealsense version v1.12.1 (last commit 7332ecadc057552c178addd577d24a2756f8789a)
  • The RealSense R200 is the only camera tested at this time.
  • Issues have been reported on the Jetson TX2 if the camera is plugged directly into the USB 3.0 port. The workaround is to use a powered hub. It’s unknown at this time if the TX2 suffers from the same issue.
  • The RealSense cameras are USB 3.0 devices. If the camera is attached to a USB hub, the USB hub must support USB 3.0
  • Intel RealSense Stereoscopic Depth Cameras is a comprehensive overview of the stereoscopic Intel RealSense RGBD imaging systems

42 Comments

  1. This article and code is insanely useful. Thank you!

    From a fresh TX2 with JetPack 3.0, L4T 27, Ubuntu 16.04 host/TX2, full install of pkgs like cuda, opencv etc. I went through the instructions to
    -Build TX2 Kernel
    -Install librealsense
    -Install ROS

    It may be helpful to also link to the Realsense ROS article here, because I missed that as the 4th step originally. The resulting error message about uvcvideo input/output error did not make it obvious that I needed to install realsense for ROS, but that was the solution.

    I got 2 R200s running on my TX2 simultaneously with ROS, using those 4 articles.

  2. My primary purpose for this setup is playing with drones. For the video that you did outside, showing guys waving arms and such, was that done during the day or the night? At what kind of range could you make the point cloud work?
    Lastly, could you use the IR cameras to actually get IR pictures and that you could derive temperatures from? That would be very cool.
    Mine are on order and are supposed to be here within a couple of weeks.
    Truly fascinating stuff!

  3. I don’t remember doing a RealSense Camera video outside. Someone else perhaps?
    You can get the IR images from libRealsense , you have to do the calculations to convert them to temperature information.

  4. Hey, thanks for the article. Would be nice if you could change the buildPatchKernel to take both “Y” and “y”, I could not understand why it kept saying kernel changes canceled until I opened the source and found out it had to be capital xD

  5. Are all your outstanding scripts designed to be run on the host or, for example, does installing librealsense occur only on the TX2? Does librealsense work on the host (in Ubuntu?)

    • Most of the scripts here are for running on the Jetson Dev Kits. librealsense has instructions for building/installing on the host pc. Thanks for reading!

  6. Hi,
    we are using the SR300 and your guide is sadly not working. We don’t get the RGB image and the IR image looks wrong. We got the SR300 working on TX1 but just can’t seem to get it working on TX2. Any idea what could be wrong? We would love a guide for the SR300 🙂

  7. Unfortunately I do not have a SR300, so it’s difficult to help. Are you using the same USB hub as you did with the TX1? That seems to be one of the issues that you can run into.

  8. There is some kind of bug in the usb stack as seen in the logs when using a realsense R200:

    [ 81.899869] USB Video Class driver (1.1.1)
    [ 89.644529] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 5
    [ 89.651540] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work ignore firmware MBOX_CMD_DEC_SSPI_CLOCK request
    [ 112.790658] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 4
    [ 112.797636] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work ignore firmware MBOX_CMD_INC_SSPI_CLOCK request

    • This is certainly good news! The process noted here is much too difficult for casual users. Thanks for the heads up, I’ll have to update the article once I get everything working.

  9. So to clarify, am I right in thinking that Jetpack3.1 on TX2 will already recognise the R200 camera? But to also have the examples working requires a rebuild of the kernel (as per previous article), but no further patches are required?

    Now on my second day on this, but determined to get it working! My experience is that the kernel build goes well but I get an error ‘Hunk#1 failed at 621’ when I try to apply the patch as above. This might explain why, of course.

    As always, a thousand thanks to those who contribute!

    • Hi Mike,
      My current understanding is that no modifications to the kernel are needed.
      There is a patch that is needed so that librealsense recognizes the internal UVC module: https://github.com/IntelRealSense/librealsense/pull/442/files
      Another difference between newer versions of librealsense and this version is that newer versions use CMake. You should be able to follow the librealsense Ubuntu installation directions and get everything working fairly easily once you apply the patch in the librealsense library code. Good luck!

  10. Great. So for info on a completely fresh install of Jetpack 3.1, dmesg shows the camera is detected:

    [ 7793.739594] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
    [ 7793.746786] usb 2-1: Product: Intel RealSense 3D Camera R200
    [ 7793.752496] usb 2-1: Manufacturer: Intel Corp

    I’m pretty new to working with Ubuntu in this way, but I’ll try to do the patch and go from there My main focus is to be able to get on with using the R200 from within ROS… ho hum!

    Your advice has been fantastic – thanks.

  11. So I started with a fresh install of jetpack 3.1 and modified the librealsense code as in pull request 442 above. lsmod lists uvcvideo but examples give an error ‘kernel module not loaded’. I tried again but with a rebuilt kernel with acm and ch341 as modules as per your recent video. Same result.

    I think I have reached the limit of my expertise – I feel so close yet so far still!! I’ll have to wait now and hope you will do another of your marvellous videos!

    Cheers, Mike

    • I played with it last night, the patch isn’t quite right for the TX2. I’ll be updating the repository, but if you want to play with it sooner all you have to do is delete the part where it’s checking for the uvcvideo module. You know it is built into the kernel, so no need to check for it. An alternative is to comment out the place where it throws the exception.

  12. I am not able to find . buildPatchedKernel.sh file on librealsense git repository ,can I neglect that step? If no what should I do?

  13. Hi, thanks for your amazing video.
    I just flash my TX2 with JetPack3.1 . When I connect the RealSense R200 it seems it isn’t recognized. I Doesn’t appear in lsusb and dmesg output. Is it possible that ovcvideo module/part of kernel is not installed properly?
    the following output is making me suspicious:
    dmesg | grep uvcvideo
    [ 6936.890131] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
    [ 6936.897746] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
    [ 6936.897748] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
    [ 7794.114136] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
    [ 8394.921380] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
    [ 8394.931965] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’
    [ 8394.943546] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’

    If I try to run example available in build/example folder, I got the following error:
    what(): uvcvideo kernel module is not loaded
    I am trying to understand if it is a kernel problem or a patch problem or what else. Any advise?
    PS: camera is working on other ubuntu machines

  14. I flash with jetpack 3.1, than I build the kernel with your tutorial without modifications to config file. Went through the 3 steps and reboot.

    • There is a note in the first line of this article. It reads: Note: This article has been updated for use with L4T 28.1+. Please see: Intel RealSense Camera librealsense – NVIDIA Jetson TX Dev Kits

      You appeared to find this confusing. How can I reword it so that you would go to the new article instead of following the directions in this article?

      The directions in this article are for L4T 27.1 (JetPack 3.0). For L4T 28.1 and above (JetPack 3.1), please refer to the article: https://www.jetsonhacks.com/2017/08/14/intel-realsense-camera-librealsense-nvidia-jetson-tx-dev-kits/ You should not need to recompile the kernel in order to get the R200 working with the Jetson TX2.

  15. Thanks for the support. As I said, it is 3 days I’ve been working and I think I tried all possible way. My first attempt was to flash the TX2 and then install the realsense lib for TX2. Can someone confirm if the uvcvideo module should appear in the lsmod output of a flashed TX2?

    • Hi Laupl0082,
      The RealSense R200 is a USB 3.0 device. First, lsusb should indicate that the device is present:
      ID 8086:0aB0 Intel Corp
      If the R200 is detected, then uvcvideo will show in:
      $ lsmod
      uvcvideo

      My guess is that you are using a USB 2.0 hub. If the R200 does not show up in lsusb, then it does not load the uvcvideo module.

    • lsusb doesn’t show anything about the R200 when I plug it in. No changes between plugged and unplugged.
      the last row in dmesg is the following:
      [ 6936.890131] uvcvideo: version magic ‘4.4.83+ SMP preempt mod_unload aarch64’ should be ‘4.4.38-tegra SMP preempt mod_unload aarch64’

      • It appears as if your kernel does not match your uvcvideo module. Is this a modified kernel?
        There are two issues:
        1) 4.4.83 indicates that you have the wrong kernel number in you module, it should be 4.4.38
        2) There is no local version number set on your uvcvideo module, it should match the kernel you are using (e.g. -tegra)
        I would suggest that you flash the Jetson with JetPack 3.1. Plug the camera in and check to see if it shows up in lsusb, and if the uvcvideo module is loaded. Then you can install librealsense following the instructions on: https://www.jetsonhacks.com/2017/08/14/intel-realsense-camera-librealsense-nvidia-jetson-tx-dev-kits/
        Please report back after you do this.

Leave a Reply

Your email address will not be published.


*