Intel RealSense Camera Installation – librealsense – NVIDIA Jetson TK1

Installation of the Intel RealSense Cameras on the Jetson TK1 is made possible by the use of the open source library librealsense. Looky here:

Background

Note: This article is intended for intermediate users who are comfortable (or want to be) with Linux kernel development, and can read and modify simple shell scripts if needed.

In a previous post, Intel RealSense Camera, we discussed adding the RealSense R200 camera to the Jetson TK1.

Intel has made available an open source library, librealsense on Github. librealsense is a cross platform library which allows developers to interface with the RealSense family of cameras, including the R200. Support is provided for Windows, Macintosh, and Linux.

There are two parts to getting the R200 camera to work with the Jetson. First, operating system level files must be modified to recognize the camera video formats. When doing development on Linux based machines you will frequently hear the terms “kernel” and “modules”. The kernel is the code that is the base of the operating system, the interface between hardware and the application code.

A kernel module is code that can be loaded into the kernel image at will, without having to modify the kernel. These modules provide ancillary support for different types of devices and subsystems. The code for these modules is either in the kernel itself, in which case it is called ‘built-in’, or designated to built as a module. When built as a module the compiled code is stored separately from the kernel, typically with a .ko extension. The advantage of having a module is that it can be easily changed without having to rebuild the entire kernel. We will be building a module called uvcvideo to help interface with the RealSense camera.

The second part of getting the R200 to work is to build and install librealsense.

Kernel and Module Building

Note: In the video above, the installation was performed on a newly flashed L4T 21.4 TK1 using JetPack 2.2.

In order to build the kernel and the modules, download the kernel sources. The sources should be placed into /usr/src

A convenience script has been created to help with this task in the installLibrealsense repository on the JetsonHacks Github account.

$ git clone https://github.com/jetsonhacks/installLibrealsense.git
$ cd installLibrealsense/UVCKernelPatches
$ ./getKernelSources.sh

The above commands gets the repository and runs a script which downloads the kernel sources. Note: The convenience files are simple shell scripts which automate the tasks described above. They are provided as a guideline on how to accomplish the task, but due to various issues (such as network or directory layout) you may need to modify them to suit your needs.

Once the kernel sources have been downloaded and decompressed into the /usr/src directory, a configuration editor opens. In the configuration editor, set the local version of the kernel to that of the current configuration. The current local version number is available through the command:

$ uname -r

which displays:

3.10.40-gdacac96

The local version number consists of the digits following the 40 in this case, i.e. -gdacac96.
Remember the – sign, it is important! This identifier is used to ensure that the module matches the build of the kernel and should match exactly. Place the local version number in the field:

General Setup -> Local version – append to kernel release:

Next, we will be modify the USB Video Class (UVC) to understand RealSense video formats.

The option to compile UVC as a module is located in:

Device Drivers -> Multimedia Support -> Media USB Adapters -> USB Video Class (UVC)

Once you find the entry, right-click on the entry until you see a small circle. The circle indicates that the option will be compiled as a module. Save the configuration file.

Patching the UVC module

A patch file is provided to apply on the module source and a shell script is provided to apply the patch. Again, these are convenience files, you may have to modify them for your particular situation.

$ ./applyUVCPatch.sh

Next, compile the kernel and module files:

$ ./buildKernel.sh

This takes several minutes as the kernel and modules are built and the modules installed. Once the build is complete, you have a couple of options. The first option is to make a backup of the new kernel and modules to place them on a host system to flash a Jetson system with the new kernel. We will not be covering that here, but for reference:

Build own kernel for Jetson TK1 – NVIDIA Developer Forum

The second option is to copy the kernel over to the boot directory. A convenience script is provided:

$ ./copyzImages.sh

In addition to copying the new kernel into the boot directory, the newly built module, uvcvideo, is added to the file /etc/modules to indicate that the module should be loaded at boot time.

The RealSense cameras require USB 3.0. The USB port is set for USB 2.0 from the factory. Also, the stock kernel uses what is called ‘autosuspend’ to minimize power usage for USB peripherals. This is incompatible with most USB video devices. If you have not changed these settings on your TK1, a convenience script has been provided:

$ ./setupTK1.sh

Now reboot the system.

Building and Installing librealsense

Once the machine has finished rebooting, open a Terminal:

$ cd installLibrealsense
$ ./installLibrealsense.sh

This will build librealsense and install it on the system. This will also setup udev rules for the RealSense device so that the permissions will be set correctly and can be accessed from user space. Once installation is complete, you will be able to play with the examples. For example:

$ cd ~/librealsense/bin
$ ./cpp-config-ui

The example allows you to set the camera parameters. Hit the ‘Start Capture’ button to start the camera.

Qt Creator

There are Qt Creator files in librealsense which may be used to build the examples and librealsense itself. A convenience script, ./installQtCreator.sh , is provided in the installLibrealsense directory to install Qt Creator.

Note

Intel RealSense Stereoscopic Depth Cameras is a comprehensive overview of the stereoscopic Intel RealSense RGBD imaging systems

Conclusion

So there you have it. This has been a little bit more involved than some of our other projects here, but if you are interested in this kind of device, well worth it.

28 Comments

  1. this saved me a ton of time, thanks.

    I’m not sure how to make it work with grinch, but the stock 21.4 kernel worked fine.

  2. I have one on the way. Unfortunately the retrofit kit for my drone won’t be available till later in the year. Don’t want to wait that long to play with this new technology. Thinking about getting another gimbal and mounting the cam on that instead of the regular camera. It would be much better suited to mapping for the rover that way.

  3. Thanks Kangalow!
    Great stuff and save me lots of time as well :).
    Any plans for making it work for TX1?
    It’s now giving me some headake the cross compiling of the kerner in 64arch
    Please let me know if you have any pointers.
    JL

    • Thanks for the kind words. I will need to revisit this for the TX1, I have been waiting for things to settle down a little in the 32/64 bit transition. With the advent of 24.X, me thinks it’s time to get started. It will be a while though, the whole cross compile the kernel, multiple toolchain thing makes my head hurt. I need to gather enough courage to wade in. Thanks for reading!

      • hey mate I’m a huge fan and really appreciate the work you publish. Could you please advise how I can go about setting up a tk1 with a realsense. I have done a bit of research and can see that there is a bit of work to do in turtlebot_description. however what I really need to know is when I install librealsense and ros realsense will I have kernel issues with grinch???

  4. Thanks so much for your work to share your knowledge. Your website has been extremely helpful for me to jump into the world of DIY robotics! Question: can the UVC component of the kernel be set as a module through shell script commands? I would like to automate this step so that if/when I need to re-flash my Jetson, I can do so with minimal effort and also have a full record of the changes I make to the JetPack version. I’ve been Googling this and not found an answer.

  5. There are a couple of ways to do this. The first way is to make your changes to the config file, and then use that in your script as the basis for rebuilding the kernel. One alternative to that is to figure out the text changes needed and write a script to do string substitution.

    A better way is to build up the Jetson with all the programs and settings that you want, and then do what is called ‘clone’ the image. This saves an image of the entire drive with everything installed on to your host system. Then you can flash the Jetson fully built, ready to go. You’ll probably need to rebuild everything occasionally, but by having a working base image you save a lot of time loading all the support packages.
    Thanks for reading!

  6. I followed these instructions above and after I issue the command
    $ ./cpp-config-ui
    I see the CPP Configuration Example Windows appear on screen.

    I click on Start Capture and the entire system locks up and putty from remote notebook loose network connection. After several attempts of reset the problem continues.

    • Please check to see if arm.patch was applied correctly. You’ll need to check the patch with:

      src/uvc-v4l2.cpp

      The code:
      v4l2_buf_type type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
      – for(int i=0; i<10; ++i)
      - {
      - if(xioctl(fd, VIDIOC_STREAMON, &type) < 0) - { - std::this_thread::sleep_for(std::chrono::milliseconds(100)); - } - } results in bad things if it is not patched/removed. If the error: throw_error("VIDIOC_STREAMON") is thrown, the system hard locks.

  7. Hi,

    I followed the instructions after your post in the jetson Tk1 community page and got the following errors,

    INSTALL drivers/video/tegra/host/vi/nvhost-vi.ko
    INSTALL fs/cifs/cifs.ko
    cp: cannot stat ‘fs/cifs/cifs.ko’: No such file or directory
    INSTALL fs/configfs/configfs.ko
    cp: cannot stat ‘fs/configfs/configfs.ko’: No such file or directory
    INSTALL net/bluetooth/bluetooth.ko
    cp: cannot stat ‘net/bluetooth/bluetooth.ko’: No such file or directory
    INSTALL net/bluetooth/bnep/bnep.ko
    cp: cannot stat ‘net/bluetooth/bnep/bnep.ko’: No such file or directory
    INSTALL net/bluetooth/hidp/hidp.ko
    cp: cannot stat ‘net/bluetooth/hidp/hidp.ko’: No such file or directory
    INSTALL net/bluetooth/rfcomm/rfcomm.ko
    cp: cannot stat ‘net/bluetooth/rfcomm/rfcomm.ko’: No such file or directory
    INSTALL net/mac80211/mac80211.ko
    cp: cannot stat ‘net/mac80211/mac80211.ko’: No such file or directory
    INSTALL net/rfkill/rfkill-gpio.ko
    cp: cannot stat ‘net/rfkill/rfkill-gpio.ko’: No such file or directory
    INSTALL net/rfkill/rfkill.ko
    cp: cannot stat ‘net/rfkill/rfkill.ko’: No such file or directory
    INSTALL net/wireless/cfg80211.ko
    cp: cannot stat ‘net/wireless/cfg80211.ko’: No such file or directory
    DEPMOD 3.10.40-gdacac96

    ubuntu@rudrabot:/usr/src/installLibrealsense/UVCKernelPatches$ ./copyzImage.sh
    cp: cannot stat ‘arch/arm/boot/zImage’: No such file or directory
    uvcvideo

    Please let me know where I went wrong or what else should I do.

  8. Didn’t work for me as well.
    I tried 2 times. The first I used a Jetson TK1 not sure which version, but I know it was different from yours. After issuing $ ./copyzImage.sh i got

    cp: cannot stat ‘arch/arm/boot/zImage’: No such file or directory

    The second time I flashed using this configuration, because I’m on kobuki so it had to be this, otherwise won’t work

    https://devtalk.nvidia.com/default/topic/906018/jetson-tk1/-customkernel-the-grinch-21-3-4-for-jetson-tk1-developed/

    And now the building step gave a lot of warnings in the end (and some throughout) such as
    “cp: cannot stat ‘sound/drivers/snd-virmidi.ko’: No such file or directory
    INSTALL sound/firewire/snd-firewire-lib.ko ”

    and in the end there wasn’t even a copyzImages.sh file.

    It’s worth mentioning that I had previouslly installed some ROS modules in the board, but I don’t think it it would interfere that much.

    • ‘cannot stat’ means that it couldn’t find a compiled image file. This probably indicates that the kernel did not finish building. You will need to examine the make log for the issue.
      The source of your problem is most likely that you are not using the correct kernel sources for the version of the operating system you are working with. The JetsonHacks scripts download the kernel sources for the L4T version that is running.

      If you need to use the Grinch kernel, you will need to download the Grinch kernel sources.
      In either case, you will need to modify the get kernel script to download the correct kernel for your use case. Good luck!

  9. Hello!
    Sorry for disturbing you.
    Now I want to install the driver of r200 on a Jetson tk1 with ubuntu 14.04.
    So I followed what you do in this page.
    I have two questions:
    1. I saw that when you input
    $ uname -r,
    the local version was
    3.10.40-gdacac96.
    However, in my case it only showed 3.10.40. What should I do if I don’t have a local version in the next steps? I can’t reinstall the ubuntu 14.04 following what you did in another article because I have installed many other drivers and packages in the present ubuntu I’m using.

    2. I noticed that in this article it uses libusb 1.0-0 but I have installed libusb 1.0-9 in my tk1 and it must be remained because other device needs this library.
    Some errors occurred when I i was installing libusb 1.0-0 to tk1 since it already has 1.0-9.
    So I want to known if the driver of r200 could be installed with libusb 1.0-9?
    Should I change something in the code?

    Thank you for your help!

    • The lack of a local version indicates that you did not set the local version when you rebuilt the kernel.
      You will need to port the code to libusb 1.0-9.
      Thanks for reading!

      • Thank you for replying!
        I tried to port the code to libusb 1.0-9 by changing the include file paths of libusb.
        So I tried to change the include file paths by the makefile under the directory of librealsense.
        I changed the LIBUSB_FLAGS to “LIBUSB_FLAGS := -L/home/ubuntu/libusb-1.0.9/libusb/.libs -lusb-1.0 -I/home/ubuntu/libusb-1.0.9/libusb” because I installed libusb-1.0.9 here.

        I confirmed the directory of libusb.h by executing $ :locate libusb.h
        Then the terminal showed:
        /home/ubuntu/libusb-1.0.9/libusb/libusb.h
        /usr/local/include/libusb-1.0/libusb.h

        Also confirmed where the libusb-1.0.so is
        /home/ubuntu/libusb-1.0.9/libusb/.libs/libusb-1.0.so
        /opt/ros/indigo/lib/libusb_cam.so
        /usr/local/lib/libusb-1.0.so

        However the same error occurred.
        I’m not very familiar with makefile, so could you tell me where I wrote the correct information to the makefile?

        Thank you very much!

Leave a Reply

Your email address will not be published.


*