Jetson Nano – RealSense Depth Camera

For recent L4T releases, Intel now recommends using a different installation technique, it only takes 5 minutes! Looky here: https://wp.me/p7ZgI9-34j. The following is for historical purposes.

Getting full support for the Intel RealSense Depth Camera on the NVIDIA Jetson Nano Developer Kit is simplified by using a couple of installation scripts. Looky here:

Introduction

In an earlier article, we installed an Intel RealSense Tracking Camera on the Jetson Nano along with the librealsense SDK. We’ve have used the RealSense D400 cameras a lot on the other Jetsons, now it’s time to put them to work on the Jetson Nano.

For best performance and support of the RealSense Depth Camera features, Intel recommends modifying the Linux kernel and modules.

To remind you of the different cameras available, here’s a couple of the more popular models which points out their features:

Intel® RealSenseTM Depth Camera D415

  • Intel® RealSenseTM Vision Processor D4
  • Up to 1280×720 active stereo depth resolution
  • Up to 1920×1080 RGB resolution
  • Depth Diagonal Field of View over 70°
  • Dual rolling shutter sensors for up to 90 FPS depth streaming
  • Range 0.3m to over 10m (Varies with lighting conditions)

Intel® RealSenseTM Depth Camera D435/D435i 

  • Intel® RealSenseTM Vision Processor D4
  • Up to 1280×720 active stereo depth resolution
  • Up to 1920×1080 RGB resolution
  •  Depth Diagonal Field of View over 90°
  • Dual global shutter sensors for up to 90 FPS depth streaming
  • Range 0.2m to over 10m (Varies with lighting conditions)
  • Intel® RealSenseTM Depth Camera D435i includes Inertial Measurement Unit (IMU) for 6 degrees of freedom (6DoF) data

For robotics applications, the D435 is popular due to its global shutter and wide field of view.

Software Installation

To interface with the camera,  Intel provides the open source library librealsense. On the JetsonHacksNano account on Github, there is a repository named installLibrealsense. The repository contains convenience scripts to install librealsense.

Note: Starting with L4T 32.2.1/JetPack 4.2.2 a swap file is now part of the default install. You do not need to create a swap file if you are using this release or later. Skip the following step if using 32.2.1 or above.

In order to use the install script, you will either need to create a swapfile to ease an out of memory issue, or modify the install script to run less jobs during the make process. In the video, we chose the swapfile route. To install the swapfile:

$ git clone https://github.com/jetsonhacksnano/installSwapfile
$ cd installSwapfile
$ ./installSwapfile.sh
$ cd ..

You’re now ready to install librealsense.

$ git clone https://github.com/jetsonhacksnano/installLibrealsense
$ cd installLibrealsense
$ ./installLibrealsense.sh

While the installLibrealsense.sh script has the option to compile the librealsense with CUDA support. If you want to add CUDA support to the librealsense SDK, add a -c switch to the shell script:


$ ./installLibrealsense.sh -c

The location of librealsense SDK products after installation:

  • The library is installed in /usr/local/lib
  • The header files are in /usr/local/include
  • The demos and tools are located in /usr/local/bin

Kernel and Modules

For the RealSense Depth Cameras, you will find that performance is much better if you apply the patches to the kernel models. Note: If you have a D435i, the camera will not be detected without the patches.

This video covers the differences between only installing librealsense versus librealsense plus the kernel modifications. Looky here:

To install the kernel and module patches, build them, and then install first switch to the installLibrealsense directory then:

You’re now ready to install librealsense.


$ ./patchUbuntu.sh

This will start the patch, build and install process. On a micro SD card this will take ~ one hour, 20 minutes. Note: If you compile the kernel and modules on a USB SSD, remember to copy the new Image to the /boot directory of the SD card which you boot from.

Demos

Go to the demos and tools directory in /usr/local/bin, and checkout the realsense-viewer application and all of the different demonstrations! There are a wide variety of code samples for different uses in the librealsense SDK. When you go to program against the SDK, you’ll benefit from having a good catalog with which to work.

Notes

The following issue appears to be resolved:

July 1, 2019 – There appears to be an issue with the bzip2 utility which is used in the patchUbuntu.sh script. If you encounter this error, which usually looks something like:

Current working directory: /usr/src/kernel/kernel-4.9
bash: scripts/config: No such file or directory
make: *** No rule to make target ‘olddefconfig’. Stop.

See here for a workaround: https://github.com/JetsonHacksNano/installLibrealsense/issues/12

72 Comments

  1. Hi,

    I followed your instructions with a fresh Jetson Nano install. it all seems to work well but my D435i does not connect. I can see the device in cheese and via libusb.

    When i run realsense-viewer it does not show the d435i

    it says ;

    (ds5-factory.cpp:678) DS5 group_devices is empty.
    5 times.

    Is there any step before or after i should be performing?

    Thanks for all your work!

      • is there any way to check where they may have gone wrong- because this is exactly what happened to me too- and I’ve updated the FW and tried re-flashing the sd card as well to no avail.

        • I’ve modified patchUbuntu.sh as follows and my D435i works fine with Nano.

          patchUbuntu.sh Line 143

          Before :
          sudo ./scripts/configureKernel.sh

          After:
          sudo ./scripts/configureKernelUSB.sh

          • Thank you very much for sharing this. This gave me the clue as to what was wrong with the scripts, which I was able to fix relatively easily after this receiving this information. I really appreciate your contribution, it saved me a lot of time coming up with a fix!

    • Hi,

      I had the exact same issue from a fresh Jetson Nano install… Flashed fresh SD card. Setup Nano. Installed swap file and reboot. Run installLibrealsense script. Wait 1-2 hours. Run patchUbuntu script. Wait 1-2 hours. Reboot. Test Camera D435i…”(ds5-factory.cpp:678) DS5 group_devices is empty.” and not recognized by realsense viewer.

      So wondering if something had silently failed during the kernel rebuild, I ran the patchUbuntu script a second time, and now it works.

      So you might want to try that too.

      BTW.. Jim, thanks so much for posting this info and scripts. It’s been really frustrating trying to get things up and running on the Nano and the info on your website has been invaluable.

      Cheers,
      Will

      • Thank you very much for the kind words. Your description allowed me to fix the scripts for the D435i.

        I understand the pain in getting everything to work, it’s taken me about 3 long weeks to get the RealSense cameras working properly. And this is after having set them up on other Jetsons in the past!

        However, once the scripts are written and in this case people test them out, a lot of people benefit. When the official release of the Jetson Module and new motherboard comes out, it will probably be worth it to come up with an disk image with librealsense pre-installed. Thanks for the helping!

  2. Hi, Curious if this will work with the r200? Or will following the tx1/2 tutorial for the r200 work for the nano as well?

    • This is for librealsense2, the R200 requires the legacy librealsense. You would need to figure out how to patch the kernel and modules for version 4.9. Thanks for reading!

      • hello, I’m on the same track, trying to make an r200 work on jetson nano,
        could your other post “Intel RealSense Camera Installation – librealsense – NVIDIA Jetson TK1” be useful.
        sorry, but I have no much experience with this.

  3. Hello, i found the following issue and could not solve it.

    Current working directory: /usr/src/kernel/kernel-4.9
    bash: scripts/config: No such file or directory
    make: *** No rule to make target ‘olddefconfig’. Stop.

    Thank you so much !!

    • I have also had the same error message. The full terminal looks like this:

      ~/installLibrealsense$ ./patchUbuntu.sh
      Jetson Model: jetson-nano
      Jetson L4T: 32.1.0
      buildModules and Kernel previously installed
      Proposed source path: /usr/src/kernel/kernel-4.9
      ==== Kernel source appears to already be installed ===============
      Patching and configuring kernel
      Configuring Kernel for librealsense
      Current working directory: /usr/src/kernel/kernel-4.9
      bash: scripts/config: No such file or directory
      make: *** No rule to make target ‘olddefconfig’. Stop.

      It appears that everything runs fine until it gets to the .config file, but then something is missing, and I can’t figure out what is missing because the .config file is in the correct folder. Any ideas? Thanks

  4. I am interested in installing RealSense T265 tracking camera on RacecarJ which has Jetosn TX2. Is the installation process the same on Jetson Tx2 as the Nano?

  5. D435i worked perfectly without any issues. Not incomplete frames at all either. I followed your release v0.8. I used a 64 GB sd on my Nano. Thanks so much for your post, videos and git…. 🙂

  6. After solving the bzip error above, I ran into this following issue while running patchUbuntu.sh. Googling returns very little on the subject. Any ideas for solving this issue?

    /usr/src/kernel/nvidia/drivers/net/ethernet/nvidia/pcie/tegra_vnet.c: In function ‘process_ctrl_msg’:
    /usr/src/kernel/nvidia/drivers/net/ethernet/nvidia/pcie/tegra_vnet.c:867:11: error: ‘msg.msg_id’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
    else if (msg.msg_id == CTRL_MSG_LINK_DOWN_ACK)
    ^
    cc1: all warnings being treated as errors
    scripts/Makefile.build:335: recipe for target ‘drivers/net/ethernet/nvidia/pcie/tegra_vnet.o’ failed
    make[5]: *** [drivers/net/ethernet/nvidia/pcie/tegra_vnet.o] Error 1
    scripts/Makefile.build:649: recipe for target ‘drivers/net/ethernet/nvidia/pcie’ failed
    make[4]: *** [drivers/net/ethernet/nvidia/pcie] Error 2
    scripts/Makefile.build:649: recipe for target ‘drivers/net/ethernet/nvidia’ failed
    make[3]: *** [drivers/net/ethernet/nvidia] Error 2
    scripts/Makefile.build:649: recipe for target ‘drivers/net/ethernet’ failed
    make[2]: *** [drivers/net/ethernet] Error 2
    scripts/Makefile.build:649: recipe for target ‘drivers/net’ failed
    make[1]: *** [drivers/net] Error 2
    Makefile:1104: recipe for target ‘drivers’ failed
    make: *** [drivers] Error 2
    Make did not successfully build
    Please fix issues and retry build

    • kangalow or anyone, do you have any ideas for how to solve this issue? I have been trying to solve it for a week and have not been able to fix it. Thank you.

  7. Nano and D435.
    Installed the software via your great instructions, but now can’t find the compiled examples. I especially want to run the viewer just to validate the install. Where are the demos, please? Executables are neither in home/librealsense/xxx or in /usr/local/bin. Did I not build them during installation?
    Thanks

  8. Hi Kangalow,

    I have really enjoyed installing the kernel and librealsense with your scripts, however I have run into problems now that I am booting from am ssd.

    Although it works perfectly when using a microsd, when using an SSD realsense-viewer will start and the entire board will instantly crash. Afterwards, the green pwr LED isn’t even on. I have tried with 10W mode and from batteries with a converter able to supply 5v/10A max. so supplier power is not a problem. It doesn’t seem too hot either so don’t think it’s thermal shutdown.

    I am wondering what the maximum power is able to be supplied by the USB bus. I reckon the SSD must be taking up nearly 5W and the camera would be 1.5W (300mA) and wireless mouse, etc. probably 0.5W.

    After this, realsense-viewer will not launch although there are no error messages. Nothing either in syslog or dmesg.

    Any help would be appreciated!

    Thanks,

    Tom.

    I can open a issue in GitHub if that would be more useful

  9. There are NO executables associated with realsense that I can find. Did I miss a step? Or is there another shell script that will build all the examples? Call me:
    Lost in the ozone.

    Thanks,
    David

  10. No drastic error messages. A few warnings about this or that variable declared but not used and such.
    is there another shell script that will build all the examples?

  11. Can this ever be reached from Python? If so, how? I have installed it, and can run almost all the demos, but I want to import this on Python so I can do some work on it myself. Thanks!

    • You will need to modify the script to support python development. Typically it is something like adding the following build flags:

      cmake …/ -DBUILD_PYTHON_BINDINGS=bool:true -DBUILD_EXAMPLES=true

      but you will need to read the documentation for it to suit your needs. You will also need to set the PYTHONPATH and import pyrealsense2 in your code.

      • Thank you for your reply, will try it! One more important question though, I’m trying to install ROS wrapper for realsense, but it can never find the librealsense SDK … How can I point realsense-ros wrapper to what we’ve installed with this tutorial?

  12. I have also had the same error message. The full terminal looks like this:

    ~/installLibrealsense$ ./patchUbuntu.sh

    Jetson Model: jetson-nano
    Jetson L4T: 32.1.0
    buildModules and Kernel previously installed
    Proposed source path: /usr/src/kernel/kernel-4.9
    Jetson Model: jetson-nano
    Getting L4T Version
    Jetson BSP Version: L4T R32.1.0
    Jetson_L4T=32.1.0
    ==== Jetson Board Mismatch! =============
    Currently this script works for the NVIDIA Jetson Nano Developer Kit.
    This processor appears to be a jetson-nano, which does not have a corresponding script

    Exiting

    D435i is not working, but D415 is ok. Any ideas? Thanks

      • thank you. and i have an another problem.

        git: ‘check’ is not a git command. See ‘git –help’.
        The most similar commands are
        checkout
        check-attr
        cherry
        I deleted the old version and try v0.8, it is still -Jetson Board Mismatch-

        • It is ‘git checkout’. I still suggest that you use vL4T32.1.0 which appears to match your L4T version. Also, you do not need to rebuild the kernel if you are only using the T265 Depth Camera which is what this article is about.

          • i want to use d435i and I redownload v0.8(does v0.8 match my version?).
            I deleted ‘librealsense’ and rebuild with v0.8, but still not ok.
            I did follow this:
            ./installSwapfile.sh
            ./installLibrealsense.sh -c
            ./patchUbuntu.sh (wrong)

            • I do not know the version of L4T that you installed. There have been three. Each L4T has it’s own kernel version. There are also different versions of librealsense that are incorporated in the script. v0.8 is for the first L4T version, vL4T32.1.0 is for the second version, vL4T32.2 is for the third version (as of this writing current).

              As noted before, it appears as if you should use vL4T32.1.0 of installLibrealsense. It is not clear why you decided to use v0.8.

              Recommend that you start a clean install by flashing the (at this time current) JetPack 4.2.1 version which has addressed several USB issues. Then run installLibrealsense vL4T32.2.

  13. My computer reboot during $ ./patchUbuntu.sh and next try I have ‘Reversed (or previously applied) patch detected! Assume -R? [n]’
    I tried N/Y, but still patch make error,
    ‘Make did not successfully build
    Please fix issues and retry build’
    If necessary, how can I delete all patches and re-patch again?
    Thank you.

  14. I followed your setup steps for the 435i on my Nano and the camera works great!

    I found that after patching the kernel and installing the RealSense software that NVIDIA Jetson inference demos couldn’t access the CSI camera for image classification anymore. I just made another disc image on a separate memory card to use for the inference software library. Am I right in thinking that patching the kernel has caused the issue? It’s no big deal, I’m just curious.

    By the way thank you so much for your clear, in-depth tutorials. Your site is an incredibly useful resource.

    • Thank you for the kind words. The kernel should not have changed that much, I don’t know what specific change might be causing issues with the CSI camera. It could be that there is some issue with the installation if the kernel is swapped out from underneath it. Thanks for reading!

  15. Is it okay that before running patch script there was around 20 GB free space on the SD card, but after patching there are only 5 GB left. Maybe I should clean something.

    Could you help me, because I have only 32 GB SD card.

    I also have another question , is it possible top run patching with L4T 32.2.0?? Or only with 32.2.1??

  16. First off, these scripts are a great help.
    I just moved an application on a Jetson Nano from (Nano r32.2.1 Jetpack 4.2.2) to (Nano r32.3.1 Jetpack 4.3) in hopes of gaining some performance and leveraging Tensorflow 2.0.

    I created new image and rebuilt the Realsense libraries using your ./installLibrealsense.sh and ./buildLibrealsense.sh scripts. In the 4.2.2 image I also ran ./patchUbuntu.sh.

    Everything works the same but the performance of the 4.3 image is nearly 5x slower than the 4.2.2 image. I have a c++ driver accessing the realsense streams and in this application I’m only using the color stream.

    As a side note, I’m also finding that the keras model I’m using is performing nearly 10x slower on 4.3 than on 4.2.2 (looking at that separately).

    I wondered whether you have any insight as to why the performance hit on the latest Jetpack or more specifically on the Realsense library.

    Thanks

      • Thanks for the response,
        Yes, I’m running on a Jetson Nano. As to why I ran both scripts, initially I ran the installLibRealsense.sh – saw it was using prebuilt files via apt-get.
        I ran my project and saw the CPU utilization (via top) was much higher than the version I’m running on my Nano w/Jetpack 4.2.2. The new version on (Jetpack 4.3 Realsense V2.31) was running 105%, the version on (Jetpack 4.2.2 Realsense V2.25) runs at 19-20%.
        I speculated perhaps the apt-get repo versions may not have been compiled correctly so I attempted to build from sources. The results were nearly identical.
        The version of librealsense I’m running on the 4.2.2 image is 2.25 and I used your installLibrealsense and patchUbuntu scripts to install.

        A little background on the software I’m testing. It enables a realsense stream, processes the callback, does some cv2 post processing then publishes on a ROS2 topic, all in all pretty simple. Would be happy to share code if that would help.

        I have not had any time to profile the code or process to see where the cpu is being consumed and you’ve clearly spent much more time on realsense integration so I was curious to see if you’ve noticed any performance degradation between Realsense release 2.25 (running on jetpack 4.2.2) vs 2.31 (running on jetpack 4.3).
        Please let me know any other information you need. In the meantime I’ll be looking at where all my cpu is going 🙂

        • The version that is installed using apt-get install a UVC backend, which is different than earlier versions. While this may be the cause of the slowdown, it seems more likely that with the high CPU utilization you describe there is lack of a processor yield, or insufficient time spent on the yield itself.

          The realsense-viewer felt to me like it was running both versions at approximately the same frame rates.

  17. This might sound like a really dumb question but I’ve been looking around for ages and I can’t find any script called ./patchubuntu.sh. It is not on the github repo and the other methods for doing it manually seem somewhat above my current level. How can I downloaded the script to patch the kernel?

    My jetson Nano also seems to be crashing quite frequently while running realsense-viewer at the moment, and I’m hoping that the patches will fix that issues but any insight as to why that might be would be appreciated.

    • You will need to check the ‘tags’ section of the Github repository to find the version that matches your L4T release. Intel now recommends using UVC as the backend for librealsense on the Jetsons, which is what the current Github repository installs. See:

      https://www.jetsonhacks.com/2019/12/22/install-realsense-camera-in-5-minutes-jetson-nano/

      You do not mention which version of L4T you are running, or how you are powering the camera or the Jetson. That makes it difficult to offer any advice.

      • Hey, thanks for responding! I’m currently running L4T version 32.3.1 which I believe is the latest version. I ran the installation script from the article which you linked there, and then installed the latest version of ROS Melodic as well. I believe the UVC backend is experiencing difficulties in my setup, as one of the error messages I am getting is “UVC metadata payloads not available”. Unfortunately I am not the most experienced with this stuff and thus I am not sure what steps to take to roll back the version of librealsense.

        Apologies for not including more information earlier. I am currently using the Micro USB connection powered with a 5v 2.1 amp plug, which I believe is adequate for running the system. I find the nano slows to a crawl when running “roslaunch realsense2_camera rs_camera.launch” and when trying to run realsense-viewer it crashes entirely after about 20 seconds. The camera (D415) is being powered exclusively from the usb 3.0 cable connecting it to the Nano.

        I am reasonably confident I am missing some steps regarding the versions of the software I am using, but I am not sure which ones I should use or how to access them.

        • David is correct. The Jetson alone can easily use 2A @ 5V if you are in 10W mode. There isn’t enough power to run both the Jetson and the RealSense simultaneously from the power supply that you are using.

          • I am getting much better results now that I have switched over to a 5v 4A power supply. My crashing issue is now solved, thank you! I’m still getting that worrying message about UVC payloads however (UVC metadata payloads not available). It seems to refer to a mistake in installation, but I’m not sure precisely what the mistake may be.

  18. Actually, I think you’re starving the system for power. Correct that and see if the problems disappear (at least some of them.) 🙂

    • You make a fair point! That will almost definitely clear up the crashing. I was plugging in a power supply into the power jack but didn’t know about the necessity of the jumpers.

Leave a Reply

Your email address will not be published.


*