Install Kinect V2 using libfreenect2 on NVIDIA Jetson TK1

This article has been superseded. Please see: Install Kinect V2 – Part II

The article below is for historical purposes only, and does not reflect the current state of the project.

Open source drivers (OpenKinect libfreenect2) are available for the Kinect for Windows V2. Most installations are for PC based, x86 machines. The Jetson TK1, being an ARM V7 machine poses some special challenges for porting the source over. Most of those issues have been addressed, and an installation script is now available. Looky here:

Introduction

While most of the porting issues are straightforward, some special code needed to be written to get good performance out of the Kinect sensor on the Jetson. There are also some hardware issues that need to be addressed.

Fortunately Lingzhu Xiang added Tegra support for the Jetson to libfreenect2 which addressed the performance issues. The code is available on Github at: https://github.com/xlz/libfreenect2. Here is the “how to”: Jetson TK1 HOWTO. The Tegra code allows frame processing at 60Hz. Way excellent!

You might remember from an earlier post (Kinect V2 with libfreenect2) that there were issues using the built in USB controller. So a Syba 19-Pin USB 3.0 Header Mini PCI-Express Card with Female USB 3.0 Cable SD-MPE20142 was acquired. Note: It is a full size Mini PCI-Express card, so it does hang over the edge of the Jetson when installed. Also, the card requires a floppy drive power connector. I found one at a local Frys electronics store that hooked into the Jetson onboard Molex power connector.

Also, remember that in order to use the Kinect for Window V2 with the Jetson, the Kinect needs to be initialized with a Windows 8.1 machine.

This installation was run on a Jetson with L4T 21.2 with OpenCV and CUDA 6.5 installed.
(See JetPack installation). After the flash installation, the Post Flash Setup procedure was performed, which prepares the Jetson for desktop operation. After using JetPack and performing the post flash setup, the Jetson is prepared as the starting point for libfreenect installation.

Another note, the Jetson feature of USB autosuspend needs to be disabled for the Kinect V2 to work correctly. This is taken care of by the post flash setup mentioned above.

Issues, Issues, Issues …

When I tried to install Lingzhu Xiang’s code, I ran into several issues. The installation script below to install libfreenect2 addresses these issues.

The first issue is that the original installation recommended installing opencv-dev. Unfortunately this overwrites the hardware accelerated Tegra OpenCV library that JetPack installs. This was easy enough to rectify, the only real issue was that the include file turbojpeg.h was missing. I built a little repository for the installation that was forked from Lingzhu Xiang’s and added turbojpeg.h to it. Also, the link to libturbojpeg.so needed to be moved around a little.

The second issue is with libusb. It appears as if libusb/libfreenect2 was changed after the original repository was published. In this installer, there is a patch included for the libusb headers (usbfs.h) which sets the MAX_ISO_BUFFER_LENGTH to be 49152.
Note: Earlier versions of libfreenect2 worked with the built in USB 3.0 of the Jetson, as Lingzhu Xiang reported. The current version (even with the patch) does not. This is an outstanding issue.

By adding the file 90-kinect2.rules to the udev permissions, Kinect V2 is given permission to run in user space. This is included in the script setup.

For the demo, a colormap of the depth map is added. By pressing one of the number keys, different colormaps can be used.

Note: The NVIDIA Gstjpeg headers are needed to enable hardware acceleration and JPEG decoding. These headers are downloaded using the installation script and placed into the libfreenect/depends directory.

Installation

The installation script is available on Github:

Download it to the Jetson. In the video above, the script is placed into the Home (~/) directory.

$ sh installLibfreenect2.sh

This will install the code dependencies, download the libfreenect source and dependent libraries, and compile them.

Once the installation is complete, you can reboot the Jetson. This will enable the udev Kinect rules, which allows user access to the Kinect. If you do not reboot the Jetson, you will have to use the ‘sudo’ command before running the demo.

Demo

Switch to the demo directory:

$ cd libfreenect2/examples/protonect/bin

(This assumes that libfreenect2 was installed in your home directory). To run the demo:

$ ./Protonect

If all went well, you should see the depth and rgb images.

Good luck!

31 Comments

  1. hey,

    The information was really helpful. Are you working on multiple kinect implementation?
    CAn you help me out with that?

  2. Hi JetsonHacks,

    When I ran sh installLibfreenect2.sh, it reached a fatal error at 47%. It shows:

    [ 47%] Building NVCC (Device) object CMakeFiles/cuda_compile.dir/src/./cuda_compile_generated_cuda_depth_packet_processor.cu.o
    /home/sebastian/gist1bd2830cc1a5790a6ac2-3d786bbb39cd936a5e823a13bea703817447c2fb/libfreenect2/examples/protonect/src/cuda_depth_packet_processor.cu:29:25: fatal error: helper_math.h: No such file or directory
    #include

    Do you know what the problem is?

  3. For CUDA, I tried the command and it said command not found. For L4T, I could not find the file in the place that you listed. Do I have to download it?

  4. Now I trying that run libfreenect2 on Jetson TK1

    and, there are some issues.

    If you know, please some adbises.

    ————————————————

    1.I tried to adjust CPU clock and GPU clock

    # cat /sys/kernel/cluster/active
    G

    # cat /sys/kernel/debug/clock/gbus/rate
    852000000

    ————————————————

    2.And I use a mini PCI-Express USB3.0 19pin (Renesas nPD720200)

    Power is on mora 4Pin on Jetson.

    After some tried, I used external power supply, but the same issue is happening

    ————————————————

    3.Result of lsusb

    $ lsusb
    Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 002 Device 003: ID 045e:02c4 Microsoft Corp.
    Bus 002 Device 002: ID 045e:02d9 Microsoft Corp.
    Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 002: ID 045e:02d9 Microsoft Corp.
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

    ————————————————

    4. I have installed them as described below.

    https://gist.github.com/jetsonhacks/1bd2830cc1a5790a6ac2#file-installlibfreenect2-sh

    That .sh file is as described below

    sudo apt-get install -y build-essential libturbojpeg libtool autoconf libudev-dev cmake mesa-common-dev freeglut3-dev libxrandr-dev doxygen libxi-dev libjpeg-turbo8-dev

    git clone https://github.com/jetsonhacks/libfreenect2.git

    wget http://developer.download.nvidia.com/mobile/tegra/l4t/r21.2.0/sources/gstjpeg_src.tbz2

    tar -xvf gstjpeg_src.tbz2 gstjpeg_src/nv_headers

    mv gstjpeg_src/nv_headers/ libfreenect2/depends/

    rmdir gstjpeg_src/

    cd libfreenect2/depends

    sh install_ubuntu.sh

    sudo ln -s /usr/lib/arm-linux-gnueabihf/libturbojpeg.so.0.0.0 /usr/lib/arm-linux-gnueabihf/libturbojpeg.so

    cd ../examples/protonect/

    cmake CMakeLists.txt

    make && sudo make install

    cd ../..

    sudo cp extras/90-kinect2.rules /etc/udev/rules.d/90-kinect2.rules

    /bin/echo -e “\e[1;32mFinished.\e[0m”

    ————————————————-

    The following message is displayed when I start the Protonect.

    ————————————————

    $ sudo ./bin/Protonect
    [sudo] password for ubuntu:
    modprobe: FATAL: Module nvidia not found.
    [CudaDepthPacketProcessorKernel::initDevice] device 0: GK20A @ 852MHz Memory 1892MB
    [CudaDepthPacketProcessorKernel::initDevice] selected device 0
    [Freenect2Impl] enumerating devices…
    [Freenect2Impl] 8 usb devices connected
    [Freenect2Impl] found valid Kinect v2 @2:3 with serial 017234451747
    [Freenect2Impl] found 1 devices
    [Freenect2DeviceImpl] opening…
    [Freenect2DeviceImpl] opened
    [Freenect2DeviceImpl] starting…
    [Freenect2DeviceImpl] ReadData0x14 response
    92 bytes of raw data
    0x0000: 00 00 15 00 00 00 00 00 01 00 00 00 43 c1 1f 41 …………C..A
    0x0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0x0020: 0a 21 33 55 20 00 17 ba 00 08 00 00 10 00 00 00 .!3U ………..
    0x0030: 00 01 00 00 00 10 00 00 00 00 80 00 01 00 00 00 ……….�…..
    0x0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …………….
    0x0050: 00 00 00 00 00 00 00 00 07 00 00 00 …………

    ==== CuDepthPakePro CommandResponse ==== 1302882
    [Freenect2DeviceImpl] ReadStatus0x090000 response
    4 bytes of raw data
    0x0000: 00 22 00 00 .”..

    [Freenect2DeviceImpl] ReadStatus0x090000 response
    4 bytes of raw data
    0x0000: 00 22 00 00 .”..

    [Freenect2DeviceImpl] enabling usb transfer submission…
    [Freenect2DeviceImpl] submitting usb transfers…
    [Freenect2DeviceImpl] started
    device serial: 017234451747
    device firmware: 2.3.3913.0.7
    [TegraJpegRgbPacketProcessor] avg. time: 15.8725ms -> ~63.0022Hz
    [RgbPacketStreamParser::handleNewData] skipping rgb packet!
    [RgbPacketStreamParser::handleNewData] skipping rgb packet!
    [TegraJpegRgbPacketProcessor] avg. time: 15.9096ms -> ~62.8553Hz
    [TegraJpegRgbPacketProcessor] avg. time: 15.1333ms -> ~66.0795Hz
    [TegraJpegRgbPacketProcessor] avg. time: 15.1105ms -> ~66.1793Hz
    [TegraJpegRgbPacketProcessor] avg. time: 15.1506ms -> ~66.0038Hz
    [TegraJpegRgbPacketProcessor] avg. time: 15.1426ms -> ~66.039Hz
    [TegraJpegRgbPacketProcessor] avg. time: 15.1493ms -> ~66.0098Hz
    [TegraJpegRgbPacketProcessor] avg. time: 14.9997ms -> ~66.6679Hz
    [TegraJpegRgbPacketProcessor] avg. time: 14.4716ms -> ~69.101Hz
    [TegraJpegRgbPacketProcessor] avg. time: 14.429ms -> ~69.3047Hz
    [RgbPacketStreamParser::handleNewData] skipping rgb packet!
    [TegraJpegRgbPacketProcessor] avg. time: 14.908ms -> ~67.0779Hz
    [TegraJpegRgbPacketProcessor] avg. time: 15.0251ms -> ~66.5553Hz
    [TegraJpegRgbPacketProcessor] avg. time: 15.1235ms -> ~66.1221Hz
    [TegraJpegRgbPacketProcessor] avg. time: 15.1282ms -> ~66.1015Hz
    [TegraJpegRgbPacketProcessor] avg. time: 15.1414ms -> ~66.0441Hz

    • I believe that the current version, libfreenect2 0.2, does *not* support audio. From what I understand, people have been working on it, but most people are interested just in the video/depth so there hasn’t been a lot of progress in this area. Thanks for reading!

  5. Hello, I installed the libfreenect2 but ~/libfreenect2/examples/protonect/bin folder does not exist after the installation. I installed the Jetpack 3.0 and the grinch kernel
    in my TK1.

      • CMake Error at CMakeLists.txt:115 (MESSAGE):
        Did not find Tegra JPEG library

        — Configuring incomplete, errors occurred!
        See also “/home/ubuntu/libfreenect2/examples/protonect/CMakeFiles/CMakeOutput.log”.
        See also “/home/ubuntu/libfreenect2/examples/protonect/CMakeFiles/CMakeError.log”.
        ubuntu@tegra-ubuntu:~/libfreenect2/examples/protonect$ make
        make: *** No targets specified and no makefile found. Stop.

        • The complete messages when i run cmake:
          ubuntu@tegra-ubuntu:~/libfreenect2/examples/protonect$ cmake CMakeLists.txt
          — The C compiler identification is GNU 4.8.4
          — The CXX compiler identification is GNU 4.8.4
          — Check for working C compiler: /usr/bin/cc
          — Check for working C compiler: /usr/bin/cc — works
          — Detecting C compiler ABI info
          — Detecting C compiler ABI info – done
          — Check for working CXX compiler: /usr/bin/c++
          — Check for working CXX compiler: /usr/bin/c++ — works
          — Detecting CXX compiler ABI info
          — Detecting CXX compiler ABI info – done
          — Performing Test LIBFREENECT2_THREADING_STDLIB
          — Performing Test LIBFREENECT2_THREADING_STDLIB – Failed
          — using tinythread as threading library
          — Could NOT find OpenCL (missing: OPENCL_INCLUDE_DIRS)
          — Found CUDA: /usr/local/cuda-6.5 (found version “6.5”)
          — Looking for XOpenDisplay in /usr/lib/arm-linux-gnueabihf/libX11.so;/usr/lib/arm-linux-gnueabihf/libXext.so
          — Looking for XOpenDisplay in /usr/lib/arm-linux-gnueabihf/libX11.so;/usr/lib/arm-linux-gnueabihf/libXext.so – found
          — Looking for gethostbyname
          — Looking for gethostbyname – found
          — Looking for connect
          — Looking for connect – found
          — Looking for remove
          — Looking for remove – found
          — Looking for shmat
          — Looking for shmat – found
          — Looking for IceConnectionNumber in ICE
          — Looking for IceConnectionNumber in ICE – found
          — Found X11: /usr/lib/arm-linux-gnueabihf/libX11.so
          — Found OpenGL: /usr/lib/arm-linux-gnueabihf/libGL.so
          — Looking for include file pthread.h
          — Looking for include file pthread.h – found
          — Looking for pthread_create
          — Looking for pthread_create – not found
          — Looking for pthread_create in pthreads
          — Looking for pthread_create in pthreads – not found
          — Looking for pthread_create in pthread
          — Looking for pthread_create in pthread – found
          — Found Threads: TRUE
          — Using X11 for window creation
          — Using GLX for context creation
          — Looking for glXGetProcAddress
          — Looking for glXGetProcAddress – found
          — Looking for glXGetProcAddressARB
          — Looking for glXGetProcAddressARB – found
          — Looking for glXGetProcAddressEXT
          — Looking for glXGetProcAddressEXT – not found
          CMake Error at CMakeLists.txt:115 (MESSAGE):
          Did not find Tegra JPEG library

          — Configuring incomplete, errors occurred!
          See also “/home/ubuntu/libfreenect2/examples/protonect/CMakeFiles/CMakeOutput.log”.
          See also “/home/ubuntu/libfreenect2/examples/protonect/CMakeFiles/CMakeError.log”.

  6. Thank you so much for this great tutorial!

    Would love to use NiTE-2.0.0 for Skeleton Tracking and running into some compiling issues. It doesn’t seem as straight forward to get NiTE-2.0.0 working and been navigating many compile errors. Would love to see a tutorial on this sometime… especially before Halloween. Got some Skeletons to track 🙂 Many thanks for all you do!

  7. Hi,

    At first thanks for the tutorial. I have a little problem when i use your scrip. In the step when you use cmake CMakeLists.txt, appear this:
    CMake Error at CMakeLists.txt:115 (MESSAGE)
    did not find Tegra JPEG library

    –configuring incomplete, errors occurred!

    I use Jetson TK1, and is a new installation using JetPack3.1 and I don’t know how fix this problem.

    Thanks for all(my English is so bad, i know)

Leave a Reply

Your email address will not be published.


*