MS Kinect V2 on NVIDIA Jetson TX1

With a USB firmware patch and an updated libfreenect2, the Microsoft Kinect V2 now runs on the Jetson TX1 Development Kit. Looky here:

Background

For a stretch there it was not possible to run the open source Kinect V2 driver libfreenect2 on the Jetson TX1 because of an issue with the USB firmware. Fortunately NVIDIA has issued a firmware patch
(see the Jetson TX1 Forum, USB 3 Transfer Failures ) which fixes the issue. As you might recall, Microsoft now offers the Kinect V2 as a two part kit a Xbox One Kinect Sensor Bar along with a Kinect Adapter for Windows. You will need both a Kinect Xbox One sensor and the adapter for use with the Jetson, or the discontinued Kinect for Windows. The Kinect Adapter for Windows converts the output from the Kinect to USB 3.0. The advantage of this setup is that you can use the Kinect sensor from your Xbox One, or at least have an excuse to get a Xbox One + Kinect for “research” purposes.

Installation

The installLibfreenect2 repository on the JetsonHacks Github account contains convenience scripts for installing libfreenect2 and the USB firmware patch. First, get the repository:

$ git clone https://github.com/jetsonhacks/installLibfreenect2

Second, install libfreenect2 and compile the library and examples:

$ cd installLibfreenect2
$ ./installLibfreenect2

Third, you will need to patch the USB firmware:

$ ./firmwarePatch.sh

After installing the USB firmware patch, it is necessary to reboot the machine in order for the firmware changes to take effect.

When the machine reboots, you can run the example:

$ cd ~/libfreenect2/build/bin
$ ./Protonect

Some Notes

The installation of libfreenect2 in the video is on L4T 24.1, flashed by JetPack 2.2. CUDA is required. Both 32 bit and 64 bit versions of L4T are shown in the video, installation of libfreenect2 and the firmware patch is the same in both cases.

The Mesa libs installed by libfreenect2 overwrite libGL.so, which causes issues. The fix as of this writing (July, 2016) is to link libGL.so to the Tegra version.

The repository contains a script jetson_max_l4t.sh which sets the CPU and GPU to maximum clock values. This will increase the Jetson TX1 performance at the cost of power consumption.

L4T 23.X Notes

The JPEG decompressor under L4T 23.X produces RGBA format, where as the Protonect viewer consumes BGRA format. This makes the video appear with a purple hue.

Update 8-28-16 In the NVIDIA Jetson Dev Forum user kassinen wrote:

Actually you can fix it by changing the line 56 from viewer.h

typedef ImageFormat<4, GL_RGBA, GL_BGRA, GL_UNSIGNED_BYTE> F8C4;

to

typedef ImageFormat<4, GL_RGBA, GL_RGBA, GL_UNSIGNED_BYTE> F8C4;

End Update.

The repository contains a patch which adds a simplistic algorithm to rearrange the bytes appropriately. If you intend to use this script with L4T 23.X, you will need to uncomment the line:

# patch -p 1 -i $PATCHDIR/bgra.patch

Also, if you plan to use this library in production L4T 23.X code, your application should consider writing specialized code to do the RGBA→BGRA conversion more efficiently.

For L4T 24.1, there is no patch applied as the JPEG decompressor produces BGRA format natively.

27 Comments

    • Hi Frank,
      Short answer is my usual waffle, it depends.

      It depends on what you measure. In the video above, the example app Protonect runs 4 panes. Pane 1 is the IR stream, Pane 2 is a color RBG stream, Pane 3 is a Depth frame, and Pane 4 is a registered color/depth frame.

      When data is received from the Kinect, it is decompressed with a JPEG decoder. On the TK1 and TX1 I believe that this is done in CUDA code. Both decompress frames faster than “real time”. The actual Kinect camera provides all streams at up to 30 fps, so both the JTK1 and JTX1 consume and decompress the streams with a minimum of lag.

      If all you were to do is collect the data both the JTK1 and JTX1 are equivalent speed wise as they read and decode the data in parallel.

      If you want to manipulate the data, or display a complex representation, then in practice the JTX1 has about a 50%-100% performance gain. As an example, the registration of the color and depth maps is done in software in the Protonect example, you can see the difference in speed cranking the CPU and GPU clocks on the JTX1 makes. While I haven’t test the JTK1 using the same test, I would guess the frame rate on a cranked JTK1 would be a little less than the normally clocked JTX1.

      One important point is that the JTX1 has more computational reserve left running the Kinect, that’s the advantage of having more CUDA cores and a different processor architecture. A lot of use cases are to gather data from the Kinect and process it on board. The Jetson then passes it to another control module. That would be things like identifying people, objects, hands, and so on. So while the display of something like a point cloud is great eye candy, in practice a lot of applications will build the point cloud, display the point cloud, but not do both at the same time. Or do something all together different. Like I wrote, it depends 😉

  1. Hi kangalow

    Once we got Protonect working with the Jetson TK1 and Kinect V2 the next logical step was to find a driver and the tools needed to receive data from the Kinect V2 sensor, in a way useful for robotics. Specifically, a data bridge between libfreenect2 and ROS. As a start, we just want the Kinect V2 to send data to a mobile base (i.e., Kobuki) to do SLAM. The great work done by Thiemo Wiedemeyer on the “IAI Kinect2” package (https://github.com/code-iai/iai_kinect2) seemed like a good next step (and perhaps the only next step). After several attempts to get the “kinect2_bridge” tool to work, I reached a stalemate, as follows:

    1. The current release of JetPack for the Jetson TK1 and TX1 does not support OpenCL, primarily to do with issue related to installing it on the ARM7 architecture, and OpenCL seems to be a prerequisite for the “kinect2_bridge” tool (on a Jetson TK1 anyway). NVIDIA is understandably pushing CUDA for both the TK1 (https://devtalk.nvidia.com/default/topic/761627/opencl-support-for-jestson-tk1) and the TX1 https://devtalk.nvidia.com/default/topic/899013/jetson-tx1/tegra-x1-opencl-support/ .

    2. The current release of the “IAI Kinect2” package does not support CUDA (as implemented on the Jetson TK1 anyway). Several forks have been attempted in the “IAI Kinect2” package to get the Jetson working but ultimately there were issue in the root code and Thiemo said: “Since I don’t own a Jetson, I can’t provide a release, therefore someone else has to do and maintain it. For now the answer would be: no/maybe, if someone volunteers.” https://github.com/code-iai/iai_kinect2/issues/149

    So I am hoping you might have some insight into the following. Are you aware of any:

    1. new efforts to support OpenCL on the Jetson TK1?
    2. alternative drivers and the tools that would allow a bridge between libfreenect2 and ROS, using a Jetson TK1 or TX1?
    3. new efforts to get Thiemo’s “kinect2_bridge” tool working with the Jetson TK1 version of CUDA?

    The Jetson TK1 and TX1 show so much promise for robotics and the Kinect V2 is arguably the best depth sensor on the market, with respect to availability, cost and resolution. But if we don’t have a bridge between libfreenect2 and ROS, we can never realize that potential.

  2. Thank you for the reference to the OpenPTrack project and the potential ROS bridge! Sometimes these jewels of info just seem to evade one’s effort to find them. 🙂

    • You’re welcome. If you’re planning on using the Kinect V2 for robotics, you may want to also consider the Intel RealSense cameras. Here’s a some info on the R200 and the Jetson: http://www.jetsonhacks.com/2016/06/15/intel-realsense-camera-jetson/
      There’s a ROS bridge for it. The libraries are not quite mature yet on the Jetson (but mostly because the library is new), and the cameras need some tuning to work in any given environment. However, the camera does work outdoors and with ROS. The depth map processing and color registration are done in hardware onboard the camera itself, and the packaging physically lends itself to robotics. The packaging and power wiring alone will save you quite a bit of headache when mounting a camera on a robot.

  3. Have anyone succeeded in running kinect v2 on the latest l4t 24.2? Compilation of protonect leads to ‘undefined reference to drm…’ error, and replacing libGL.so doesn’t help.

  4. Hi, when I run $ ./installLibfreenect2.sh I got the following error, I am using ubuntu 16.04, ROS kinetic, kinect v2, NIVIDIA Quadro K600, CUDA 7.5. Please help me to fix the error?
    .
    .
    .
    [ 4%] Linking CXX executable bin/generate_resources_tool
    [ 4%] Built target generate_resources_tool
    [ 7%] Building NVCC (Device) object CMakeFiles/cuda_compile.dir/src/cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o
    /home/grasp001/libfreenect2/src/cuda_kde_depth_packet_processor.cu:39:25: fatal error: helper_math.h: No such file or directory
    compilation terminated.
    CMake Error at cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o.cmake:207 (message):
    Error generating
    /home/grasp001/libfreenect2/build/CMakeFiles/cuda_compile.dir/src/./cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o

    CMakeFiles/freenect2.dir/build.make:82: recipe for target ‘CMakeFiles/cuda_compile.dir/src/cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o’ failed
    make[2]: *** [CMakeFiles/cuda_compile.dir/src/cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o] Error 1
    CMakeFiles/Makefile2:136: recipe for target ‘CMakeFiles/freenect2.dir/all’ failed
    make[1]: *** [CMakeFiles/freenect2.dir/all] Error 2
    Makefile:127: recipe for target ‘all’ failed
    make: *** [all] Error 2
    [ 4%] Built target generate_resources_tool
    [ 7%] Building NVCC (Device) object CMakeFiles/cuda_compile.dir/src/cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o
    /home/grasp001/libfreenect2/src/cuda_kde_depth_packet_processor.cu:39:25: fatal error: helper_math.h: No such file or directory
    compilation terminated.
    CMake Error at cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o.cmake:207 (message):
    Error generating
    /home/grasp001/libfreenect2/build/CMakeFiles/cuda_compile.dir/src/./cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o

    CMakeFiles/freenect2.dir/build.make:82: recipe for target ‘CMakeFiles/cuda_compile.dir/src/cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o’ failed
    make[2]: *** [CMakeFiles/cuda_compile.dir/src/cuda_compile_generated_cuda_kde_depth_packet_processor.cu.o] Error 1
    CMakeFiles/Makefile2:136: recipe for target ‘CMakeFiles/freenect2.dir/all’ failed
    make[1]: *** [CMakeFiles/freenect2.dir/all] Error 2
    Makefile:127: recipe for target ‘all’ failed
    make: *** [all] Error 2
    Finished.

      • I tried this link before with another machine it worked perfectly, but now I am using NVDIA quadro k600 with different machine, I am facing many problems as you see my error above, I would be grateful for any help

          • Thax for reply, I have used this script before, the kinect v2 worked properly with my laptop (Ubuntu 16.04, kernel 4.8, ROS kenetic).

            But now, I am using different machine (Ubuntu 16.04, kernel 4.10, ROS kenetic,and NVIDIA Quadro k600), when I run sh installLibfreenect2.sh
            I get the following error I don’t understand what is the reason ?? can u help me plz
            .
            .
            .
            .
            [ 93%] Building CXX object CMakeFiles/freenect2.dir/src/opencl_depth_packet_processor.cpp.o
            /home/grasp001/libfreenect2/examples/protonect/src/opencl_depth_packet_processor.cpp:45:21: fatal error: CL/cl.hpp: No such file or directory
            compilation terminated.
            CMakeFiles/freenect2.dir/build.make:508: recipe for target ‘CMakeFiles/freenect2.dir/src/opencl_depth_packet_processor.cpp.o’ failed
            make[2]: *** [CMakeFiles/freenect2.dir/src/opencl_depth_packet_processor.cpp.o] Error 1
            CMakeFiles/Makefile2:105: recipe for target ‘CMakeFiles/freenect2.dir/all’ failed
            make[1]: *** [CMakeFiles/freenect2.dir/all] Error 2
            Makefile:127: recipe for target ‘all’ failed
            make: *** [all] Error 2
            Finished.

  5. Then, I removed CUDA completely, but I get this error
    .
    .
    .
    .
    [ 90%] Building C object CMakeFiles/freenect2.dir/src/flextGL.c.o
    [ 93%] Building CXX object CMakeFiles/freenect2.dir/src/opengl_depth_packet_processor.cpp.o
    [ 95%] Linking CXX shared library lib/libfreenect2.so
    /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libturbojpeg.a(libturbojpeg_la-turbojpeg.o): relocation R_X86_64_32 against `.data’ can not be used when making a shared object; recompile with -fPIC
    /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libturbojpeg.a: error adding symbols: Bad value
    collect2: error: ld returned 1 exit status
    CMakeFiles/freenect2.dir/build.make:599: recipe for target ‘lib/libfreenect2.so’ failed
    make[2]: *** [lib/libfreenect2.so] Error 1
    CMakeFiles/Makefile2:105: recipe for target ‘CMakeFiles/freenect2.dir/all’ failed
    make[1]: *** [CMakeFiles/freenect2.dir/all] Error 2
    Makefile:127: recipe for target ‘all’ failed
    make: *** [all] Error 2
    Finished.

  6. i mistakenly ran the install and firmware patch on a TK1 rather than TX1 board and it is now having problems finding or running xserver or unity desktop –
    ‘exec: /usr/bin/X not found’
    ‘unable to connect to x server: connection refused.

    any ideas?

  7. Everything works great on my TX2 on the install running Protonect.

    However, I’m not able to split the streams into individual lanes. I’d like to record them synchronously, but as different files. I at least want to be able to drop a pair for performance.

    I tried installing the separate gspca-kinect2 drivers that you mentioned before here: https://github.com/yoshimoto/gspca-kinect2, but I can’t overwrite the gspca_mainthis line gives me an error: $sudo /sbin/insmod ./gspca_main.ko

    Do have any suggestions about how to split these signals?

      • Ok, I think I moved forward here. However, I want to record the video stream, and I still can’t find a way to do it. The stream on Protonect is not at /dev/video0, that’s the cam that comes with the Jetson.

        nvgstcapture-1.0 onlygives me the cam the comes with the TX2 as well.

        I use v4l2-ctl –list-devices and get:
        VIDEIOC_QUERYCAP: Failed: Inappropriate ioctl for device
        /dev/video0
        /dev/v4l-subdev1
        /dev/v4l-subdev0

        I tried FFMPEG, but I still need a source for that too. Any ideas?

    • Hi Rich,

      I’m having a problem with my TX2 install. Protonect fails to detect the kinect:

      $ sudo ./Protonect
      Version: 0.2.0
      Environment variables: LOGFILE=
      Usage: ./Protonect [-gpu=] [gl | cl | clkde | cuda | cudakde | cpu] []
      [-noviewer] [-norgb | -nodepth] [-help] [-version]
      [-frames ]
      To pause and unpause: pkill -USR1 Protonect
      [Info] [Freenect2Impl] enumerating devices…
      [Info] [Freenect2Impl] 12 usb devices connected
      [Info] [Freenect2Impl] found 0 devices
      no device connected!

      I’ve install the udev rules and you can it also fails using sudo. The kinect is detected as lsusb output shows:

      $ lsusb
      Bus 002 Device 003: ID 2109:8110 VIA Labs, Inc. Hub
      Bus 002 Device 002: ID 2109:8110 VIA Labs, Inc. Hub
      Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
      Bus 001 Device 013: ID 045e:02ae Microsoft Corp. Xbox NUI Camera
      Bus 001 Device 011: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor
      Bus 001 Device 012: ID 045e:02ad Microsoft Corp. Xbox NUI Audio
      Bus 001 Device 010: ID 0409:005a NEC Corp. HighSpeed Hub
      Bus 001 Device 006: ID 045e:0719 Microsoft Corp. Xbox 360 Wireless Adapter
      Bus 001 Device 004: ID 2109:2811 VIA Labs, Inc. Hub
      Bus 001 Device 003: ID 2109:2811 VIA Labs, Inc. Hub
      Bus 001 Device 002: ID 046d:c52b Logitech, Inc. Unifying Receiver
      Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

      I’m running through 7-port USB 3.0 hub, but I get the same results going directly to the TX2.

      Any thoughts/advice?

      Thanks

      • So, I found that the product id of my kinect is not the same as in libkinectv2 library:

        //static const unsigned int ProductId = 0x02D8;
        static const unsigned int ProductId = 0x02ae;

        Making this change allows the device to be detected, but results in another problem – see below

        $ bin/Protonect
        Version: 0.2.0
        Environment variables: LOGFILE=
        Usage: bin/Protonect [-gpu=] [gl | cl | clkde | cuda | cudakde | cpu] []
        [-noviewer] [-norgb | -nodepth] [-help] [-version]
        [-frames ]
        To pause and unpause: pkill -USR1 Protonect
        [Info] [Freenect2Impl] enumerating devices…
        [Info] [Freenect2Impl] 11 usb devices connected
        [Info] [Freenect2Impl] found valid Kinect v2 @1:52 with serial A00367D05782136A
        [Info] [Freenect2Impl] found 1 devices
        [Info] [Freenect2DeviceImpl] opening…
        [Error] [protocol::UsbControl] failed to claim interface with IrInterfaceId(=1)! LIBUSB_ERROR_NOT_FOUND Entity not found. Try debugging with environment variable: export LIBUSB_DEBUG=3 .
        [Info] [Freenect2DeviceImpl] closing…
        libusb: error [submit_bulk_transfer] submiturb failed error -1 errno=2
        [Error] [protocol::CommandTransaction] bulk transfer failed: LIBUSB_ERROR_IO Input/Output Error
        libusb: error [submit_bulk_transfer] submiturb failed error -1 errno=2
        [Error] [protocol::CommandTransaction] bulk transfer failed: LIBUSB_ERROR_IO Input/Output Error
        [Info] [Freenect2DeviceImpl] deallocating usb transfer pools…
        [Info] [Freenect2DeviceImpl] closing usb device…
        [Info] [Freenect2DeviceImpl] closed
        [Error] [Freenect2Impl] failed to open Kinect v2: @1:52
        failure opening device!

        After poking around, I found that this error shows when the kinect is connected to a usb 2.0 port. I’m connecting directly to the tx2 (blue) USB port which I expected to be 3.0, but running lsusb suggests otherwise. According to lsusb -t, Bus 2 is USB 3.0 and Bus 1 is USB 2.0:

        /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-tegra/3p, 5000M
        |__ Port 1: Dev 6, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 3: Dev 7, If 0, Class=Hub, Driver=hub/4p, 5000M
        /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-tegra/4p, 480M
        |__ Port 1: Dev 45, If 0, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 1: Dev 45, If 1, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 1: Dev 45, If 2, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 2: Dev 46, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 3: Dev 47, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 4: Dev 49, If 0, Class=Hub, Driver=hub/3p, 480M
        |__ Port 1: Dev 51, If 0, Class=Vendor Specific Class, Driver=, 480M
        |__ Port 2: Dev 50, If 0, Class=Vendor Specific Class, Driver=, 12M
        |__ Port 3: Dev 52, If 0, Class=Vendor Specific Class, Driver=, 480M

        And the output of lsusb clearly shows kinect is attached to bus 1. Is there something I need to do in the TX2 to enable USB 3.0? I’m feeling little like Alice at this point.

        Bus 002 Device 007: ID 2109:8110 VIA Labs, Inc. Hub
        Bus 002 Device 006: ID 2109:8110 VIA Labs, Inc. Hub
        Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
        Bus 001 Device 052: ID 045e:02ae Microsoft Corp. Xbox NUI Camera
        Bus 001 Device 050: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor
        Bus 001 Device 051: ID 045e:02ad Microsoft Corp. Xbox NUI Audio
        Bus 001 Device 049: ID 0409:005a NEC Corp. HighSpeed Hub
        Bus 001 Device 047: ID 2109:2811 VIA Labs, Inc. Hub
        Bus 001 Device 046: ID 2109:2811 VIA Labs, Inc. Hub
        Bus 001 Device 045: ID 046d:c52b Logitech, Inc. Unifying Receiver
        Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

          • I’m running 28.1. If I plug in a USB 3.0 hub, it shows on Bus 2 (3.0), but the kinect along with my mouse/keyboard dongle (clearly 2.0) show up on Bus 1 (2.0). That suggests to me that the kinect is a 2.0 device, yes? I assumed it was 3.0 because of the libfreenect wiki site says a 3.0 port is required. If I plug the kinect into the USB 3.0 hub, the kinect still ends up on bus 1.

            The fact that libfreenect2 does not support the kinect model 1473 (the product id is different) may also suggest a larger support problem. Thanks.

          • Yep, that is the case and certainly explains some (if not all) of my difficulties. Thank you for noticing. I’ll give libfreenect a try.

  8. I have successfully built libfreenect on TX2 and I am able to run most of the examples (all except audio). Thanks again for your feedback and assistance.

Leave a Reply

Your email address will not be published.


*