JetsonHacks

Developing on NVIDIA® Jetson™ for AI on the Edge

Building OpenNI2 for the Structure Sensor


Porting to a new platform is rarely easy. The steps are pretty straightforward, but all the niggles add up to make it easy to fiddle away the better part of a couple days for even easy projects. For a new development board, this is especially true. In my case, I haven’t been around Linux recently enough to be entirely comfortable.

With that said, here’s all the tips and tricks I did to get OpenNI2 running on the Jetson to interface with the Occipital Structure Sensor. It might not be optimal, and it may not be absolutely correct, but it appears to work. We all know, to look good is to be good. Appearances are everything.

Remember this is all pretty low level stuff, drivers for the sensor, usb interface, etc. So there’s a lot of little stuff that needs to be done.

For the sake of the video, I installed the dependencies before going on camera. You should be able to install the dependencies without any issues. Here’s the steps I took (the steps should follow the steps in the video):


# Install the dependencies for OpenNI2
$ sudo apt-get install -y g++ python libusb-1.0-0-dev freeglut3-dev doxygen graphviz
# Added 6-15-2015; Sometimes libudev is missing, add it just to be sure
$ sudo apt-get install libudev-dev
# Navigtate to where you would like to build, then:
$ git clone https://github.com/occipital/OpenNI2.git
$ cd OpenNI2
# Set the compile flags to build for the Jetson
$ gedit ThirdParty/PSCommon/BuildSystem/Platform.Arm
Change:
CFLAGS += -march=armv7-a -mtune=cortex-a9 -mfpu=neon -mfloat-abi=softfp #-mcpu=cortex-a8
to:
CFLAGS += -march=armv7-a -mtune=cortex-a15 -mfpu=neon-vfpv4 -mfloat-abi=hard
# Save the file
# You should also modify ThirdParty/PSCommon/BuildSystem/CommonCppMakefile
# to add the lpthread library.

$ gedit ThirdParty/PSCommon/BuildSystem/CommonCppMakefile

Here's the diff:

--- OpenNI2-2.2.0.30/ThirdParty/PSCommon/BuildSystem/CommonCppMakefile.old 2014-03-28 19:09:11.572263107 -0700
+++ OpenNI2-2.2.0.30/ThirdParty/PSCommon/BuildSystem/CommonCppMakefile 2014-03-28 19:09:55.600261937 -0700
@@ -95,6 +95,9 @@
OUTPUT_NAME = $(EXE_NAME)
# We want the executables to look for the .so's locally first:
LDFLAGS += -Wl,-rpath ./
+ ifneq ("$(OSTYPE)","Darwin")
+ LDFLAGS += -lpthread
+ endif
OUTPUT_COMMAND = $(CXX) -o $(OUTPUT_FILE) $(OBJ_FILES) $(LDFLAGS)
endif
ifneq "$(SLIB_NAME)" ""

# Save the file
# Add Samples and Tools to the build chain if desired
$ gedit Makefile
# Add these two lines to the end of the file:
core_samples: $(CORE_SAMPLES)
tools: $(ALL_TOOLS)
# Save the file
# You're ready to build!
$ make
$ make core_samples # this probably isn't necessary, they should already be built
#GLUT_SUPPORTED tells the make to compile NiViewer for OpenGL
$ GLUT_SUPPORTED=1 make tools

# Build and install libfreenect
$ sudo apt-get install cmake freeglut3-dev pkg-config build-essential libxmu-dev libxi-dev libusb-1.0-0-dev -y

$ git clone git://github.com/OpenKinect/libfreenect.git
$ cd libfreenect
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install
# Build the OpenNI2 driver
$ cmake .. -DBUILD_OPENNI2_DRIVER=ON
$ make
# Copy libfreenect to the OpenNI2 driver directory
$ Repository=../../Bin/Arm-Release/OpenNI2/Drivers
$ cp -L lib/OpenNI2-FreenectDriver/libFreenectDriver* ${Repository}
# You may have to set permissions to be able to use the Sensor
$ sudo usermod -a -G video ubuntu
# Switch back to OpenNI2 directory here
# Now copy the libraries and include files to /usr
$ sudo cp -r Include /usr/include/openni2
$ sudo cp -r Bin/Arm-Release/OpenNI2 /usr/lib/
$ sudo cp Bin/Arm-Release/libOpenNI2.* /usr/lib/
# Create a package config file
# this will enable ubuntu to find the location of the drivers, libraries and include files.
$ sudo gedit /usr/lib/pkgconfig/libopenni2.pc

and fill it with this:

prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include/openni2

Name: OpenNI2
Description: A general purpose driver for all OpenNI cameras.
Version: 2.2.0.0
Cflags: -I${includedir}
Libs: -L${libdir} -lOpenNI2 -L${libdir}/OpenNI2/Drivers -lDummyDevice -lOniFile -lPS1080.so

# To make sure it is correctly found, run
$ pkg-config --modversion libopenni2

# which should give the same version as defined in the file above (2.2.0.0)

# Linux has used the udev system to handle devices such as USB connected peripherals.

$ cd Packaging/Linux
$ sudo cp primesense-usb.rules /etc/udev/rules.d/557-primesense-usb.rules

At this point, you should be good to go. Switch over to the Bin/Arm-Release directory and run NiViewer to make sure it works. I’ll point out that you’ll need to plug in the sensor and have it running before NiViewer will work. It might be embarrassing to try to figure out why it’s not working when it’s not plugged in. It’s even more embarrassing if you happen to be on camera when you’re trying to demo. Don’t ask me how I know that. Also, please see the note below.

Note on using ASUS XTION PRO

If you have an ASUS XTION Pro, Walter Lucetti has step by step instructions for installation of OpenNI2 here:

http://myzharbot.robot-home.it/blog/software/configuration-nvidia-jetson-tk1/asus-xtion-pro-live-openni2-compilation-install-instructions/

Note on USB Autosuspend

There is an issue with the default setting for USB ports being turned off when not in use. The Jetson does this to save energy. If this happens when the Structure is connected, the Jetson will not recognize the Structure. A solution is to execute commands automatically on every boot-up that alter this behavior. You can put your commands into the “/etc/rc.local” boot up script. For example, run this on your board:

$ sudo gedit /etc/rc.local

Then add this near the bottom of the file but before the “exit” line:

# Disable USB auto-suspend, since it disconnects some devices such as webcams on Jetson TK1.
echo -1 > /sys/module/usbcore/parameters/autosuspend

Save the file.

You then reboot the machine so that the changes take effect.

$ sudo reboot

NOTE: If you would like to do this more selectively, you can enter this command on the command line:

sudo bash -c ‘echo -1 > /sys/module/usbcore/parameters/autosuspend’

which will cause auto suspend to be turned off until the next boot cycle.

Note that you can also modify the kernel firmware. That’s beyond the scope of this article, but you can read more about it here: Installing Grinch LinuxForTegra (L4T) on NVIDIA Jetson TK1

Facebook
Twitter
LinkedIn
Reddit
Email
Print

55 Responses

  1. Hi, I’m following your guide to install OpenNI2 on my Jetson TK1 to be used with the Asus XTion PRo Live on my robot.
    I have a doubt: do I need libfreenect driver? I thoughtit was needed only for Microsoft Kinect… doesn’t it?

    Anyway, good job… this is a really clear guide!!!

    1. Hi Myzhar,
      The libfreenect driver is a device driver for PrimeSense based devices. The Kinect, the Asus XTion Pro Live, and the Occipital Structure Sensor are all based on the PrimeSense chipsets. It is my understanding that you can use libfreenect as the driver, and must configure it as an OpenNI2 driver as noted above. Of course you could always try it without it, and see what happens.

      When they first started, libfreenect and OpenNI were two “competing” projects and were not at all compatible. Over the last couple of years, people started integrating OpenNI with libfreenect as the device driver. It is unclear to me what the minimal configuration is. I believe that ROS also has some integration with depth cameras too, including the ASUS.

      Your robot looks like it’s coming along, it was great to see the videos from Makers Faire!

  2. This article is superb! I was trying to install OPENNI one my TK1 as well as on my pcduino for days without luck! something used to fail. But then your article came along and installation was a breeze. Can’t thank you enough. I got my Xtion working on tk1 now (atleast the NiViewer is showing depth). Now I’ll get to building a depth sensing robot based on the tk1.

    I’ll also try to use your instructions on the pcduino and see if it works.

    Thanks again

  3. Thank you so much for doing this! One additional dependency I found that I was missing was libudev which I got with apt-get install libudev-dev.

    1. Hi edu,
      Thank you for reading the article!

      The dependency is apparently from a change made in the source since this article was published. I have added it to the above code snippet. Thanks for pointing this out!

  4. Hi, I having trouble installing openni2 on my jetson.

    When I run the command

    cp -L lib/OpenNI2-FreenectDriver/libFreenectDriver* ${Repository}

    I get this error

    cp: target ‘”../../Bin/Arm-Release/OpenNI2/Drivers”’ is not a directory

    Any help would be great.

    1. Hi Kalvik,
      You’ll need to check if the directory is actually there. This step is at here in the video: https://youtu.be/Bn9WqbYtNBw?t=6m4s
      Of course the full path will be different on your machine. You should be in the ‘OpenNI2/libfreenect2/build’ directory. The full repository path looks to be ‘OpenNI2/Bin/Arm-Release/OpenNI2/Drivers’
      You should check to make sure that it there.

      1. I am trying to make this work for libfreenect2. What would be my command for it?
        cp -L lib/OpenNI2-FreenectDriver/libFreenectDriver* ${Repository}

  5. I went the the
    lib/OpenNI2-FreenectDriver/
    and the folder contains 3 files with the name libFreenectDriver with different extensions.

    So it is the right directory right?
    But I still get the error

    1. Hmm, you’re trying to copy the binary files into the directory ‘OpenNI2/Bin/Arm-Release/OpenNI2/Drivers’
      Your error message says that the directory does not exist. Is that the case?

  6. I went to the ‘OpenNI2/Bin/Arm-Release/OpenNI2/Drivers’ directory and it exists but I keep getting the error. So I went ahead and copies the files from the ‘lib/OpenNI2-FreenectDriver/’ file to ‘OpenNI2/Bin/Arm-Release/OpenNI2/Drivers’ and then when I run
    $ pkg-config –modversion libopenni2
    from OpenNI2 directory the terminal dosent show me anything.

    1. Hi Kalvik,
      It looks like there’s some formatting issues with the code. Sorry, horrible teething pains, but thank you for catching it.

      Try:
      Repository=../../Bin/Arm-Release/OpenNI2/Drivers
      $ cp -L lib/OpenNI2-FreenectDriver/libFreenectDriver* ${Repository}

      and:

      $ pkg-config –-modversion libopenni2

      There were no quotes in the Repository line. There are two hyphens instead of one in the pkg-config command.

  7. Reading DIFFs gives me a headache, not to mention I still might do it wrong.

    Any way you can take:

    $ gedit ThirdParty/PSCommon/BuildSystem/CommonCppMakefile

    Here’s the diff:

    — OpenNI2-2.2.0.30/ThirdParty/PSCommon/BuildSystem/CommonCppMakefile.old 2014-03-28 19:09:11.572263107 -0700
    +++ OpenNI2-2.2.0.30/ThirdParty/PSCommon/BuildSystem/CommonCppMakefile 2014-03-28 19:09:55.600261937 -0700
    @@ -95,6 +95,9 @@
    OUTPUT_NAME = $(EXE_NAME)
    # We want the executables to look for the .so’s locally first:
    LDFLAGS += -Wl,-rpath ./
    + ifneq (“$(OSTYPE)”,”Darwin”)
    + LDFLAGS += -lpthread
    + endif
    OUTPUT_COMMAND = $(CXX) -o $(OUTPUT_FILE) $(OBJ_FILES) $(LDFLAGS)
    endif
    ifneq “$(SLIB_NAME)” “”

    And just post what needs to be there ?

    Thanks,
    JT

  8. Hi Kangalow. I would like to know what changes need to be made for OpenNI2 to work with the Jetson TX1.

      1. I tried what you told me and the result I get is that the host platform cannot be determined. I also tried changing the cortex number with no success. Please help me with this issue.

        1. I’m also trying to compile. As far as I know, Tx1 use cortex-a57, so at least you need to change that line.

          1. I am also sitting with a TX1 here and have the same problem. When I am going to save the Makefile i get this;

            (gedit:10873): Gtk-CRITICAL **: _gtk_widget_captured_event: assertion ‘WIDGET_REALIZED_FOR_EVENT (widget, event)’ failed

            (gedit:10873): Gtk-CRITICAL **: _gtk_widget_captured_event: assertion ‘WIDGET_REALIZED_FOR_EVENT (widget, event)’ failed
            ubuntu@tegra-ubuntu:~/Downloads/OpenNI2$ gedit Makefile
            ubuntu@tegra-ubuntu:~/Downloads/OpenNI2$ make
            ThirdParty/PSCommon/BuildSystem/CommonDefs.mak:22: *** Can’t determine host platform. Stop.

    1. Hi, Jesse!

      Could you run that on Jetson TX1?

      I have this problem: ThirdParty/PSCommon/BuildSystem/CommonDefs.mak:22: *** Can’t determine host platform. Stop.

      Can someone help me with that?

  9. Quick question. Is OpenNI2 the only platform that can run the Structure sensor with the Jetson TX1?

    1. I believe that it just uses libfreenect
      You may have more success asking questions about the Structure sensor in their forums, I have limited experience with the device.

  10. hello,when I run this command that”cp -L lib/OpenNI2-FreenectDriver/libFreenectDriver* ${Repository}”, my full command is “haha@ubuntu:~/libfreenect/build$ cp -L lib/OpenNI2-FreenectDriver/libFreenectDriver* ${Repository}” Unfortunatelly,the error says that “cp: target ‘”../../Bin/Arm-Release/OpenNI2/Drivers”’ is not a directory” i don’t know if my path is wrong.BTW,i have read the question like mine,but I still can’t handle this trouble,thx

  11. Hi
    When I tried to run NiViewer, I got the follwoing error. Do you know how to fix them?

    $ sudo ./NiViewer
    OpenNI2-FreenectDriver: Using libfreenect v0.6.0
    OpenNI2-FreenectDriver: Found device freenect://0
    Found sibling device [same parent]
    Failed to set the LED of K4W or 1473 device: -4
    OpenNI2-FreenectDriver: Opening device freenect://0
    [Stream 70] Negotiated packet size 1920
    Failed to submit isochronous transfer 0: -1
    Failed to submit isochronous transfer 1: -1
    Failed to submit isochronous transfer 2: -1
    Failed to submit isochronous transfer 3: -1
    Failed to submit isochronous transfer 4: -1
    Failed to submit isochronous transfer 5: -1
    Failed to submit isochronous transfer 6: -1
    Failed to submit isochronous transfer 7: -1
    Failed to submit isochronous transfer 8: -1
    Failed to submit isochronous transfer 9: -1
    Failed to submit isochronous transfer 10: -1
    Failed to submit isochronous transfer 11: -1
    Failed to submit isochronous transfer 12: -1
    Failed to submit isochronous transfer 13: -1
    Failed to submit isochronous transfer 14: -1
    Failed to submit isochronous transfer 15: -1
    write_register: 0x0105 <= 0x00
    write_register: 0x0006 <= 0x00
    write_register: 0x0012 <= 0x03
    write_register: 0x0013 <= 0x01
    write_register: 0x0014 <= 0x1e
    write_register: 0x0006 <= 0x02
    write_register: 0x0017 <= 0x00
    [Stream 80] Negotiated packet size 1920
    Failed to submit isochronous transfer 0: -1
    Failed to submit isochronous transfer 1: -1
    Failed to submit isochronous transfer 2: -1
    Failed to submit isochronous transfer 3: -1
    Failed to submit isochronous transfer 4: -1
    Failed to submit isochronous transfer 5: -1
    Failed to submit isochronous transfer 6: -1
    Failed to submit isochronous transfer 7: -1
    Failed to submit isochronous transfer 8: -1
    Failed to submit isochronous transfer 9: -1
    Failed to submit isochronous transfer 10: -1
    Failed to submit isochronous transfer 11: -1
    Failed to submit isochronous transfer 12: -1
    Failed to submit isochronous transfer 13: -1
    Failed to submit isochronous transfer 14: -1
    Failed to submit isochronous transfer 15: -1
    write_register: 0x000c <= 0x00
    write_register: 0x000d <= 0x01
    write_register: 0x000e <= 0x1e
    write_register: 0x0005 <= 0x01
    write_register: 0x0047 <= 0x00
    ubuntu@tegra-ubuntu:/home/xiaodong/OpenNI/OpenNI2/Bin/Arm-Release$ sudo ./NiViewer
    OpenNI2-FreenectDriver: Using libfreenect v0.6.0
    OpenNI2-FreenectDriver: Found device freenect://0
    Found sibling device [same parent]
    Failed to set the LED of K4W or 1473 device: -4
    OpenNI2-FreenectDriver: Opening device freenect://0
    [Stream 70] Negotiated packet size 1920
    Failed to submit isochronous transfer 0: -1
    Failed to submit isochronous transfer 1: -1
    Failed to submit isochronous transfer 2: -1
    Failed to submit isochronous transfer 3: -1
    Failed to submit isochronous transfer 4: -1
    Failed to submit isochronous transfer 5: -1
    Failed to submit isochronous transfer 6: -1
    Failed to submit isochronous transfer 7: -1
    Failed to submit isochronous transfer 8: -1
    Failed to submit isochronous transfer 9: -1
    Failed to submit isochronous transfer 10: -1
    Failed to submit isochronous transfer 11: -1
    Failed to submit isochronous transfer 12: -1
    Failed to submit isochronous transfer 13: -1
    Failed to submit isochronous transfer 14: -1
    Failed to submit isochronous transfer 15: -1
    write_register: 0x0105 <= 0x00
    write_register: 0x0006 <= 0x00
    write_register: 0x0012 <= 0x03
    write_register: 0x0013 <= 0x01
    write_register: 0x0014 <= 0x1e
    write_register: 0x0006 <= 0x02
    write_register: 0x0017 <= 0x00
    [Stream 80] Negotiated packet size 1920
    Failed to submit isochronous transfer 0: -1
    Failed to submit isochronous transfer 1: -1
    Failed to submit isochronous transfer 2: -1
    Failed to submit isochronous transfer 3: -1
    Failed to submit isochronous transfer 4: -1
    Failed to submit isochronous transfer 5: -1
    Failed to submit isochronous transfer 6: -1
    Failed to submit isochronous transfer 7: -1
    Failed to submit isochronous transfer 8: -1
    Failed to submit isochronous transfer 9: -1
    Failed to submit isochronous transfer 10: -1
    Failed to submit isochronous transfer 11: -1
    Failed to submit isochronous transfer 12: -1
    Failed to submit isochronous transfer 13: -1
    Failed to submit isochronous transfer 14: -1
    Failed to submit isochronous transfer 15: -1
    write_register: 0x000c <= 0x00
    write_register: 0x000d <= 0x01
    write_register: 0x000e <= 0x1e
    write_register: 0x0005 <= 0x01
    write_register: 0x0047 <= 0x00

          1. This what I see from lsusb
            Bus 003 Device 040: ID 045e:02bf Microsoft Corp.
            Bus 003 Device 039: ID 045e:02be Microsoft Corp.
            Bus 003 Device 038: ID 045e:02c2 Microsoft Corp.

  12. It looks like Occipital has a download page for OpenNI, that includes precomipiled binaries for 2.2.0.33 on ARM (https://structure.io/openni ). Am I better off building from source on Jetson, or just downloading / installing the prebuilt packages?

  13. Tried following the above instructions, but when it comes time to run “Make”, I get the following error. Any ideas?

    ThirdParty/PSCommon/BuildSystem/CommonDefs.mak:22: *** Can’t determine host platform. Stop.

    1. Well, I updated CommonDefs.mak to be hardcoded to the “arm” platform, now I’m getting these errors:
      ===
      g++: error: unrecognized command line option ‘-mfpu=neon-vfpv4’
      g++: error: unrecognized command line option ‘-mfloat-abi=hard’

      1. I guess it doesn’t help that I’m trying to follow these instructions on a TX2. But it’s the closest to an actual working installation guide out there…

  14. Hi Kangalow. I would like to know what changes need to be made for OpenNI2 to work with the Xtion on Jetson TX2.

  15. Hi Kanglow,
    I am installing libfreenect2 and OPENNI2 to make Kinect V2 work. I had issues while executing the below statement.
    cp -L lib/OpenNI2-FreenectDriver/libFreenectDriver* ${Repository}

    the location of the OpenNI2 libraries on my local is as follows:
    /usr/lib/OpenNI2/Drivers
    There are four files uner it
    and my repository location is
    ../../Bin/x64-Release/OpenNI2/Drivers

    I executed the following command:
    cp -L lib/OpenNI2/Drivers* ${Repository}

    Now it throws me the following error
    cp: cannot stat ‘lib/OpenNI2/Drivers*’: No such file or directory
    Can you please let me know what mistake i am doing here? Thank you

      1. Hello Kanglow,
        I am using the instructions you posted above to make Kinect2 working on my machine. I have installed OPENNI2 and when i am trying to install libfrenect2 (the latest one, not the old one – libfreenect), i am stuck at one the steps.
        I am at the following location
        root@shankar:~/OpenNI2/libfreenect2/build$
        I am executing the following command at the above location
        cp -L lib/OpenNI2/Drivers* ${Repository}
        and i am getting error stated above

            1. The instructions given are for a ARM-64 embedded development system, not a x86 desktop. Please follow the instructions for your type of machine.

Leave a Reply

Your email address will not be published. Required fields are marked *

Disclaimer

Some links here are affiliate links. If you purchase through these links I will receive a small commission at no additional cost to you. As an Amazon Associate, I earn from qualifying purchases.

Books, Ideas & Other Curiosities