GStreamer xvimage

The NVIDIA Jetson TK1 uses Gstreamer as its official multi-media interface. In previous entries we’ve installed two webcams, a Microsoft LifeCam Studio and a Logitech c920. There are several longer range goals with utilizing the webcams, but first up is to show them on the screen. Here’s the video of what they look like:

I’d like to tell you that it’s a straightforward process. Gstreamer tends to be a sprawling, “I can do everything” type of program with plugins galore. The architecture is that of a pipeline, starting at a “source” and ending at a “sink”.

To me, the nomenclature that they use does not match anything in the naming convention that computer programmers have used for the last 60 years. In the world of computers that I dwell in, you have “input” and “output”. I can understand the concept of “source” and “sink”. However, the idea that there are components in the pipeline that act both as a “sink” and a “source” is just strange. I translate that in my head as a “source” is an output, and a “sink” is the input side.

With that said, the Jetson is a little confusing on which version of Gstreamer to use. In the initial release, the documentation indicated Gstreamer 0.10, when LT4 19.3 was release, the release notes stated that Gstreamer 1.0 was now supported. It’s not quite clear what that means in either context, or what the full extent of supported means.

So with a lot of trials, here are some of the first results. Both the Logitech and the Microsoft webcams deliver a variety of video resolutions and different encodings. The higher resolutions come in some sort of compressed format, MJPEG for both webcams. In addition, the Logitech also delivers H264.

I’ll use Gstreamer 1.0 for this example. It is straightforward to show one webcam using Gstreamer:


gst-launch-1.0 -v v4l2src device=/dev/video0 \
! image/jpeg, width=1920, height=1080, framerate=30/1 \
! jpegparse ! jpegdec \
! videoconvert ! videoscale \
! xvimagesink sync=false

Here’s a shell file, gstreamerPreviewWebcam.sh which will allow you to run this example.

A simplified explanation:

Basically the source is Video 4 Linux Version 2 (v4l2src) using video device 0. Ask the camera for MJPG (image/jpeg) 1920×1080 resolution at a frame rate of 30 frames per second. Gstreamer calls this a capability. Because the video is encoded as MJPG (a compressed image), you have to convert it to something that the display will understand. The display is represented by xvimagesink in this example.

The ! (exclamation point, some people call it a bang) acts as a separator between different pipeline elements.

The first thing to do is to take the video stream coming from the webcam, and decode it into raw pixel data. jpegparse makes sure that the JPEG decoder (jpegdec) understands the version of MJPEG that the camera is speaking. jpegdec then converts it into a “raw” video RGB pixel format.

videoconvert makes sure that the RGB pixel format is suitable for xvimagesink. videoconvert may observe that the stream is already in the correct format and just perform a pass through. In this instance, I added videoscale so that the video will resize when the window is resized.

xvimagesink represents the window on the display. “sync=false” tells Gstreamer that it’s ok to drop frames if it gets behind. For a display, that’s probably a good policy as you don’t want to fall behind. For storing video streams, not so much.

It wasn’t horrible, horrible. So I figured two webcams wouldn’t be difficult either. I figured wrong, but it’s because the core issues were tweeners between the Cameras, USB, and Gstreamer.

First, here’s the shell file for two webcams, sideBySide.sh.

There is only one trick that you need to know. There is a Gstreamer pipeline element called a “tee” which allows for parallel execution routes. So to start up two webcams, create a tee and send one of the webcams through it. Mark the end of the path with the name of the tee, in this case splitter, followed by a period. Then describe the next step in the pipeline. A tee may have more than two elements. Having worked around physical pipelines for a long time, my brain made sure that it let me know how much I was making it hurt with terrible metaphors.

In any case, you can wire this all up and still not know the secret ingredient to getting larger resolution webcams running on USB. It turns out that these webcams reserve the amount of bandwidth on the USB device that their maximum resolution will generate. You can do the math, but I’ll just let you know that I couldn’t put these two 1920×1080 30fps MPEG USB 2.0 webcams on one USB 3.0 device. However, you can run them both at 1280×720 (720p).

If you try to run both at 1920×1080 you’ll get a “cannot allocate memory” error in the Terminal. Of course, you’ll think that has something to do with the program (it doesn’t), it’s simply that Gstreamer thinks that all of the bandwidth of the USB device has been reserved.

Anyway, at least we got a good demo out of it, and hopefully you’ll find some use for it too!

Note: You may have to install some of the GStreamer plugins to have this work correctly:

$ sudo apt-get install gstreamer1.0-tools gstreamer1.0-alsa
gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-
plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav

Another resource is the Jetson TK1/TEGRA Linux Driver Package Multimedia User Guide

10 Comments

  1. Hi,
    I tried to use Gstreamer in my Jetson Tk1 to show my see3cam_80 camera. I ran your code
    “gst-launch-1.0 -v v4l2src device=/dev/video0 \
    ! image/jpeg, width=1920, height=1080, framerate=30/1 \
    ! jpegparse ! jpegdec \
    ! videoconvert ! videoscale \
    ! xvimagesink sync=false”

    but it does not work, appears the next error message:

    “Setting pipeline to PAUSED …
    ERROR: Pipeline doesn’t want to pause.
    ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not initialise Xv output
    Additional debug info:
    xvimagesink.c(1765): gst_xvimagesink_open (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
    Could not open display (null)
    Setting pipeline to NULL …
    Freeing pipeline …”

    I will appreciate if you can help me, any suggestion will be helpfull
    thanks a lot

    • Hi Higuera,

      I am part of the camera team at e-consystems. The camera you are using (See3CAM_80) supports only YUV422 format, whereas you are trying to get JPEG data from the camera itself. You can modify the gstreamer command as follows to directly display the data from the camera.

      gst-launch-0.10 -v v4l2src device=/dev/video0 \
      ! “video/x-raw-yuv, format=(fourcc)YUY2, \
      width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1” \
      ! xvimagesink sync=false

      Also you need to be running this command from a GUI based terminal window and not through the serial port of the Jetson TK1.

      If you have any other queries regarding the camera, please feel free to contact sales@e-consystems.com or visit our website http://www.e-consystems.com to chat with our experts.

  2. hello thanks for you articles .I want to ask you how use the Gstreamer to achieve usb camera video capture and as same while I should send the video to internet server. how could i write the shell command .I am a student and search the TK1 only for a week . Thank you

  3. Hi,
    I tried to use Gstreamer in my Jetson to view my webcam I am using a cheap webcam I bought on ebay. I ran your code and changed the resolution to 640*420
    gst-launch-1.0 -v v4l2src device=/dev/video0 \
    ! image/jpeg, width=640, height=420, framerate=30/1 \
    ! jpegparse ! jpegdec \
    ! videoconvert ! videoscale \
    ! xvimagesink sync=false
    and I get this error

    Setting pipeline to PAUSED …
    ERROR: Pipeline doesn’t want to pause.
    ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Could not initialise Xv output
    Additional debug info:
    xvimagesink.c(1765): gst_xvimagesink_open (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
    XVideo extension is not available
    Setting pipeline to NULL …
    Freeing pipeline …

    Any help would be greatly appreciated.

1 Trackback / Pingback

  1. GStreamer Network Video Stream and Save to File - NVIDIA Jetson & Tegra Development

Leave a Reply

Your email address will not be published.


*