GStreamer is useful for handling media components on the NVIDIA Jetsons. This is the main way to handle video sources, such as cameras and video files.
It’s nice to have a GUI to inspect GStreamer components. Here’s a sample: https://github.com/jetsonhacks/gst-explorer.git on how one might do just that.
GStreamer uses a Pipeline architecture, where different Elements link together to guide input from a media or data source to an output. The output is called a ‘sink’. These pipelines provide a “rich” interface, a quick translation of which means “these things can be complex”.
For many people, the number of options available for linking together Elements and configuring the Elements themselves is confusing. Elements may act as a media source, a sink, a filter, a router, or a combination of the above.
Elements have properties for configuring parameters. The number of properties are variable. Some Elements are simple and need little configuration. Other Elements are complex and provide a wide range of fine tuning.
GStreamer uses a Plugin architecture. One way to think about a Plugin in GStreamer is as a dynamic library. Plugins contain executable code for Elements and a couple of other supporting atomic GStreamer types.
There are GStreamer command line tools for working with the Pipeline and Elements. The first is gst-launch, which allows construction of a GStreamer pipeline for running tests. The second tool is gst-inspect which lists the Plugins, Elements, Types and some other bits of information. Here we are looking at gst-inspect. Since we are using GStreamer 1.0, the formal names of the tools are gst-launch-1.0 and gst-inspect-1.0.
The base GStreamer system on the Jetson has 260 Plugins and well over 1200 different GStreamer elements! Typically running gst-inspect consists of running the command and watching text scroll by for quite a bit of time. Eventually you will either capture everything to a file so you can look at it in a text editor, or use grep on the command line to find items of interest.
You may ask for just one Plugin or Element, of course. Like most command line interfaces, you can pass flags to get different views on the GStreamer system. Or you can ask for a more verbose listing which will generate a megabyte or two of text. This might not be as useful as it sounds.
So, here’s a little challenge. Given a day or three, write a GUI in prototype form for inspecting the GStreamer Plugins and Elements.
The list of features is pretty small.
- A selectable List containing GStreamer Plugins, Elements and Types
- Text Search to help narrow down which items to view
- A Filter to separate the Plugins from the Elements from the Types
- An Inspector Pane which shows detailed information about a selected entry
The above application in the JetsonHacks Github repository is written in Python3 and uses Qt. The dependencies are already installed in the default Jetson images.
The code uses a model, view, controller architecture. How would you build this?
Prototypes vs Production Code
It is worth noting the difference between prototype and production code. This is a prototype. People approach production code in a much different manner. Prototypes are typically the work of one person or a very small group. Production code begins with the assumption that there will be a large group of people working on the code, and that a large number of people will be using the code.
Prototypes are meant to be thrown away! These are sketches of interesting ideas. That does not mean that they are not useful. Indeed, that’s the whole point! In the physical world, this is why people build models. Getting hands on experience with what started out as an idea can help form the idea into something more powerful.
Production code is very much different. The scaffold for building is a project in and of itself. You must know what you are building beforehand. For example, if you were to start building production code you have a set of requirements that you incorporate right at the beginning, such as:
- Project Management
- Localization/Internationalization for language
- Error Handling/Exceptions
- Testing Harnesses
- Unit Tests
- Integration Tests
- Acceptance Tests
- Architecture Design
- Engineering Specifications
- Build System
- Source Control
The list above is just for starters. Production systems are not something just one person builds. To be clear, when working on production code, the actually writing of the code itself is not the majority of the work!
- Tested on NVIDIA Jetson Nano and Jetson Xavier NX Developer Kits
- JetPack 4.6, L4T 32.6.1