OpenCV, Python, Onboard Camera – NVIDIA Jetson TX2

In this article, we build a simple demonstration of a Canny Edge Detector using OpenCV, Python, and the onboard camera of the NVIDIA Jetson TX2 Development Kit. Looky here:

Background

Back in 1986, John F. Canny developed the Canny Edge detector. The Canny Edge is one of the image processing milestones which is still in use today.
You can read some more about the Canny Edge Detector and the technical details here: OpenCV.org Canny Edge Detector and here: Wikipedia – Canny edge detector

In this article we will use a simple Python script which will use the OpenCV library implementation of the Canny Edge Detector to read the onboard camera and run the frames through the filter. Earlier we went over on how to build the OpenCV library for the Jetson. There is a repository on the JetsonHacks Github account which contains a build script to help in the process. You will need to enable GStreamer support. As of this writing the current script enables GStreamer support (OpenCV 3.3), while earlier versions did not. GStreamer must be enabled to support the onboard camera. Note: The standard OpenCV4Tegra installed by JetPack does not have GStreamer support enabled, so the onboard camera will not work with it.

The Codez

The file cannyDetection.py is available in the Examples folder of the buildOpenCVTX2 repository on the JetsonHacks Github account. The script is also available as a Github Gist. The Gist is seen in its entirety further down below.

GStreamer Camera Pipeline

The first task is to open the onboard camera. Camera access is through a GStreamer pipeline:

cap = cv2.VideoCapture("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink")

Canny Edge Detector

The main part of the filter processing reads a frame from the camera, converts it to gray scale, runs a gaussian blur on the gray scale image, and then runs the Canny Edge Detector on that result:

ret_val, frame = cap.read();
hsv=cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
blur=cv2.GaussianBlur(hsv,(7,7),1.5)
edges=cv2.Canny(blur,0,edgeThreshold)

Not surprisingly, it is more concise to just write the code. In the code, both the GaussianBlur and Canny functions have parameters to fine tune the results.

At this point, we could simply display it on a window on the screen:

cv2.imshow('Canny Edge Detector', edges)

In the script, we add a little user interface sugar which allows us to optionally display each step. The only interesting part is that the image for each step is composited into one larger frame. This requires that the images be converted to the same color space before compositing. In the video, the Jetson TX2 is set to run at maximum performance (See note below). Here’s the full script:

#!/usr/bin/env python
# MIT License
# Copyright (c) 2017 Jetsonhacks
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
import sys
import cv2
import numpy as np
def read_cam():
# On versions of L4T previous to L4T 28.1, flip-method=2
# Use the Jetson onboard camera
cap = cv2.VideoCapture("nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink")
if cap.isOpened():
windowName = "CannyDemo"
cv2.namedWindow(windowName, cv2.WINDOW_NORMAL)
cv2.resizeWindow(windowName,1280,720)
cv2.moveWindow(windowName,0,0)
cv2.setWindowTitle(windowName,"Canny Edge Detection")
showWindow=3 # Show all stages
showHelp = True
font = cv2.FONT_HERSHEY_PLAIN
helpText="'Esc' to Quit, '1' for Camera Feed, '2' for Canny Detection, '3' for All Stages. '4' to hide help"
edgeThreshold=40
showFullScreen = False
while True:
if cv2.getWindowProperty(windowName, 0) < 0: # Check to see if the user closed the window
# This will fail if the user closed the window; Nasties get printed to the console
break;
ret_val, frame = cap.read();
hsv=cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
blur=cv2.GaussianBlur(hsv,(7,7),1.5)
edges=cv2.Canny(blur,0,edgeThreshold)
if showWindow == 3: # Need to show the 4 stages
# Composite the 2×2 window
# Feed from the camera is RGB, the others gray
# To composite, convert gray images to color.
# All images must be of the same type to display in a window
frameRs=cv2.resize(frame, (640,360))
hsvRs=cv2.resize(hsv,(640,360))
vidBuf = np.concatenate((frameRs, cv2.cvtColor(hsvRs,cv2.COLOR_GRAY2BGR)), axis=1)
blurRs=cv2.resize(blur,(640,360))
edgesRs=cv2.resize(edges,(640,360))
vidBuf1 = np.concatenate( (cv2.cvtColor(blurRs,cv2.COLOR_GRAY2BGR),cv2.cvtColor(edgesRs,cv2.COLOR_GRAY2BGR)), axis=1)
vidBuf = np.concatenate( (vidBuf, vidBuf1), axis=0)
if showWindow==1: # Show Camera Frame
displayBuf = frame
elif showWindow == 2: # Show Canny Edge Detection
displayBuf = edges
elif showWindow == 3: # Show All Stages
displayBuf = vidBuf
if showHelp == True:
cv2.putText(displayBuf, helpText, (11,20), font, 1.0, (32,32,32), 4, cv2.LINE_AA)
cv2.putText(displayBuf, helpText, (10,20), font, 1.0, (240,240,240), 1, cv2.LINE_AA)
cv2.imshow(windowName,displayBuf)
key=cv2.waitKey(10)
if key == 27: # Check for ESC key
cv2.destroyAllWindows()
break ;
elif key==49: # 1 key, show frame
cv2.setWindowTitle(windowName,"Camera Feed")
showWindow=1
elif key==50: # 2 key, show Canny
cv2.setWindowTitle(windowName,"Canny Edge Detection")
showWindow=2
elif key==51: # 3 key, show Stages
cv2.setWindowTitle(windowName,"Camera, Gray scale, Gaussian Blur, Canny Edge Detection")
showWindow=3
elif key==52: # 4 key, toggle help
showHelp = not showHelp
elif key==44: # , lower canny edge threshold
edgeThreshold=max(0,edgeThreshold1)
print 'Canny Edge Threshold Maximum: ',edgeThreshold
elif key==46: # , raise canny edge threshold
edgeThreshold=edgeThreshold+1
print 'Canny Edge Threshold Maximum: ', edgeThreshold
elif key==74: # Toggle fullscreen; This is the F3 key on this particular keyboard
# Toggle full screen mode
if showFullScreen == False :
cv2.setWindowProperty(windowName, cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN)
else:
cv2.setWindowProperty(windowName, cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_NORMAL)
showFullScreen = not showFullScreen
else:
print "camera open failed"
if __name__ == '__main__':
read_cam()

view raw
CannyDetection.py
hosted with ❤ by GitHub

Conclusion

Many people use OpenCV for everyday vision processing tasks, and the Canny Edge Detector is a valuable tool. The purpose of this article is to show how to access the onboard camera using GStreamer in a Python script with OpenCV. More importantly, I played guitar in the video.

Notes

  • In the video, the Jetson TX2 is running L4T 28.1, OpenCV 3.3 with GStreamer support enabled
  • In the video, the Jetson TX2 is running ‘$ sudo nvpmodel -m 0’

10 Comments

  1. Hi Jim thanks you for these great tutorials. I would like save a video with fourcc = cv2.VideoWriter_fourcc(*’XVID’)out = cv2.VideoWriter(‘output.avi’,fourcc, 20.0, (640,480)) However i am getting an error when trying play video “Could not demultiplex stream” in loading Video recorded

    Thank you again

      • Thank you jim. I’m getting the same error with all codecs installed. I’m going crazy :(.

        Could you update this great post with your python code to record the screen or record/save to a file, the camera videostreaming? It would be a great help.

        Thanks a lot again!

  2. Thank you so much for your videos. Like Hyperrjas, I’m also having issues with recording the video. I’m very new to programming in general so I’m fumbling my way through all of this.

    Any chance you could throw in an update to show how you would record the video.

    • hey I was able to figured it out. I made a chance in the fourcc call and used ‘M’,’J’,’P’,’G’ for my supplied argument. this canceled any errors I was getting and allowed me to write to file.

      Hyperrjas try this if your still having issues.

      fHeight = int(cap.get(3))
      fWidth = int(cap.get(4))
      fourcc = cv2.VideoWriter_fourcc(‘M’,’J’,’P’,’G’)
      out = cv2.VideoWriter(‘output.avi’, fourcc, 30, (fWidth, fHeight)
      out.write(frame) #inside while loop

  3. I’m still having the same error.

    Could you paste/share your code like gist recipe? I would like take a look to know where is my code problem.

    I would like to record canny edge option.

    Thank you so much again :).

    Regards

  4. Hello,
    thank you for the code, it’s helpful as I wanted to access the camera on board. I having one issue regarding memory, after sometime the kernel kills the application as memory consumption is too much. Is there any way by which i can use the camera continuously without killing the application.

    thanks

  5. thank you for t reply 🙂

    L4t Version: R28.2
    OpenCv: 4.0.0-pre
    No I didn’t modify the code, before using the application for my purpose i was just checking the code.

    • How long does it run before it runs out of memory? Are you running any other programs? If you open the System Monitor and monitor memory usage, does the memory gradually grow every few minutes? I ran the program overnight and did not experience any issues.

Leave a Reply

Your email address will not be published.


*