MicroPython-OpenCV

Welcome to SparkFun's MicroPython port of OpenCV! This is the first known MicroPython port of OpenCV, and as such, there may be some rough edges. Hardware support is limited to SparkFun products.

Quick Start

1. Flash MicroPython-OpenCV firmware
   • Back up any files you want to keep, they will be overwritten!
   • Download the latest firmware for your board from the Releases tab.
   • If you don't know how to flash firmware to your board, find your board here and follow the instructions using the OpenCV firmware.
   • After first boot, the opencv-examples directory will be automatically extraced to the MicroPython filesystem for easy access to all the examples.
2. Configure hardware driver initialization
   • The MicroPython port of OpenCV depends on hardware drivers to interface with cameras and displays. Drivers are built into the firmware, so there is no need to install them manually.
   • An example module called cv2_hardware_init is imported by all examples to initialize the drivers. You will likely need to edit the files for your specific hardware and board configuration.
3. Write and run OpenCV code
   • Any IDE should work, so use your favorite!
   • Start with the examples! Go through them in order, which will verify your hardware is working and demonstrate some basics of OpenCV. Read the comments to understand the differences with the MicroPython port.
   • The code block below contains snippets to highlight major features.

# Import OpenCV, just like any other Python environment!
import cv2 as cv

# Initialize hardware drivers by importing the example module (you'll likely 
# need to modify it for your specific hardware configuration).
from cv2_hardware_init import *

# Import ulab NumPy and initialize an image, almost like any other Python
# environment! 
from ulab import numpy as np
img = np.zeros((240, 320, 3), dtype=np.uint8)

# Call OpenCV functions just like any other Python environment!
img = cv.putText(img, "Hello OpenCV!", (50, 200), cv.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)
img = cv.Canny(img, 100, 200)

# Call `cv.imshow()`, almost like any other Python environment! Instead of a
# window name string, you pass a display driver that implements an `imshow()` 
# method that takes a NumPy array as input
cv.imshow(display, img)

# Call `cv.waitKey()`, just like any other Python environment! This waits for a
# key press on the REPL. Standard OpenCV requires this to update windows, but
# MicroPython OpenCV does not.
key = cv.waitKey(0)

# Use a camera, similar to any other Python environment! `cv.VideoCapture(0)`
# is not used in MicroPython OpenCV, the driver is initialized separately.
camera.open()
success, frame = camera.read()
camera.release()

# Call `cv.imread()` and `cv.imwrite()` to read and write images to and from
# the MicroPython filesystem, just like in any other Python environment! Can 
# also point to an SD card.
# 
# Note - only BMP and PNG formats are currently supported in MicroPython OpenCV
img = cv.imread("path/to/image.png")
success = cv.imwrite("path/to/image.png", img)

Hardware Support and Requirements

Hardware support in this repository is mostly limited to SparkFun products. The current list of supported proudcts is very small, but may be expanded in the future. Users are welcome to fork this repository to add support for other products, following our licence requirements. Assistance in adding support for other hardware will not be provided by SparkFun. We may consider pull requests that add support for additional hardware, see #Contributing.

The OpenCV firmware adds over 3MiB on top of the standard MicroPython firmware, which itself be up to 1MiB in size (depending on platform and board). You'll also want some storage space, so a board with at least 8MB of flash is recommended.

PSRAM is a requirement to do anything useful with OpenCV. A single 320x240 RGB888 frame buffer requires 225KiB of RAM; most processors only have a few hundred KiB of SRAM. Several frame buffers can be needed for even simple vision pipelines, so you really need at least a few MiB of RAM available. The more the merrier!

Below is the list of currently supported hardware:

MicroPython Devices

Status	Device	Notes
✔️	XRP Controller

Camera Drivers

Status	Device	Notes
✔️	HM01B0
⚠️	OV5640	See #22

Display Drivers

Status	Device	Notes
✔️	ST7789

Touch Screen Drivers

Status	Device	Notes
✔️	CST816

Performance

Limit your expectations. OpenCV typically runs on full desktop systems containing processors running at GHz speeds with dozens of cores optimized for computing speed. In contrast, microcontrollers processors typically run at a few hundred MHz with 1 or 2 cores optimized for low power consumtion. Exact performance depends on many things, including the processor, vision pipeline, image resolution, colorspaces used, RAM available, etc. But for reference, the RP2350 can run the SparkFun Logo Detection Example at 2 to 2.5 FPS at 320x240 resolution.

Something to consider is that MicroPython uses a garbage collector for memory management. As images are created and destroyed in a vision pipeline, RAM will be consumed until the garbage collector runs. The collection process takes longer with more RAM, so this can result in noticable delays during collection (typically a few hundred milliseconds). To mitigate this, it's best to pre-allocate arrays and utilize the optional dst argument of OpenCV functions to avoid allocating new arrays when possible. Pre-allocation also helps improve performance by avoiding repeated delays from allocating memory.

Below are some typical execution times for various OpenCV functions. All were tested on the XRP (RP2350) with the splash image, which is 320x240.

Function	Execution Time
dst = cv.blur(src, (5, 5))	115ms
dst = cv.blur(src, (5, 5), dst)	87ms
retval, dst = cv.threshold(src, 127, 255, cv.THRESH_BINARY)	76ms
retval, dst = cv.threshold(src, 127, 255, cv.THRESH_BINARY, dst)	46ms
dst = cv.cvtColor(src, cv.COLOR_BGR2HSV)	114ms
dst = cv.cvtColor(src, cv.COLOR_BGR2HSV, dst)	84ms
dst = cv.Canny(src, 100, 200)	504ms
dst = cv.Canny(src, 100, 200, dst)	482ms

Another way to improve performance is to select the best hardware drivers for your setup. For example, the default SPI driver for the ST7789 is limited to the max SPI baudrate for the processor's SPI peripheral. That's 24MHz in the case of the RP2350, but another driver is provided that uses the PIO peripheral that runs at 75MHz, so displaying images can be ~3x faster (ignoring any required colorspace conversions).

For users wanting maximum performance, it may be desireable to bypass the high-level functions of the display/camera drivers, and instead work directly with the buffer member variables and read/write functions. This can avoid computationally expensive colorspace conversions when reading and writing images if they're not needed, but this is for advanced users only.

Included OpenCV Functions

Below is a list of all OpenCV functions included in the MicroPython port of OpenCV. This section follows OpenCV's module structure.

Only the most useful OpenCV functions are included. The MicroPython environment is extremely limited, so many functions are omitted due to prohibitively high RAM and firmware size requirements. Other less useful functions have been omitted to reduce firmware size. If there are additional functions you'd like to be included, see #Contributing.

If you need help understanding how to use these functions, see the documentation link for each function. You can also check out OpenCV's Python Tutorials and other tutorials online for more educational experience. This repository is simply a port of OpenCV, so we do not document these functions or how to use them, except for deviations from standard Python OpenCV.

core

Note

The core module includes many functions for basic operations on arrays. Most of these can be performed by numpy operations, so they have been omitted to reduce firmware size.

Operations on arrays

Function	Notes
cv.convertScaleAbs(src[, dst[, alpha[, beta]]]) -> dst Scales, calculates absolute values, and converts the result to 8-bit. Documentation
cv.inRange(src, lowerb, upperb[, dst]) -> dst Checks if array elements lie between the elements of two other arrays. Documentation
cv.minMaxLoc(src[, mask]) -> minVal, maxVal, minLoc, maxLoc Finds the global minimum and maximum in an array. Documentation

imgproc

Image Filtering

Function	Notes
cv.bilateralFilter(src, d, sigmaColor, sigmaSpace[, dst[, borderType]]) -> dst Applies the bilateral filter to an image. Documentation
cv.blur(src, ksize[, dst[, anchor[, borderType]]]) -> dst Blurs an image using the normalized box filter. Documentation
cv.boxFilter(src, ddepth, ksize[, dst[, anchor[, normalize[, borderType]]]]) -> dst Blurs an image using the box filter. Documentation
cv.dilate(src, kernel[, dst[, anchor[, iterations[, borderType[, borderValue]]]]]) -> dst Dilates an image by using a specific structuring element. Documentation
cv.erode(src, kernel[, dst[, anchor[, iterations[, borderType[, borderValue]]]]]) -> dst Erodes an image by using a specific structuring element. Documentation
cv.filter2D(src, ddepth, kernel[, dst[, anchor[, delta[, borderType]]]]) -> dst Convolves an image with the kernel. Documentation
cv.GaussianBlur(src, ksize, sigmaX[, dst[, sigmaY[, borderType[, hint]]]]) -> dst Blurs an image using a Gaussian filter. Documentation
cv.getStructuringElement(shape, ksize[, anchor]) -> retval Returns a structuring element of the specified size and shape for morphological operations. Documentation
cv.Laplacian(src, ddepth[, dst[, ksize[, scale[, delta[, borderType]]]]]) -> dst Calculates the Laplacian of an image. Documentation
cv.medianBlur(src, ksize[, dst]) -> dst Blurs an image using the median filter. Documentation
cv.morphologyEx(src, op, kernel[, dst[, anchor[, iterations[, borderType[, borderValue]]]]]) -> dst Performs advanced morphological transformations. Documentation
cv.Scharr(src, ddepth, dx, dy[, dst[, scale[, delta[, borderType]]]]) -> dst Calculates the first x- or y- image derivative using Scharr operator. Documentation
cv.Sobel(src, ddepth, dx, dy[, dst[, ksize[, scale[, delta[, borderType]]]]]) -> dst Calculates the first, second, third, or mixed image derivatives using an extended Sobel operator. Documentation
cv.spatialGradient(src[, dx[, dy[, ksize[, borderType]]]]) -> dx, dy Calculates the first order image derivative in both x and y using a Sobel operator. Documentation

Miscellaneous Image Transformations

Function	Notes
cv.adaptiveThreshold(src, maxValue, adaptiveMethod, thresholdType, blockSize, C[, dst]) -> dst Applies an adaptive threshold to an array. Documentation
cv.threshold(src, thresh, maxval, type[, dst]) -> retval, dst Applies a fixed-level threshold to each array element. Documentation

Drawing Functions

Function	Notes
cv.arrowedLine(img, pt1, pt2, color[, thickness[, line_type[, shift[, tipLength]]]]) -> img Draws an arrow segment pointing from the first point to the second one. Documentation
cv.circle(img, center, radius, color[, thickness[, lineType[, shift]]]) -> img Draws a circle. Documentation
cv.drawContours(image, contours, contourIdx, color[, thickness[, lineType[, hierarchy[, maxLevel[, offset]]]]]) -> image Draws contours outlines or filled contours. Documentation
cv.drawMarker(img, position, color[, markerType[, markerSize[, thickness[, line_type]]]]) -> img Draws a marker on a predefined position in an image. Documentation
cv.ellipse(img, center, axes, angle, startAngle, endAngle, color[, thickness[, lineType[, shift]]]) -> img Draws a simple or thick elliptic arc or fills an ellipse sector. Documentation
cv.fillConvexPoly(img, points, color[, lineType[, shift]]) -> img Fills a convex polygon. Documentation
cv.fillPoly(img, pts, color[, lineType[, shift[, offset]]]) -> img Fills the area bounded by one or more polygons. Documentation
cv.line(img, pt1, pt2, color[, thickness[, lineType[, shift]]]) -> img Draws a line segment connecting two points. Documentation
cv.putText(img, text, org, fontFace, fontScale, color[, thickness[, lineType[, bottomLeftOrigin]]]) -> img Draws a text string. Documentation
cv.rectangle(img, pt1, pt2, color[, thickness[, lineType[, shift]]]) -> img Draws a simple, thick, or filled up-right rectangle. Documentation

Color Space Conversions

Function	Notes
cv.cvtColor(src, code[, dst[, dstCn[, hint]]]) -> dst Converts an image from one color space to another. Documentation

Structural Analysis and Shape Descriptors

Function	Notes
cv.approxPolyDP(curve, epsilon, closed[, approxCurve]) -> approxCurve Approximates a polygonal curve(s) with the specified precision. Documentation
cv.approxPolyN(curve, nsides[, approxCurve[, epsilon_percentage[, ensure_convex]]]) -> approxCurve Approximates a polygon with a convex hull with a specified accuracy and number of sides. Documentation
cv.arcLength(curve, closed) -> retval Calculates a contour perimeter or a curve length. Documentation
cv.boundingRect(array) -> retval Calculates the up-right bounding rectangle of a point set or non-zero pixels of gray-scale image. Documentation
cv.boxPoints(box[, points]) -> points Finds the four vertices of a rotated rect. Useful to draw the rotated rectangle. Documentation
cv.connectedComponents(image[, labels[, connectivity[, ltype]]]) -> retval, labels computes the connected components labeled image of boolean image Documentation	ltype defaults to CV_16U instead of CV_32S due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.connectedComponentsWithStats(image[, labels[, stats[, centroids[, connectivity[, ltype]]]]]) -> retval, labels, stats, centroids computes the connected components labeled image of boolean image and also produces a statistics output for each label Documentation	labels, stats, and centroids are returned with dtype=np.float instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.contourArea(contour[, oriented]) -> retval Calculates a contour area. Documentation
cv.convexHull(points[, hull[, clockwise[, returnPoints]]]) -> hull Finds the convex hull of a point set. Documentation	hull is returned with dtype=np.float instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.convexityDefects(contour, convexhull[, convexityDefects]) -> convexityDefects Finds the convexity defects of a contour. Documentation	convexityDefects is returned with dtype=np.float instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.findContours(image, mode, method[, contours[, hierarchy[, offset]]]) -> contours, hierarchy Finds contours in a binary image. Documentation	contours and hierarchy are returned with dtype=np.float and dtype=np.int16 respectively instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.fitEllipse(points) -> retval Fits an ellipse around a set of 2D points. Documentation
cv.fitLine(points, distType, param, reps, aeps[, line]) -> line Fits a line to a 2D or 3D point set. Documentation
cv.isContourConvex(contour) -> retval Tests a contour convexity. Documentation
cv.matchShapes(contour1, contour2, method, parameter) -> retval Compares two shapes. Documentation
cv.minAreaRect(points) -> retval Finds a rotated rectangle of the minimum area enclosing the input 2D point set. Documentation
cv.minEnclosingCircle(points) -> center, radius Finds a circle of the minimum area enclosing a 2D point set. Documentation
cv.minEnclosingTriangle(points[, triangle]) -> retval, triangle Finds a triangle of minimum area enclosing a 2D point set and returns its area. Documentation
cv.moments(array[, binaryImage]) -> retval Calculates all of the moments up to the third order of a polygon or rasterized shape. Documentation
cv.pointPolygonTest(contour, pt, measureDist) -> retval Performs a point-in-contour test. Documentation

Feature Detection

Function	Notes
cv.Canny(image, threshold1, threshold2[, edges[, apertureSize[, L2gradient]]]) -> edges Finds edges in an image using the Canny algorithm. Documentation
cv.HoughCircles(image, method, dp, minDist[, circles[, param1[, param2[, minRadius[, maxRadius]]]]]) -> circles Finds circles in a grayscale image using the Hough transform. Documentation
cv.HoughCirclesWithAccumulator(image, method, dp, minDist[, circles[, param1[, param2[, minRadius[, maxRadius]]]]]) -> circles Finds circles in a grayscale image using the Hough transform and get accumulator. Documentation
cv.HoughLines(image, rho, theta, threshold[, lines[, srn[, stn[, min_theta[, max_theta[, use_edgeval]]]]]]) -> lines Finds lines in a binary image using the standard Hough transform. Documentation
cv.HoughLinesP(image, rho, theta, threshold[, lines[, minLineLength[, maxLineGap]]]) -> lines Finds line segments in a binary image using the probabilistic Hough transform. Documentation	lines is returned with dtype=np.float instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.HoughLinesWithAccumulator(image, rho, theta, threshold[, lines[, srn[, stn[, min_theta[, max_theta[, use_edgeval]]]]]]) -> lines Finds lines in a binary image using the standard Hough transform and get accumulator. Documentation

Object Detection

Function	Notes
cv.matchTemplate(image, templ, method[, result[, mask]]) -> result Compares a template against overlapped image regions. Documentation

imgcodecs

Function	Notes
cv.imread(filename[, flags]) -> retval Loads an image from a file. Documentation	filename can be anywhere in the full MicroPython filesystem, including SD cards if mounted. Only BMP and PNG formats are currently supported.
cv.imwrite(filename, img[, params]) -> retval Saves an image to a specified file. Documentation	filename can be anywhere in the full MicroPython filesystem, including SD cards if mounted. Only BMP and PNG formats are currently supported.

highgui

Function	Notes
cv.imshow(winname, mat) -> None Displays an image in the specified window. Documentation	winname must actually be a display driver object that implements an imshow() method that takes a NumPy array as input.
cv.waitKey([, delay]) -> retval Waits for a pressed key. Documentation	Input is taken from sys.stdin, which is typically the REPL.
cv.waitKeyEx([, delay]) -> retval Similar to waitKey, but returns full key code. Documentation	Input is taken from sys.stdin, which is typically the REPL. Full key code is implementation specific, so special key codes in MicroPython will not match other Python environments.

Building

Below are instructions to build the MicroPython-OpenCV firmware from scratch. Instructions are only provided for Linux systems.

1. Install dependencies
   • sudo apt install cmake python3 build-essential gcc-arm-none-eabi libnewlib-arm-none-eabi libstdc++-arm-none-eabi-newlib
2. Clone this repo
   • cd ~
   • git clone https://github.com/sparkfun/micropython-opencv.git
   • cd micropython-opencv
   • git submodule update --init
3. Build mpy-cross
   • make -C micropython/mpy-cross
4. Clone submodules for your board
   • make -C micropython/ports/rp2 BOARD=SPARKFUN_XRP_CONTROLLER submodules
   • Replace rp2 and SPARKFUN_XRP_CONTROLLER with your platform and board name respectively
5. Set environment variables (optional)
   • Some platforms require environment variables to be set. Examples:
   • export PICO_SDK_PATH=~/micropython-opencv/micropython/lib/pico-sdk
6. Build OpenCV
   • make -C src/opencv PLATFORM=rp2350 --no-print-directory -j4
   • Replace rp2350 with your board's platform
7. Build firmware
   • make BOARD=SPARKFUN_XRP_CONTROLLER -j4
   • Replace SPARKFUN_XRP_CONTROLLER with your board name
   • Your firmware file(s) will be located in ~/micropython-opencv/micropython/ports/<port-name>/build-<board-name>-OPENCV/

Adding New Board

Because OpenCV adds ~3MiB to the firmware size, it is necessary to define variants that reduce the storage size to avoid it overlapping with the firmware. It is also beneficial to adjust the board name to include OpenCV (or similar) to help customers and tech support identify whether the MicroPython-OpenCV is actually flashed to the board.

Below is the variant for the XRP Controller as an example. The variant is defined by creating a file called mpconfigvariant_OPENCV.cmake in micropython/ports/rp2/boards/SPARKFUN_XRP_CONTROLLER with contents:

list(APPEND MICROPY_DEF_BOARD
    # Board name
    "MICROPY_HW_BOARD_NAME=\"SparkFun XRP Controller (OpenCV)\""
    # 8MB (8 * 1024 * 1024)
    "MICROPY_HW_FLASH_STORAGE_BYTES=8388608"
)

Some board definitions do not have #ifndef wrappers in mpconfigboard.h for MICROPY_HW_BOARD_NAME and MICROPY_HW_FLASH_STORAGE_BYTES. That should be added if needed so the variant can build properly.

Then, the firmware can be built with make BOARD=<board-name> -j4

Adding New Platform

Only RP2350 exists currently, so the all requirements for adding new platforms is not fully known yet. However, it should be along the lines of:

1. Create a valid toolchain file for the platform
   • See rp2350.toolchain.cmake for reference
   • This loosely follow's OpenCV's platform definitions
2. Ensure OpenCV builds correctly
   • make -C src/opencv PLATFORM=<new-platform> --no-print-directory -j4
3. Create new board(s) for that platform
   • See #Adding New Board
4. Build MicroPython-OpenCV firmware for that board
   • make BOARD=<board-name> -j4

Contributing

Found a bug? Is there a discrepancy between standard OpenCV and MicroPython-OpenCV? Have a feature request? Want support for other hardware?

First, please see if there is an existing issue. If not, then please open a new issue so we can discuss the topic!

Pull requests are welcome! Please keep the scope of your pull request focused (make separate ones if needed), and keep file changes limited to the scope of your pull request.

Keep in mind that we only intend to support SparkFun products in this repository, though we may be open to hosting support for some hardware from other vendors. Please first open an issue to check if we're open to it. If not, you're always welcome to create your own fork following our license requirements!