MicroPython-OpenCV

Welcome to SparkFun's MicroPython port of OpenCV! This is the first known MicroPython port of OpenCV, and as such, there may be some rough edges. Hardware support is limited to SparkFun products.

Quick Start

1. Flash MicroPython-OpenCV firmware
   • Back up any files you want to keep, they will be overwritten!
   • Download the latest firmware from the Releases tab.
   • If you don't know how to flash firmware to your board, find your board here and follow the instructions using the OpenCV firmware.
2. Copy examples (optional)
   • It is suggested to copy the entire examples folder to your MicroPython board to get started. This can be done simply with mpremote:
     • cd micropython-opencv/examples
     • mpremote cp -r . :
3. Configure hardware drivers
   • The MicroPython port of OpenCV depends on hardware drivers to interface with cameras and displays. Drivers are built into the firmware, so there is no need to install them manually.
   • An example module called cv2_hardware_init is imported by all examples to initialize the drivers. You will likely need to edit the files for your specific hardware and board configuration.
4. Write OpenCV code!
   • Any IDE should work, so use your favorite!
   • The code block below contains snippets from various examples to highlight major features.

# Import OpenCV, just as you would in any other Python environment!
import cv2 as cv

# Standard OpenCV leverages the host operating system to access hardware, but we
# don't have that luxury in MicroPython. Instead, drivers are provided for
# various hardware components, which need to be initialized before using them.
# The exmples import a module called `cv2_hardware_init`, which initializes the
# drivers. You may need to edit the contents of the `cv2_hardware_init` module
# based on your specific board and hardware configuration
from cv2_hardware_init import *

# Import NumPy, almost like any other Python environment! The only difference is
# the addition of `from ulab` since MicroPython does not have a full NumPy
# implementation; ulab NumPy is a lightweight version of standard NumPy
from ulab import numpy as np

# Initialize an image (NumPy array) to be displayed, just like in any other
# Python environment! Here we create a 240x320 pixel image with 3 color channels
# (BGR order, like standard OpenCV) and a data type of `uint8` (you should
# always specify the data type, because NumPy defaults to `float`)
img = np.zeros((240, 320, 3), dtype=np.uint8)

# OpenCV's drawing functions can be used to modify the image. Here is the
# obligatory "Hello OpenCV!" text in red
img = cv2.putText(img, "Hello OpenCV!", (50, 200), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)

# Once we have an image ready to show, just call `cv.imshow()`, almost like any
# other Python environment! However, there is one important difference:
# 
# Standard OpenCV takes a window name string in `cv.imshow()`, which is used
# to display the image in a window. We don't have windows in MicroPython, so
# there is an API change where the first argument must be a display driver. Any
# display driver can be used, as long as it implements an `imshow()` method that
# takes a NumPy array as input
cv.imshow(display, img) # Can alternatively call `display.imshow(img)`

# Standard OpenCV requires a call to `cv.waitKey()` to process events and
# actually display the image. However the display driver shows the image
# immediately, so it's not necessary to call `cv.waitKey()` in MicroPython.
# But it is available, and behaves almost like any other Python environment! The
# only difference is that it requires a key to be pressed in the REPL instead of
# a window. It will wait for up to the specified number of milliseconds (0 for
# indefinite), and return the ASCII code of the key pressed (-1 if no key press)
# 
# Note - Some MicroPython IDEs (like Thonny) don't actually send any key presses
# until you hit Enter on your keyboard
key = cv.waitKey(0) # Not necessary to display image, can remove if desired

# Open a camera, similar to any other Python environment! In standard OpenCV,
# you would use `cv.VideoCapture(0)` or similar, and OpenCV would leverage the
# host operating system to open a camera object and return it as a
# `cv.VideoCapture` object. However, we don't have that luxury in MicroPython,
# so a camera driver is required instead. Any camera driver can be used, as long
# as it implements the same methods as the standard OpenCV `cv.VideoCapture`
# class, such as `open()`, `read()`, and `release()`
camera.open()

# Read a frame from the camera, just like any other Python environment! It
# returns a tuple, where the first element is a boolean indicating success,
# and the second element is the frame (NumPy array) read from the camera
success, frame = camera.read()

# Release the camera, just like in any other Python environment!
camera.release()

# Call `cv.imread()` to read an image from the MicroPython filesystem, just
# like in any other Python environment! Make sure to copy the image to the
# MicroPython filesystem first, and set the path to the image file as needed
# 
# If your board can mount an SD card, you can instead load the image to the SD
# card and change the path to point to the SD card
# 
# Note - only BMP and PNG formats are currently supported in MicroPython OpenCV
img = cv.imread("test_images/sparkfun_logo.png")

# Let's modify the image! Here we use `cv2.Canny()` to perform edge detection
# on the image, which is a common operation in computer vision
edges = cv2.Canny(img, 100, 200)

# Now we'll save the modified image to the MicroPython filesystem using
# `cv.imwrite()`, just like in any other Python environment!
# 
# Again, SD cards are supported, just change the path to point to the SD card
# 
# Note - only BMP and PNG formats are currently supported in MicroPython OpenCV
success = cv.imwrite("test_images/sparkfun_logo_edges.png", edges)

Hardware Support and Requirements

Hardware support in this repository is mostly limited to SparkFun products. The current list of supported proudcts is very small, but may be expanded in the future. Users are welcome to fork this repository to add support for other products, following our licence requirements. Assistance in adding support for other hardware will not be provided by SparkFun. We may consider pull requests that add support for additional hardware, see #Contributing.

The OpenCV firmware adds ~3MiB on top of the standard MicroPython firmware, which itself be up to 1MiB in size (depending on platform and board). So a board with at least 8MB of flash is recommended, to also have space available for file storage.

PSRAM is a requirement to do anything useful with OpenCV. A single 320x240 RGB888 frame buffer requires 225KiB of RAM; most processors only have a few hundred KiB of SRAM. Several frame buffers can be needed for even simple vision pipelines, so you really need at least a few MiB of RAM available. The more the merrier!

Below is the list of supported hardware devices:

• MicroPython Devices
  • XRP Controller
• Camera Drivers
  • HM01B0
  • OV5640 (not fully working yet)
• Display Drivers
  • ST7789
• Touch Screen Drivers
  • CST816

Performance

Limit your expectations. OpenCV typically runs on full desktop systems containing processors running at GHz speeds with dozens of cores optimized for computing speed. In contrast, microcontrollers processors typically run at a few hundred MHz 1 or 2 cores optimized for low power consumtion. Exact performance depends on many things, including the processor, vision pipeline, image resolution, colorspaces used, RAM available, etc. But for reference, the RP2350 can run the SparkFun Logo Detection Example at about 2.5 FPS at 320x240 resolution.

Something to consider is that MicroPython uses a garbage collector for memory management. As images are created and destroyed in a vision pipeline, RAM will be consumed until the garbage collector runs. The collection process takes longer with more RAM, so this can result in noticable delays during collection (typically a few hundred milliseconds). To mitigate this, it's best to pre-allocate arrays and utilize the optional dst argument of OpenCV functions to avoid allocating new arrays when possible. Pre-allocation also helps improve performance by avoiding repeated delays from allocating memory.

Another way to improve performance is to select the best hardware drivers for your setup. For example, the default SPI driver for the ST7789 is limited to the max SPI baudrate for the processor's SPI peripheral. That's 24MHz in the case of the RP2350, but another driver is provided that uses the PIO peripheral that runs at 75MHz, so displaying images can be ~3x faster (ignoring any required colorspace conversions).

For users wanting maximum performance, it may be desireable to bypass the high-level functions of the display/camera drivers, and instead work directly with the buffer member variables and read/write functions. This can avoid computationally expensive colorspace conversions when reading and writing images if they're not needed.

Included OpenCV Functions

Below is a list of all OpenCV functions included in the MicroPython port of OpenCV. This section follows OpenCV's module structure.

Only the most useful OpenCV functions are included. The MicroPython environment is extremely limited, so many functions are omitted due to prohibitively high RAM and firmware size requirements. Other less useful functions have been omitted to reduce firmware size. If there are additional functions you'd like to be included, see #Contributing.

If you need help understanding how to use these functions, see the documentation link for each function. You can also check out OpenCV's Python Tutorials and other tutorials online for more educational experience. This repository is simply a port of OpenCV, so we do not document these functions or how to use them, except for deviations from standard Python OpenCV.

core

Note

The core module includes many functions for basic operations on arrays. Most of these can be performed by numpy operations, so they have been omitted to reduce firmware size.

Operations on arrays

Function	Notes
cv.convertScaleAbs(src[, dst[, alpha[, beta]]]) -> dst Scales, calculates absolute values, and converts the result to 8-bit. Documentation
cv.inRange(src, lowerb, upperb[, dst]) -> dst Checks if array elements lie between the elements of two other arrays. Documentation
cv.minMaxLoc(src[, mask]) -> minVal, maxVal, minLoc, maxLoc Finds the global minimum and maximum in an array. Documentation

imgproc

Image Filtering

Function	Notes
cv.bilateralFilter(src, d, sigmaColor, sigmaSpace[, dst[, borderType]]) -> dst Applies the bilateral filter to an image. Documentation
cv.blur(src, ksize[, dst[, anchor[, borderType]]]) -> dst Blurs an image using the normalized box filter. Documentation
cv.boxFilter(src, ddepth, ksize[, dst[, anchor[, normalize[, borderType]]]]) -> dst Blurs an image using the box filter. Documentation
cv.dilate(src, kernel[, dst[, anchor[, iterations[, borderType[, borderValue]]]]]) -> dst Dilates an image by using a specific structuring element. Documentation
cv.erode(src, kernel[, dst[, anchor[, iterations[, borderType[, borderValue]]]]]) -> dst Erodes an image by using a specific structuring element. Documentation
cv.filter2D(src, ddepth, kernel[, dst[, anchor[, delta[, borderType]]]]) -> dst Convolves an image with the kernel. Documentation
cv.GaussianBlur(src, ksize, sigmaX[, dst[, sigmaY[, borderType[, hint]]]]) -> dst Blurs an image using a Gaussian filter. Documentation
cv.getStructuringElement(shape, ksize[, anchor]) -> retval Returns a structuring element of the specified size and shape for morphological operations. Documentation
cv.Laplacian(src, ddepth[, dst[, ksize[, scale[, delta[, borderType]]]]]) -> dst Calculates the Laplacian of an image. Documentation
cv.medianBlur(src, ksize[, dst]) -> dst Blurs an image using the median filter. Documentation
cv.morphologyEx(src, op, kernel[, dst[, anchor[, iterations[, borderType[, borderValue]]]]]) -> dst Performs advanced morphological transformations. Documentation
cv.Scharr(src, ddepth, dx, dy[, dst[, scale[, delta[, borderType]]]]) -> dst Calculates the first x- or y- image derivative using Scharr operator. Documentation
cv.Sobel(src, ddepth, dx, dy[, dst[, ksize[, scale[, delta[, borderType]]]]]) -> dst Calculates the first, second, third, or mixed image derivatives using an extended Sobel operator. Documentation
cv.spatialGradient(src[, dx[, dy[, ksize[, borderType]]]]) -> dx, dy Calculates the first order image derivative in both x and y using a Sobel operator. Documentation

Miscellaneous Image Transformations

Function	Notes
cv.adaptiveThreshold(src, maxValue, adaptiveMethod, thresholdType, blockSize, C[, dst]) -> dst Applies an adaptive threshold to an array. Documentation
cv.threshold(src, thresh, maxval, type[, dst]) -> retval, dst Applies a fixed-level threshold to each array element. Documentation

Drawing Functions

Function	Notes
cv.arrowedLine(img, pt1, pt2, color[, thickness[, line_type[, shift[, tipLength]]]]) -> img Draws an arrow segment pointing from the first point to the second one. Documentation
cv.circle(img, center, radius, color[, thickness[, lineType[, shift]]]) -> img Draws a circle. Documentation
cv.drawContours(image, contours, contourIdx, color[, thickness[, lineType[, hierarchy[, maxLevel[, offset]]]]]) -> image Draws contours outlines or filled contours. Documentation
cv.drawMarker(img, position, color[, markerType[, markerSize[, thickness[, line_type]]]]) -> img Draws a marker on a predefined position in an image. Documentation
cv.ellipse(img, center, axes, angle, startAngle, endAngle, color[, thickness[, lineType[, shift]]]) -> img Draws a simple or thick elliptic arc or fills an ellipse sector. Documentation
cv.fillConvexPoly(img, points, color[, lineType[, shift]]) -> img Fills a convex polygon. Documentation
cv.fillPoly(img, pts, color[, lineType[, shift[, offset]]]) -> img Fills the area bounded by one or more polygons. Documentation
cv.line(img, pt1, pt2, color[, thickness[, lineType[, shift]]]) -> img Draws a line segment connecting two points. Documentation
cv.putText(img, text, org, fontFace, fontScale, color[, thickness[, lineType[, bottomLeftOrigin]]]) -> img Draws a text string. Documentation
cv.rectangle(img, pt1, pt2, color[, thickness[, lineType[, shift]]]) -> img Draws a simple, thick, or filled up-right rectangle. Documentation

Color Space Conversions

Function	Notes
cv.cvtColor(src, code[, dst[, dstCn[, hint]]]) -> dst Converts an image from one color space to another. Documentation

Structural Analysis and Shape Descriptors

Function	Notes
cv.approxPolyDP(curve, epsilon, closed[, approxCurve]) -> approxCurve Approximates a polygonal curve(s) with the specified precision. Documentation
cv.approxPolyN(curve, nsides[, approxCurve[, epsilon_percentage[, ensure_convex]]]) -> approxCurve Approximates a polygon with a convex hull with a specified accuracy and number of sides. Documentation
cv.arcLength(curve, closed) -> retval Calculates a contour perimeter or a curve length. Documentation
cv.boundingRect(array) -> retval Calculates the up-right bounding rectangle of a point set or non-zero pixels of gray-scale image. Documentation
cv.boxPoints(box[, points]) -> points Finds the four vertices of a rotated rect. Useful to draw the rotated rectangle. Documentation
cv.connectedComponents(image[, labels[, connectivity[, ltype]]]) -> retval, labels computes the connected components labeled image of boolean image Documentation	ltype defaults to CV_16U instead of CV_32S due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.connectedComponentsWithStats(image[, labels[, stats[, centroids[, connectivity[, ltype]]]]]) -> retval, labels, stats, centroids computes the connected components labeled image of boolean image and also produces a statistics output for each label Documentation	labels, stats, and centroids are returned with dtype=np.float instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.contourArea(contour[, oriented]) -> retval Calculates a contour area. Documentation
cv.convexHull(points[, hull[, clockwise[, returnPoints]]]) -> hull Finds the convex hull of a point set. Documentation	hull is returned with dtype=np.float instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.convexityDefects(contour, convexhull[, convexityDefects]) -> convexityDefects Finds the convexity defects of a contour. Documentation	convexityDefects is returned with dtype=np.float instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.findContours(image, mode, method[, contours[, hierarchy[, offset]]]) -> contours, hierarchy Finds contours in a binary image. Documentation	contours and hierarchy are returned with dtype=np.float and dtype=np.int16 respectively instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.fitEllipse(points) -> retval Fits an ellipse around a set of 2D points. Documentation
cv.fitLine(points, distType, param, reps, aeps[, line]) -> line Fits a line to a 2D or 3D point set. Documentation
cv.isContourConvex(contour) -> retval Tests a contour convexity. Documentation
cv.matchShapes(contour1, contour2, method, parameter) -> retval Compares two shapes. Documentation
cv.minAreaRect(points) -> retval Finds a rotated rectangle of the minimum area enclosing the input 2D point set. Documentation
cv.minEnclosingCircle(points) -> center, radius Finds a circle of the minimum area enclosing a 2D point set. Documentation
cv.minEnclosingTriangle(points[, triangle]) -> retval, triangle Finds a triangle of minimum area enclosing a 2D point set and returns its area. Documentation
cv.moments(array[, binaryImage]) -> retval Calculates all of the moments up to the third order of a polygon or rasterized shape. Documentation
cv.pointPolygonTest(contour, pt, measureDist) -> retval Performs a point-in-contour test. Documentation

Feature Detection

Function	Notes
cv.Canny(image, threshold1, threshold2[, edges[, apertureSize[, L2gradient]]]) -> edges Finds edges in an image using the Canny algorithm. Documentation
cv.HoughCircles(image, method, dp, minDist[, circles[, param1[, param2[, minRadius[, maxRadius]]]]]) -> circles Finds circles in a grayscale image using the Hough transform. Documentation
cv.HoughCirclesWithAccumulator(image, method, dp, minDist[, circles[, param1[, param2[, minRadius[, maxRadius]]]]]) -> circles Finds circles in a grayscale image using the Hough transform and get accumulator. Documentation
cv.HoughLines(image, rho, theta, threshold[, lines[, srn[, stn[, min_theta[, max_theta[, use_edgeval]]]]]]) -> lines Finds lines in a binary image using the standard Hough transform. Documentation
cv.HoughLinesP(image, rho, theta, threshold[, lines[, minLineLength[, maxLineGap]]]) -> lines Finds line segments in a binary image using the probabilistic Hough transform. Documentation	lines is returned with dtype=np.float instead of np.int32 due to ulab not supporting 32-bit integers. See: v923z/micropython-ulab#719
cv.HoughLinesWithAccumulator(image, rho, theta, threshold[, lines[, srn[, stn[, min_theta[, max_theta[, use_edgeval]]]]]]) -> lines Finds lines in a binary image using the standard Hough transform and get accumulator. Documentation

Object Detection

Function	Notes
cv.matchTemplate(image, templ, method[, result[, mask]]) -> result Compares a template against overlapped image regions. Documentation

imgcodecs

Function	Notes
cv.imread(filename[, flags]) -> retval Loads an image from a file. Documentation	filename can be anywhere in the full MicroPython filesystem, including SD cards if mounted. Only BMP and PNG formats are currently supported.
cv.imwrite(filename, img[, params]) -> retval Saves an image to a specified file. Documentation	filename can be anywhere in the full MicroPython filesystem, including SD cards if mounted. Only BMP and PNG formats are currently supported.

highgui

Function	Notes
cv.imshow(winname, mat) -> None Displays an image in the specified window. Documentation	winname must actually be a display driver object that implements an imshow() method that takes a NumPy array as input.
cv.waitKey([, delay]) -> retval Waits for a pressed key. Documentation	Input is taken from sys.stdin, which is typically the REPL.
cv.waitKeyEx([, delay]) -> retval Similar to waitKey, but returns full key code. Documentation	Input is taken from sys.stdin, which is typically the REPL. Full key code is implementation specific, so special key codes in MicroPython will not match other Python environments.

Building

Below are instructions to build the MicroPython-OpenCV firmware from scratch. Instructions are only provided for Linux systems.

1. Install dependencies
   • sudo apt install cmake python3 build-essential gcc-arm-none-eabi libnewlib-arm-none-eabi libstdc++-arm-none-eabi-newlib
2. Clone this repo
   • cd ~
   • git clone https://github.com/sparkfun/micropython-opencv.git
   • cd micropython-opencv
   • git submodule update --init
3. Build mpy-cross
   • make -C micropython/mpy-cross
4. Clone submodules for your board
   • make -C micropython/ports/rp2 BOARD=SPARKFUN_XRP_CONTROLLER submodules
   • Replace rp2 and SPARKFUN_XRP_CONTROLLER with your platform and board name respectively
5. Set environment variables (optional)
   • Some platforms require environment variables to be set. Examples:
   • export PICO_SDK_PATH=~/micropython-opencv/micropython/lib/pico-sdk
6. Build OpenCV
   • make -C src/opencv PLATFORM=rp2350 --no-print-directory -j4
   • Replace rp2350 with your board's platform
7. Build firmware
   • make BOARD=SPARKFUN_XRP_CONTROLLER -j4
   • Replace SPARKFUN_XRP_CONTROLLER with your board name
   • Your firmware file(s) will be located in ~/micropython-opencv/micropython/ports/<port-name>/build-<board-name>-OPENCV/

Adding New Board

Because OpenCV adds ~3MiB to the firmware size, it is necessary to define variants that reduce the storage size to avoid it overlapping with the firmware. It is also beneficial to adjust the board name to include OpenCV (or similar) to help customers and tech support identify whether the MicroPython-OpenCV is actually flashed to the board.

Below is the variant for the XRP Controller as an example. The variant is defined by creating a file called mpconfigvariant_OPENCV.cmake in micropython/ports/rp2/boards/SPARKFUN_XRP_CONTROLLER with contents:

list(APPEND MICROPY_DEF_BOARD
    # Board name
    "MICROPY_HW_BOARD_NAME=\"SparkFun XRP Controller (OpenCV)\""
    # 8MB (8 * 1024 * 1024)
    "MICROPY_HW_FLASH_STORAGE_BYTES=8388608"
)

Some board definitions do not have #ifndef wrappers in mpconfigboard.h for MICROPY_HW_BOARD_NAME and MICROPY_HW_FLASH_STORAGE_BYTES. That should be added if needed so the variant can build properly.

Then, the firmware can be built with make BOARD=<board-name> -j4

Adding New Platform

Only RP2350 exists currently, so the all requirements for adding new platforms is not fully known yet. However, it should be along the lines of:

1. Create a valid toolchain file for the platform
   • See rp2350.toolchain.cmake for reference
   • This loosely follow's OpenCV's platform definitions
2. Ensure OpenCV builds correctly
   • make -C src/opencv PLATFORM=<new-platform> --no-print-directory -j4
3. Create new board(s) for that platform
   • See #Adding New Board
4. Build MicroPython-OpenCV firmware for that board
   • make BOARD=<board-name> -j4

Contributing

Found a bug? Is there a discrepancy between standard OpenCV and MicroPython-OpenCV? Have a feature request? Want support for other hardware?

First, please see if there is an existing issue. If not, then please open a new issue so we can discuss the topic!

Pull requests are welcome! Please keep the scope of your pull request focused (make separate ones if needed), and keep file changes limited to the scope of your pull request.

Keep in mind that we only intend to support SparkFun products in this repository, though we may be open to hosting support for some hardware from other vendors. Please first open an issue to check if we're open to it. If not, you're always welcome to create your own fork following our license requirements!