Danny Griffin HTMAA 2023

Week 11 :: Interface and Application Programming

How do we incorporate interaction into video processing?

1. Write an application that interfaces a user with a device.

This week, I was aiming to learn some of the basics that I would need to set up an application to interface with my final project. Firstly, I need to find a way to use openCV libraries to process images from a camera. In Networking Week, I tried using Python with "urlib" to access video stream from the ESP-32CAM. While I found that it was possible, I'm not sure it's going to be fast enough for real-time video processing of multiple cameras. I'm testing openFrameworks because it seems to have promising user-friendly options for quickly working with openCV, and it's built specifically for real-time interactive applications.

First test, a refresher on project generation using openFrameworks:

Responsive image — Here, in main.cpp, the size of the application window is declared. A window will pop up with a resolution of 1280 x 768 pixels.

This week, I'm trying to use ofxOpenCv for motion tracking. The general process is to retrieve frames from the camera, convert to black and white, and then manipulate contrast to find the outline of certain desired objects or features.

For initial setup, I'm using the webcam on my laptop, which is identified by checking grabber.listDevices()

The update procedure, which processes every new frame from the camera.

In void ofApp::draw, we choose which processed images to display, and adjust graphics of the UI.

To build this demo app, I'm referring to bits of code from several Kyle McDonald examples.

After converting my webcam image into a threshold black/white image, then we can use contourFinder to get an outline based on contrast with the background.

To move slightly closer toward motion tracking, we can store values from previous frames. In this case, 100 contour polylines are displayed, with the oldest being erased each time a new one is generated.

A common process recommended in different examples is to save the background image - in this case, the blank wall and desk behind me. This improves recognition of figures within the frame.

One aspect of the contourFinder is the ability to select a color as threshold criteria. In this case I'm using my wax block from casting week, trying to detect the contour of the blue zone.

When the image threshold is callibrated, the contour is fairly accurate. Eventually, I hope to interpret the center of this zone as an (x,y) coordinate.

In this demo, I'm storing the contours and drawing them in an array on the screen. Movement in the tracked object can be seen in the variation across the array.

Demo screencapture.

This isn't much of a UI design, but the functionality is enough of a proof-of-concept for me to believe openFrameworks will work for my final project.
As I finalize my schematic design for the final weeks, I have a few challenges in mind for the interface development:
1. Connect to ESP-32CAM rather than webcam.
2. Connect to an output device.
3. Track the center of an object.