HTMAA 2024
Sungmoon Lim

week 13


interface and application programming

The assignment for this week was to write an application that interfaces a user with an input and/or output device that you made.

I wanted to keep things fairly simple this week as I'm traveling for a conference and thus am away from the lab. I actually had no idea what I wanted to do for this assignment and spent a bunch of time tinkering with the different types of tools (specifically graphics and interfaces) provided as examples on the HTMAA website. Unfortunately, that means I just spent several hours tinkering instead of making any real progress or decisions. Normally, this is not an issue (and possibly a good thing), but it wasn't great timing as I'd be away the entire week.

In the end, I decided to use the existing webcam on my Mac to program a motion detector and some way of visualizing said motion. This isn't particularly relevant to my final project (read: it is not relevant at all), but I figured using the built-in webcam would simplify things with limited resources while still satisfying the input requirement of this assignment, per the inputs page.

As for the output for video and audio inputs, we were provided lots of different types of examples: SDL, Pygame, PySDL2, openFrameworks, ofpython, TouchDesigner, SuperCollider, among others. I decided to go with Pygame.

I first started off with something simple in VS Code: I wanted to create an interface that would mirror the motion detected from the camera.

The idea was that we get webcam input (the motion data) and then process the input in real-time using OpenCV. The output would be an interactive visualization, using Pygame for graphics display, that displays real-time feedback with visual elements (for now, geometric shapes), text updates, and screen division.

I've copied my code below:


        import cv2
        import numpy as np
        import pygame
        import sys
        from pygame import gfxdraw  
        
        class GestureController:
            def __init__(self):
                # 1. Initialize webcam
                self.cap = cv2.VideoCapture(0)  # Use default webcam (usually built-in)
                if not self.cap.isOpened():
                    print("Error: Could not open webcam")
                    sys.exit()
        
                # Get webcam frame size
                _, frame = self.cap.read()
                self.height, self.width = frame.shape[:2]
        
                # 2. Initialize Pygame for visualization
                pygame.init()
                self.screen = pygame.display.set_mode((self.width, self.height))
                pygame.display.set_caption("Motion Detector")
        
                # 3. Setup variables for motion detection
                self.prev_frame = None
                self.motion_threshold = 30  # Adjust this value based on your needs
                
                # Colors
                self.BLACK = (0, 0, 0)
                self.WHITE = (255, 255, 255)
                self.RED = (255, 0, 0)
                self.GREEN = (0, 255, 0)
                self.BLUE = (0, 0, 255)
        
            def process_frame(self):
                # 1. Capture frame
                ret, frame = self.cap.read()
                if not ret:
                    return None, None
        
                # 2. Mirror frame (so movements are more intuitive)
                frame = cv2.flip(frame, 1)
        
                # 3. Convert to grayscale
                gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
                
                # 4. Blur the image to reduce noise
                gray = cv2.GaussianBlur(gray, (21, 21), 0)
        
                # Initialize motion direction
                motion_direction = "none"
        
                # 5. Compare with previous frame
                if self.prev_frame is not None:
                    # Calculate difference between frames
                    frame_diff = cv2.absdiff(self.prev_frame, gray)
                    
                    # Apply threshold to get regions of significant movement
                    thresh = cv2.threshold(frame_diff, self.motion_threshold, 255, cv2.THRESH_BINARY)[1]
        
                    # Divide frame into left and right halves
                    left_sum = np.sum(thresh[:, :self.width//2])
                    right_sum = np.sum(thresh[:, self.width//2:])
                    
                    # Determine direction based on which half has more motion
                    if left_sum > 10000 and left_sum > right_sum * 1.5:  # Adjust these values as needed
                        motion_direction = "left"
                    elif right_sum > 10000 and right_sum > left_sum * 1.5:
                        motion_direction = "right"
        
                # 6. Store current frame for next iteration
                self.prev_frame = gray
                
                return frame, motion_direction
        
            def draw_visualization(self, motion_direction):
                # 1. Clear screen
                self.screen.fill(self.BLACK)
                
                # 2. Draw dividing line
                pygame.draw.line(self.screen, self.WHITE, 
                                (self.width//2, 0), 
                                (self.width//2, self.height), 
                                2)
        
                # 3. Draw circles based on motion
                if motion_direction == "left":
                    # Draw filled circle on left side
                    gfxdraw.filled_circle(self.screen, 
                                        self.width//4, self.height//2, 
                                        50, self.GREEN)
                elif motion_direction == "right":
                    # Draw filled circle on right side
                    gfxdraw.filled_circle(self.screen, 
                                        3*self.width//4, self.height//2, 
                                        50, self.GREEN)
        
                # 4. Add text showing current direction
                font = pygame.font.Font(None, 36)
                text = font.render(f"Motion: {motion_direction}", True, self.WHITE)
                self.screen.blit(text, (10, 10))
        
                # 5. Update display
                pygame.display.flip()
        
            def run(self):
                try:
                    while True:
                        # 1. Check for quit event
                        for event in pygame.event.get():
                            if event.type == pygame.QUIT:
                                return
                            if event.type == pygame.KEYDOWN:
                                if event.key == pygame.K_q:
                                    return
        
                        # 2. Process frame and get motion direction
                        frame, motion_direction = self.process_frame()
                        if frame is None:
                            break
        
                        # 3. Update visualization
                        self.draw_visualization(motion_direction)
        
                        # 4. Control frame rate
                        pygame.time.delay(30)
        
                    # Clean up
                    self.cap.release()
                    pygame.quit()
        
        if __name__ == "__main__":
            controller = GestureController()
            controller.run()

The motion detection was actually much more sensitive than I'd expected! (Please excuse the extremely blurry photos and the gross computer screen...)

vinyl vinyl

Here's a video of it in motion (ft the unmistakable hotel carpet in the background):

If I have time later, I'd like to add more to the visualization portion of this assignment: I think 'drawing' something virtually via motioning into a camera would be cool, or simply adding different types of colors and shapes depending on the type or velocity of the motion could also work to make the output more interesting.