Week 12: Machine Design

This week was a fun one, despite the fact that I had to stray away from my musical theme (spoiler alert: or almost). This week's assignment required us to, as a team, design a machine that could be built from the modular-things framework. After a big brainstorming session, my group settled on designing a multi-camera trailing device to shoot very cool and dramatic scenes. The expected outcome was this kind of multi-camera shot:

The project was challenging and required us to properly scope out the different subcomponents that had to individually work for a potentially integration to see the day, and Alan was particularly fit for this role (well done!) We split the work for multiple groups to focus on various tasks, and defined a few spirals that we needed to gradually complete to obtain a final product that would be satisfying. This work is explained on our group's machine documentation, so I will not be writing those explanations again here. Instead, I will focus on the various roles I had within the team, and the subcomponents that I worked on.

My first involvement was to help build the first few rails using the Modular Things kit. It was fun to build something using only the Fusion360 file as a reference, because it felt a bit like reverse engineering. Eventually, we managed to get the ball rolling and quickly managed to get two rails done. Then, my work was to focus on implementing motion tracking in our system. We initially started working with ESP32-C3s but I realized that running computer vision processes onto the microcontrollers would be too much of a heavy load, so we pivoted towards using Raspberry Pis with Pi Cameras. I set up the Pi system and ran some OpenCV scripts to have a basic feature of motion tracking deployed onto it.

However, something came in the way: We were not able to use Raspberry Pis inside of the Modular Things framework since only Arduino was supported (at the time at least). After a few hours of trial and error, we decided to give up on the motion tracking and focus more on the post-processing of the videos.

One thing that I then started to work on was to write an algorithm that would collect the video feeds from all cameras and make a video edit that would be synchronized to a given piece of music. To do so, I used the librosa Python library which allowed me to compute the energy novelty curve of a given piece of music and perform automatic beat tracking. I then took the assumption that a downbeat was happening every four beats, and I used the detected tempo to find the timestamps of each downbeat and make the edit. With the right piece of music, the result was that every video would become much more dramatic, which was the expected effect!

In order to collect the videos of all Pis, I had to setup a network between each Pi, which individually ran a webserver to serve the last recorded video. After a few attempts at sharing the WiFi network amongst all Pis, we realized that the system was unreliable and made every debugging attempt a hassle (see attached)

We therefore opted for a wired network, with a central switch that would allow all Pis, as well as the computer that would do the automatic edit, to be connected. It ended up being a bunch of wires, but it was a fun process to go through. There was quite some DevOps to do, and I had to design a few systemd jobs on the Pis to automatically setup static IP on the LAN, configure the camera, and start the webserver. Eventually, despite a tangle of wires, we managed to get a working system:

We finally did a few tests, and managed to get a system that was good enough for all of us to celebrate with a little dance:

Finally, it was a very fun project that I am very grateful for since it allowed me to spend a few late nights with my cohort hacking a fun system together and having some fun! The system is not the most stable and the cameras tend to quickly move away from the object of focus, which limits the abilities of the system, but it is a good first prototype! Ideally, we'd implement some motion tracking and make this system completely autonomous and usable!

Notes to a future me (learning outcomes):

Raspberry Pis are a great midway between low compute power microncontrollers and laptops.
WiFi can be unreliable, and makes components harder to debug.
Working with friends is fun, especially during late nights.