Integration
Toward the end, we had the combine the pieces of code that people had written and make sure they were properly working with the machine.
Things were a little messy:
- Will and Sophie had written code to control the stepper motors in Modular Things
- The pump team had code for controlling the pump
- Catherine had written Python code to process the image and return a path for the machine to follow - but the paths weren't scaled to the size of the machine yet
- Haoheng had written a UI interface in Gradio to allow users to upload images or use the webcam
- Keonwook had designed a UI interface with Adobe
- Dünya was writing an API to connect the Python backend to Gradio and to Modular things
On top of that, there was no consolidation of code... code snippets were shared in WhatsApp and Google Drive (even as screenshots) and the gitlab was frequently being reorganized.
Also, it wasn't always communicated clearly what output was needed for the next person's input function. We would recommend clearly mapping out the data flow together in a grouped software meeting next time.
UI
Haoheng had written a beautiful interface with Gradio that performed the functions we wanted. However, Gradio is written in Python and thus couldn't run in Modular Things.
Dünya tried a couple of different approaches to making a bridge between Gradio, the image processing, and UI parts. That included writing a Flask API to return the image processing outputs to Gradio. But then the question remained how we'd get the image processing code to run in Modular Things.
We hacked a little around with Leo to build an alternative HTML webcam interface and functions. However, the Gradio interface already looked so nice and we had to connect it to the Python backend anyway, so se settled for a hacky trick (that had its own drawbacks):
Eventually, we managed to run Gradio in localhost and run it as an iframe from Modular Things. Haoheng redesigned it with some extra buttons we were hoping to access from Modular Things.
Honestly speaking, this was probably not the best approach simply because getting the data from Gradio to Modular Things was still a painful problem, and we couldn't find the DOM magic to make it work.
Websockets to transfer the data
So in order to make a process that could've been simple in JavaScript with a Flask API more challenging (because we like challenges and paining around with hacky things), Dünya tried to connect the Gradio interface to Modular Things via websockets (essentially connecting 3 clients through it...). She got it half working: while it transmits the output from the image processing function well, she did not get the Gradio app to trigger the "run" command in Modular Things.
So in order to have our app working, we need to have 3 servers running! Isn't that beautiful 🤯
Machine and Pump Control
In the meantime...
Putting things together and testing: Micheal wrote a mapping function to map the path output from the image processing script to the correct size of our machine and integrated the pump.
Now, it was time for testing!
Some learnings:
- The pump didn't always release a drop when it was meant to. Perhaps the viscocity of the dye (which for testing was just ink in water) was too high...
- Ink drops would be relatively large. Thus the drop size in the image processing script had to be quite large as well.
- It was pretty cool. Nothing is cooler than when things work. We celebrated our first smiley face.
