Week 3:

Embedded Programming

The EmoCase recognises facial emotions using machine learning and gives a visual indicator to people about the emotional state of the person it is tracking.

Process:

I looked at the data sheet for the Raspberry Pi Pico https://datasheets.raspberrypi.com/pico/pico-datasheet.pdf which is very human readable. A few cool things I found was that they included CAD files in the datasheet and a [fritzing file](https://datasheets.raspberrypi.com/pico/Pico-R3-Fritzing.fzpz) for circuits.

I then worked with the Wokwi tool to sketch out a basic circuit for a Raspberry Pi Pico. Wokwi is pretty cool as it let’s you know which pins on the Pi are suitable for the relevant pin on a input/output device.

Initially I wanted to explore an idea around plant sensing, so working with things like a humidity sensor. I saw that the Wokwi inputs included one however I couldn’t find much documentation for it working with the Raspberry Pi Pico in Wokwi. It took a lot of trial and error, with perfect code the serial monitor to actually show an output on humidity wouldn’t appear. Eventually found this example which showed that in fact we have to use Serial1 to get the serial monitor to function correctly.

I played around with a few more of the simulation tools and examples. Coincidentally, I'm also getting used to my new lab, and I also then realised that in my lab we have some electronics lying around so I had a rummage around. I couldn't find the sensors I was interested in but found some basic components like LEDs and so thought it might be fun to play with a real microcontroller and create a simple experiment, which eventually became the EmoCase.

For my final project I'm interested in how emotions and memories might be one day stored in a new type of 'personal cloud' powered by dense storage, and I'm intrigued by the new data classes that we might be able to store in that world. So I had the idea for a crude prototype of an input device that is constantly sensing my emotions and recording them, but also making them visible to others and myself.

I tested it out as Quentin had showed us during the group excercise, connecting to my computer while holding down Bootsel, the Pi shows up as a storage device on Mac.

After soldering the board, it wasn’t detected by my Mac anymore!

Debugging:

I thought I had probably made a bad connection as it was my first time soldering in quite a few years. I inspected the board and everything looked good, there was maybe one pin that seemed a bit blobby that could be refined.

• I consulted the internet and saw this thread: https://forums.raspberrypi.com/viewtopic.php?t=335911 which made me concerned that I had shorted my board!
• I then asked chatgpt to give me a debugging list to go through. Third on the list was: Check the BOOTSEL Mode. When I connected while holding down BOOTSEL, it worked again. (This should cause the Pico to mount as a USB mass storage device named “RPI-RP2).

This solved my issue and the board was detected. Next issue I ran into was that my board was no longer being detected in the Arduino IDE. I tried:

• Check all ports and board connection in the IDE (was OK) but not detecting the board.
• Checked the Pico was present in Mac Finder (it was).
• Could run a ‘blink’ program from the Raspberry Pi website (drag and drop into Finder —> Raspi-Pico).
• Couldn’t find a solution, so restarted Arduino IDE (didn’t work).
• Disconnected / reconnected Pico (didn’t work).
• Restarted Mac and restarted IDE —> initially worked.
• However, the Pico board USB connection still wasn’t reliably being detected. So, I uninstalled the Pico library, reinstalled it, and started everything again.
• I could get blink to run, but it was using the wrong port: /dev/cu.bluetooth-incoming-port. This meant that, yes, I could run a program, but I could not communicate with the board after running the program as the serial connection was not established properly.
• Then I noticed the IDE was ‘indexing’ in the bottom left corner. Once that was done, miraculously, the Port connection appeared! Not convinced I changed anything, but it seemed to be correct.

To test, I ran a small program to blink the onboard LED and stop when ‘s’ is typed on my keyboard. This would help me to see if the serial connection was well established after running the program on the board.

Developing EmoCase

Next, I wanted to try out a simple experiment to build the actual EmoCase idea out. The general premise was quite simple: Turn on a red LED for sad, green for happy, yellow for neutral.

• To prototype this, a few steps:
• First, make it work with simple keystrokes.
• Find a way to track emotions quickly (to test) as an input and stream to the output.

Firstly to make it work with keystrokes, I wrote some quick code based off the traffic lights example, and added keystrokes to be sent via the serial monitor on the Arduino IDE

          
          // Define LED pins for traffic light
          #define RED_LED_PIN 15     // Red LED connected to GP15
          #define YELLOW_LED_PIN 14  // Yellow LED connected to GP14
          #define GREEN_LED_PIN 9    // Green LED connected to GP9

          // Time variables for alternating tasks
          unsigned long previousMillis = 0;
          const long interval = 2000; // 2 seconds interval

          // Flag to switch between tasks
          bool readFromPython = true;

          void setup() {
            // Initialize the LED pins as outputs
            pinMode(RED_LED_PIN, OUTPUT);
            pinMode(YELLOW_LED_PIN, OUTPUT);
            pinMode(GREEN_LED_PIN, OUTPUT);
            
            // Start Serial communication at 9600 baud rate for Serial Monitor
            Serial.begin(9600);
            while (!Serial) {
              ; // Wait until the Serial Monitor is ready
            }

            // Inform user about the controls
            Serial.println("Press 'S' for Sad (Red), 'H' for Happy (Green), 'N' for Neutral (Yellow).");
            delay(500);

            // Turn off all LEDs initially
            turnOffAllLEDs();
          }

          void loop() {
            // Get the current time
            unsigned long currentMillis = millis();

            // Check if it's time to switch tasks
            if (currentMillis - previousMillis >= interval) {
              // Save the last time you switched tasks
              previousMillis = currentMillis;
              
              // Toggle the flag to switch between reading from Python and Serial Monitor
              readFromPython = !readFromPython;

              // Indicate which task is active
              if (readFromPython) {
                Serial.println("Listening for data from Python script...");
              } else {
                Serial.println("Listening for commands from Serial Monitor...");
              }
            }

            // Task 1: Read from Python script (Laptop sends commands)
            if (readFromPython && Serial.available() > 0) {
              char input = Serial.read(); // Read the character from Serial port (Python script input)
              input = tolower(input);

              // Handle input to control the LEDs
              switch (input) {
                case 's': // Sad
                  setLEDState(RED_LED_PIN);
                  Serial.println("Sad (Red LED)");
                  break;
                case 'h': // Happy
                  setLEDState(GREEN_LED_PIN);
                  Serial.println("Happy (Green LED)");
                  break;
                case 'n': // Neutral
                  setLEDState(YELLOW_LED_PIN);
                  Serial.println("Neutral (Yellow LED)");
                  break;
                case 'o':
                  turnOffAllLEDs(); 
                  default:
                  Serial.println("Invalid input. Use 's', 'h', or 'n'.");
                  break;
              }
            }

            // Task 2: Read from Serial Monitor (Manual testing or other commands)
            if (!readFromPython && Serial.available() > 0) {
              char input = Serial.read(); // Read the character from Serial Monitor
              Serial.print("Serial Monitor input received: ");
              Serial.println(input);
              
              // Optional: Handle specific commands from the Serial Monitor if needed
            }
          }

          // Function to turn off all LEDs
          void turnOffAllLEDs() {
            digitalWrite(RED_LED_PIN, LOW);
            digitalWrite(YELLOW_LED_PIN, LOW);
            digitalWrite(GREEN_LED_PIN, LOW);
          }

          // Function to set the state of LEDs
          void setLEDState(int activePin) {
            turnOffAllLEDs();
            digitalWrite(activePin, HIGH);
          }