Final Project

Jamming with neural spikes!

Summary:

For the final project, I made an effect pedal for my bass guitar that gets triggered wirelessly by the neuroanl spike firing of my mouse buddy. This is my reimagination of jamming with the electrophysiology spike recordings of mice, which I can get my bass signal modified by the neuronal activity. The mouse, represented by a random spike generator in this case, fires spike with a probability of 8% to 100%, depending on how still the mouse is, which is detected by an IMU(inertial measurement unit). In addition, whenever it gets moved suddenly, it will be shocked and fire a burst of spikes for a second, triggering strong effect to the bass signal.

I am using Teensy 4.0 with its audio shield for processing the bass signal, and Byron has made an multiple effect pedal with Teensy here.This is a good high-level reference for getting idea of how the rough system looks like. I also consulted a lot on Teensy and its audio library’s documentation on PJRC’s site, which also included some helpful examples. Xiao’s documentation, especially for the ESP-NOW protocol is also very useful. ChatGPT also plays a huge role along the journey, as I often start with a template code from it and combine it with different examples from various places, and debug issues with it.

For this project, I designed and made all the custom PCB, the cell-shaped glass fiber lid for my effect processing unit, the plywood cell-shaped base that goes with it, and modified an existing 3D mouse model to create the mouse with a backpack design. Few major fabrication processes used in this final project included glass fiber composites making, plywood milling with ShopBot, 3D printing with Prusa MK4, and PCB milling with Carvera. For electronics, I used a Teensy 4.0 board with audio shield, two 1/4" mono audio jack, two Xiao ESP32C6 microcontrollers, one 3.7v LiPo battery, one MPU6050 IMU, one SSD1306 OLED, one switch, which mostly comes from the CBA inventory. All the electronic components cost around $60, with Teensy 4.0 being the most expensive component. The cost for other materials and processing are harder to estimate, but they might also be around $60. A more detailed documentation of the thought process, the struggles, and the excitement is written chronologically below.

1. First thought of the final project (around September)

As a direct result of my interest in neuroscience, music, and playing electric bass, there are something that I’ve really wanted to build for a while. As I often work with in-vivo electrophysiology recording in mice, I have spent a long time looking at single-neuron firing when mice are doing all kinds of behaviors. Staring at the spike trains of these neurons are pretty mesmerizing, as you can imagine these neurons are somewhere in the mouse brain and are actually firing action potential right now, and we are able to record them through an interface in its brain and see the spikes. The spike trains also seem pretty random at first, but when you look at it long enough there seems to be patterns and motifs. We sometimes even play the spikes as audio to monitor the changes in firing rate as our ears are way more sensitive to frequency changes than our eyes.

This inspired me to explore the integration of neuronal firing and music, on which I currently have two perspectives. One is to make an instrument that has the common interface of the standard electrophysiology recording devices that we use during research, which I can then connect it with the electrodes on mice and create live music with neuronal signals. A spike could trigger a variety of sounds or serve to modulate other sounds, which should be able to encapsulate and embody the patterns among seemingly random neuronal firing. It might be able to connect with an EEG headset that will incorporate your brain signal as well, which can feel like you are jamming with the cute mice through brain signals. Another line of thinking lies in creating an effect pedal for electric bass from the neuronal firing spikes, which may be pre-loaded by a large dataset of spike train patterns and can modulate your bass signals in different ways.

2. The spike generating source (Week 2)

For obvious reasons I won’t involve real mice with electrode implants in this project (which is definitely not in our CAC protocol), and thus I am making a random spike generator board as a digital mouse to jam with me. I made this goal the task for PCB design and manufaturing weeks, which can be seen at the week 2 assignment. (Week 2 PCB Design) It will basically be generating random spikes, and the probability of firing a spike is determined by the proximity sensor, which reflects the mouse getting nervous and firing spikes more frequently when you approach it. Later on this firing rate can be modulated by the frequenct of my bass signal, and my bass signal can simultaneously be modulated by the spikes, acting like triggers. I messed up the footprint of the LCD screen though, and will have to update that.

3. The spike generating source v2 (Week 7)

After the Input Device week, I learned how to measure capacitance to sense if my hands are close to the board, which seems way cooler than using a proximity sensor and I plan to incorporate that to my design. (Week 7 Input Device)

4. Made the spike generator & LCD display (Week 8)

In the Week 8 Output Device week, I successfully hooked a LCD to my board and displayed the generated spike train and the firing probability. For the final project, I’ll just have to make the firing probability variable take in the readings of the distance measuring capacitance.

5. A rough diagram for my system (right befor midterm review)

The whole design has been in my head for a while and I’ve dialed back many functions so that I have a higher chance of actually making something, following the spiral development method. The rough system diagram looks like this:

Tasks & Timeline

1. Proximity sensing input device manufacture
     -> designed & milled
     -> solder & test input readings by 11/24
2. Amplitude modulation circuit board design (most unsure, need help)
     -> discuss with TA by 11/20
     -> start design by 11/24
3. Amplitude modulation circuit board manufacture
     -> first mill by 11/27
4. A 3D print or casted silicon containing the spiking unit as an abstract mouse   -> can enclose electronics once proximity sensing & trigger output is working
     -> leaning towards casting with silicone for now
     -> design the model by 11/27
     -> make the mold by 12/04
5. Integration
     -> start testing everything together by 12/07

6. Update after Midterm review with Alfonso

After discussing the system with Alfonso, he suggested me to use Teensy 4.0 for the signal processing unit, as it is fast and has a intuitive audio effect library. I also decided that the spiking unit should send trigger time point wirelessly since it would be so much better with a control that you can hold in the hand and move around freely. Tying back to the networking week, I tested it with nRF24L01 and send IMU data sensed with the MPU6050. As shown in week 11 page, it worked out nicely.

7. Sending bursts of spike (12/3)

As I’m waiting for the Teensy 4.0 to arrive, I made a new, better integrated spiking unit with Xiao RP2040, MPU6050, nR24L01 and an OLED. The traces and milled board with soldered components look like this:

I find it satisfying how this utilized both sides and kept the footprint small. I then updated its code to read X,Y,Z acceleration, and set a threshold that when you accelerate the board over the threshold, it sends a burst of spikes to the receiver. The bursting duration is set from 0.5~2 sec depending on how much over the threshold the acceleration is. And it works! Very satisfying to see a steak of 1s bursting out when you shake the spiking unit. The current working code is shown below:

#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>
#include <Wire.h>
#include <MPU6050_light.h>

// CE and CSN pins for nRF24L01
#define CE_PIN D2
#define CSN_PIN D7

// Capacitive sensing setup (disabled for now)
// #define PROXIMITY_PIN D0 // Pin used for capacitance sensing

RF24 radio(CE_PIN, CSN_PIN); // Create an RF24 object
const byte address[6] = "00001"; // Address of the receiver

// Initialize MPU6050
MPU6050 mpu(Wire);
unsigned long timer = 0;

// Variables
float frProb=1.0/2.5;
float thr = 1; // The threshold for considering it to be an abrupt movement
float burst_dur_range[2] = {0.5,2};

void setup() {
  pinMode(LED_BUILTIN, OUTPUT); // Set the built-in LED pin as output, for monitoring transmission status
  // Start I2C communication
  Wire.begin();

  // Start serial monitor for testing, disabled when using power bank otherwise there will be error
  Serial.begin(115200);
  while (!Serial); // Wait for Serial Monitor
  Serial.println("Transmitter Starting...");

  if (!radio.begin()) {
    // Serial.println("nRF24L01 not connected!");
    while (1); // Halt
  }

  radio.openWritingPipe(address); // Set the address of the receiver
  radio.setPALevel(RF24_PA_HIGH); // Power Amplifier level
  radio.setChannel(108); // Channel (0-125)
  radio.setDataRate(RF24_1MBPS); // Data rate
  radio.stopListening(); // Set the module as transmitter

  byte status = mpu.begin();
  Serial.print(F("MPU6050 status: "));
  Serial.println(status);
  while(status!=0){ } // stop everything if could not connect to MPU6050
  
  // Serial.println(F("Calculating offsets, do not move MPU6050"));
  delay(1000);
  mpu.upsideDownMounting = true;
  mpu.calcOffsets(); // gyro and accelero
  float offset [3] = {mpu.getAccXoffset(), mpu.getAccYoffset() ,mpu.getAccZoffset()};
  Serial.println("Done\n");
  char buff[20];
  sprintf(buff,"%f, %f, %f", offset[0],offset[1],offset[2]); // the offset will be calculated correctly only when the board is flat -> accz is 1 when not moving as it's the ratio to g
  Serial.println(buff);
}

void loop() {
  mpu.update();
  float temp, value;
  float angle [3] = {mpu.getAngleX(), mpu.getAngleY() ,mpu.getAngleZ()};
  float acc [3] = {mpu.getAccX(), mpu.getAccY() ,mpu.getAccZ()-1};
  float burst_dur = 0;
  int counter = 0; 
	// Serial.print("X : ");
	// Serial.print(acc[0]);
	// Serial.print("\tY : ");
	// Serial.print(acc[1]);
	// Serial.print("\tZ : ");
  // Serial.println(acc[2]);

  // Send the message
  value = max(max(abs(acc[0]), abs(acc[1])), abs(acc[2]));
  Serial.println(value);
  if (value > thr) {
    burst_dur=burst_dur_range[0]+((value-thr)*(burst_dur_range[1]-burst_dur_range[0]))/(2.2-thr); // 2.2 is roughly the max acc you can achieve
    counter=burst_dur*1000; // in sec.
    while (counter>0){
      Serial.print(1);
      int spike=1;
      radio.write(&spike, sizeof(spike));
      counter--;
      delay(1);
    }
    int spike=burst_dur*1000;
    radio.write(&spike, sizeof(spike));
  }
  Serial.println(burst_dur);

  // bool success = radio.write(&value, sizeof(value));

  // if (success) {
  //   // Serial.println("Message sent successfully!");
  //   digitalWrite(LED_BUILTIN, HIGH); // Turn the LED on
  //   delay(100);
  //   digitalWrite(LED_BUILTIN, LOW);  // Turn the LED off
  //   delay(100);
  // } else {
  //   // Serial.println("Message failed to send!");
  // }

  delay(100); // Delay between messages
}

8. The Final Push (a week before demo)

Upgraded spike-generating board & reflection on PCB making

The previous version using Xiao RP2040 works well, but since I wanted to make it truely wireless, I decided to power it with a 3.7v LiPo battery instead of running a USB cord. For the built-in battery recharge circuitry, I decided to make an upgraded version with Xiao ESP32C6, which has generally the same footprint and I would only need minimal adjustment. The first iteration failed (~2 week before demo) as I tried to use a bulky, satisfying mechanical switch to turn on and off of the LiPo battery, which resulted in broken soldering joints and traces. The second iteration consists of a samll and light switch with pins which worked out very nicely, and I am pretty proud of how this board turned out.

This is definitely a big improvement from my first board of this class (which is also the first board ever for me) in all aspects, including the tight integration different components, the routing of traces, and the soldering. I really feel much more confident in dsigning PCB with KiCAD and milling it with Carvera and soldering components to it. It is truly a strong sense of empowerment, and I have felt that multiple times on different subjects throughout this class at various point of time, and this probably is when I felt most stronly about actually learned and “owning” this PCB design/fabrication skil. I was used to breadboarding, and at first I didn’t see how making PCB is a rapid prototyping process, as it seems to involve more steps between idea and reality. Now, however, I am wholeheartedly team PCB since it is so clean and elegant and the amount of dangling wires will not grow exponentially with more components involved. It is also actually very fast from designing to milling and soldering, and if it works it can directly be used in your project, unlike a working prototype with breadboard is all loose and chunky and struggle to take you further. I think I am most proud of myself for creating something that’s decently compact and elegant and would not otherwise be possible without a custom PCB.

For the beautiful OLED in place, I also updated my motion-sensing & spike-generating code. I am fetching the XYZ acceleration data & calculated tilt angle data from MPU6050_light library, and sending burst firing when detected sudden motion with an acceleration threshold. The calculated angle data somehow is always steadily and linearly decreasing even when placed absolutely still, but it still changes correctly when moved, so I calculated the variance of 5 angle data point and used it as a way to describe how stable the mouse is. I then mapped a firing probability of 8% to 100% to a variance from 0 to 100, so that when the variance is high, it generates spikes more often, triggering more effect. To visualize this, I also made an animation of a ring spreading outward whenever there’s a spike generated, and a flashing screen whenever a burst is triggered. The result turned out pretty cool, and the code for it is as follow, which involves substantial help from ChatGPT and Xiao’s documentation on ESP-NOW communication, which I will talk more about in the “Wireless Communication” section below.

// Spiking with OLED & ESP32C6
// v3, send data through ESPnow
// no different burst duration for this version, burst or not first
// 2024.12.15 update, all works, sending spike & burst through ESPnow, correct OLED display, variance-dependent firing rate

#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#include <SPI.h>
#include <MPU6050_light.h>
#include <Arduino.h>
#include "WiFi.h"
#include "esp_now.h" 

// OLED display dimensions
#define SCREEN_WIDTH 128
#define SCREEN_HEIGHT 64

// I2C address for the OLED
#define OLED_RESET -1
#define SCREEN_ADDRESS 0x3C

Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, OLED_RESET);

// Ring structure to hold ring properties
struct Ring {
  int radius;
  bool active;
  unsigned long startTime;
};

// Maximum number of rings
#define MAX_RINGS 20
Ring rings[MAX_RINGS];

// Center of the screen
const int centerX = SCREEN_WIDTH / 2;
const int centerY = SCREEN_HEIGHT / 2;

// Initialize MPU6050
MPU6050 mpu(Wire);
unsigned long timer = 0;

// ESPnow
#define ESPNOW_WIFI_CHANNEL 0
#define MAX_ESP_NOW_MAC_LEN 6
#define BAUD 115200
#define MAX_CHARACTERS_NUMBER 20
#define NO_PMK_KEY false

typedef uint8_t XIAO;
typedef int XIAO_status;
static uint8_t Receiver_XIAOC6_MAC_Address[MAX_ESP_NOW_MAC_LEN] = {0x54, 0x32, 0x04, 0x21, 0x60, 0xe4};

esp_now_peer_info_t peerInfo1;

typedef struct receiver_meesage_types{
  char Reveiver_device[MAX_CHARACTERS_NUMBER];
  char Reveiver_Trag[MAX_CHARACTERS_NUMBER];
}receiver_meesage_types;

receiver_meesage_types XIAOC6_RECEIVER_INFORATION;

typedef struct message_types{
  char device[MAX_CHARACTERS_NUMBER];
  char Trag[MAX_CHARACTERS_NUMBER];
}message_types;

message_types Personal_XIAOC6_Information;

void espnow_init();
void espnow_deinit();
void SenderXIAOS3_MACAddress_Requir();
void SenderXIAOS3_Send_Data();
// void SenderXIAOS3_Send_Data_cb(const XIAO *mac_addr,esp_now_send_status_t status);
void Association_ReceiverXIAOC6_peer();
// void ReceiverXIAOC6_Recive_Data_cb(const esp_now_recv_info *info, const uint8_t *incomingData, int len);

// Variables
float frProb = 10; // in %, baseline firing probability
float thr = 1;  // The threshold for considering it to be an abrupt movement
// float burst_dur_range[2] = { 0.5, 2 };
// bool burstActive = false;
// unsigned long burstStartTime = 0;
// unsigned long burstDuration = 0;
float stat_buff[5] = {0,0,0,0,0};
float variance = 0.0 ; 

void setup() {
  // Initialize Serial (optional for debugging)
  // Serial.begin(115200);

  // Initialize the OLED display
  if (!display.begin(SSD1306_SWITCHCAPVCC, 0x3C)) {
    // Serial.println("SSD1306 allocation failed");
    for (;;)
      ;
  }
  
  display.clearDisplay();
  display.display();

  // Initialize all rings as inactive
  for (int i = 0; i < MAX_RINGS; i++) {
    rings[i].active = false;
  }

  byte status = mpu.begin();
  // Serial.print(F("MPU6050 status: "));
  // Serial.println(status);
  while (status != 0) {}  // stop everything if could not connect to MPU6050

  delay(1000);
  mpu.upsideDownMounting = true;
  mpu.calcOffsets();  // gyro and accelero
  Serial.println("Done\n");

  randomSeed(analogRead(0)); // Seed the random generator
  
  SenderXIAOS3_MACAddress_Requir();
  SenderXIAOS3_MACAddress_Requir();
  espnow_init();

  // esp_now_register_send_cb(SenderXIAOS3_Send_Data_cb);

  Association_ReceiverXIAOC6_peer();
    
  // esp_now_register_recv_cb(ReceiverXIAOC6_Recive_Data_cb);
}

void loop() {
  display.clearDisplay();  // Clear the display at the start of each loop

  mpu.update();
  float acc[3] = {mpu.getAccX(), mpu.getAccY(), mpu.getAccZ() - 1};
  float maxAcc = max(max(abs(acc[0]), abs(acc[1])), abs(acc[2]));
  for (int i = 0; i < 5; i++) {
      stat_buff[i] = stat_buff[i + 1];
    }
  stat_buff[4]=mpu.getAccAngleX();
  variance=cal_Var(stat_buff,5);
  if(variance>100) {
    variance=100;
  }

  // Serial.println(variance);

  // Check for abrupt movement
  if (maxAcc > thr ) {
    Serial.println("Burst!");
    SenderXIAOS3_Send_Data("bursts");
    for(int i = 0; i<18; i++){
      display.fillScreen(SSD1306_WHITE);
      display.display(); 
      delay(25);  
        // Turn off the screen (clear display)
      display.fillScreen(SSD1306_BLACK);
      display.display();  // Update the display
      delay(25); 
    }
  }

  display.clearDisplay();

  // Update and draw all active rings
  for (int i = 0; i < MAX_RINGS; i++) {
    if (rings[i].active) {
      display.drawCircle(centerX, centerY, rings[i].radius, SSD1306_WHITE);
      rings[i].radius++; // Increase the radius to make the ring grow

      // Deactivate the ring if it exceeds the screen size
      if (rings[i].radius > max(SCREEN_WIDTH, SCREEN_HEIGHT)) {
        rings[i].active = false;
      }
    }
  }

  // Randomly activate a new ring
  frProb = map(variance, 0, 100, 8, 100);
  if (random(100) < frProb) { // Adjust probability for ring generation
    SenderXIAOS3_Send_Data("spike");
    // Serial.println("Sending 'spike' message");
    for (int i = 0; i < MAX_RINGS; i++) {
      if (!rings[i].active) {
        rings[i].radius = 1; // Start with the smallest radius
        rings[i].active = true;
        break;
      }
    }
  }

  display.display();
  // delay(1); // Adjust for speed of animation
}


// void SenderXIAOS3_Send_Data_cb(const XIAO *mac_addr,esp_now_send_status_t status){
//   char macStr[18];
//   Serial.print("Packet to: ");
//   snprintf(macStr, sizeof(macStr), "%02x:%02x:%02x:%02x:%02x:%02x",
//            mac_addr[0], mac_addr[1], mac_addr[2], mac_addr[3], mac_addr[4], mac_addr[5]);
//   Serial.println(macStr);
//   delay(500);
//   Serial.print(" send status:\t");
//   Serial.println(status == ESP_NOW_SEND_SUCCESS ? "Delivery Success" : "Delivery Fail");
//   Serial.println("");
// }

void Association_ReceiverXIAOC6_peer(){
  Serial.println("Attempting to associate peer for XIAOC6...");
  peerInfo1.channel = ESPNOW_WIFI_CHANNEL;
  peerInfo1.encrypt = NO_PMK_KEY;

  memcpy(peerInfo1.peer_addr, Receiver_XIAOC6_MAC_Address, 6);
  esp_err_t addPressStatus = esp_now_add_peer(&peerInfo1);
  if (addPressStatus != ESP_OK)
  {
    Serial.print("Failed to add peer");
    Serial.println(addPressStatus);
  }else
  {
    Serial.println("Successful to add peer");
  }
}

void SenderXIAOS3_Send_Data(const char* info){

  strcpy(Personal_XIAOC6_Information.device, "XIAOS3"); 
  strcpy(Personal_XIAOC6_Information.Trag, info); 

  esp_err_t XIAOS3_RECEIVER_INFORATION_data2 = esp_now_send(Receiver_XIAOC6_MAC_Address, (uint8_t *)&Personal_XIAOC6_Information, sizeof(message_types));

  // if (XIAOS3_RECEIVER_INFORATION_data2 == ESP_OK)
  // {
  //   Serial.println("Sent with success: XIAOS3_RECEIVER_INFORATION_data2");
  // }
  // delay(4000);
}

// void ReceiverXIAOC6_Recive_Data_cb(const esp_now_recv_info *info, const uint8_t *incomingData, int len) {
//   memcpy(&XIAOC6_RECEIVER_INFORATION, incomingData, sizeof(XIAOC6_RECEIVER_INFORATION));
//   Serial.print("Bytes received: ");
//   Serial.println(len);
//   Serial.print("Reveiver_device: ");
//   Serial.println(XIAOC6_RECEIVER_INFORATION.Reveiver_device);
//   Serial.print("Reveiver_Trag: ");
//   Serial.println(XIAOC6_RECEIVER_INFORATION.Reveiver_Trag);
//   Serial.println();
// }

void SenderXIAOS3_MACAddress_Requir(){
    WiFi.mode(WIFI_STA);
    WiFi.setChannel(ESPNOW_WIFI_CHANNEL);
    XIAO mac[MAX_ESP_NOW_MAC_LEN];
    while(!WiFi.STA.started()){
      Serial.print(".");
      delay(100);
    }
      WiFi.macAddress(mac);
      Serial.println();
      Serial.printf("const uint8_t mac_self[6] = {0x%02x, 0x%02x, 0x%02x, 0x%02x, 0x%02x, 0x%02x};", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
      Serial.println();
}

void espnow_init(){
  XIAO_status espnow_sign = esp_now_init();
  if(espnow_sign == ESP_OK)
  {
    Serial.println("the esp now is successful init!");
  }else
  {
    Serial.println("the esp now is failed init");
  }
}

void espnow_deinit(){
  XIAO_status espnow_sign = esp_now_deinit();
  if(espnow_sign == ESP_OK){
    Serial.println("the esp now is successful deinit!");
  }else
  {
    Serial.println("the esp now is failed deinit!");
  }
}

float cal_Var(float data[], int size) {
  if (size <= 1) {
    return 0.0; // Variance is undefined for size <= 1
  }

  // Calculate the mean
  float sum = 0.0;
  for (int i = 0; i < size; i++) {
    sum += data[i];
  }
  float mean = sum / size;

  // Calculate the variance
  float variance = 0.0;
  for (int i = 0; i < size; i++) {
    float diff = data[i] - mean;
    variance += diff * diff;
  }

  variance /= size; // Divide by N for population variance (use size - 1 for sample variance)

  return variance;
}

Glass Fiber Case

I chose to do composites for the wildcard week, which is technically the last topic before final project. As a result, I decided to make a casing with glass fiber with glass fiber, which is documneted in my week13 page. The result turned out nice, and now I have a cover for my effect processing unit.

Teensy 4.0 + Audio Shield (huge struggle #1)

At this point, however, I don’t have my processing unit itself ready yet. I got a used Teensy 4.0 a while ago from Quentin, but I thought it’s broken and was waiting for another order to arrive. It never arrived and I was running out of time, so I bothered Quentin a few more times and got myself a new Teensy 3.6. I would prefer not to switch to Teensy 3.6 though since I just got a reused Teensy audio shield from Tony, which will make my life so much easier, and it has a pin layout for 4.0. But I don’t have much choice as Teensy is not one of the very abundant microcontrollers. (none is very abundant in the last few days actually) I tested out 3.6, and it didn’t light up either, which made me question my cable, although I tested it and it charged my bike light with no problem. Since the cable looks shady, I changed to a cable for charging phones, and it lighted the board up. I guess this is one of the scenarios where the cable has to be a data transmission cable rather than just a power cable, but I thought this would not affet turning the on-board LED up. I then pluged in my supposedly broken 4.0 with this new cable, and it turned on! This spared me the hassle of an adapter PCB.

Since the audio shiled is a reused one and has gone through desoldering, there is solder in the pin holes and is preventing me from stacking all the pins directly on to Teensy 4.0 at the same time. As a result, I cut short, straight wires and put them in the pin holes one by one, and then solder them to the existing solder joints of Teensy. Around the same time, the 1/4" audio jack with small pins has arrived, and now I have all the components I need to be the audio processing unit.

At this point, I planned to input my bass signal through line-in on the board and output signal from line-out to the bass amp. However, there were multiple doubts which made me question about many things when it didn’t just work out initially. The long list of doubts included but not limited to:
      (1) not sure if it is fine to input/output mono signal as the pins are designed for stereo               input/output
      (2) not sure if the preamp on my active bass will output the signal at the right
            impedence and level for line-in, as it is not meant for raw signal from the pick-ups
      (3) not sure if the line-out of Teensy should be treated as an audio and go into the aux             jack on my bass amp or as normal input
      (4) not sure if the audio board is still functional after desoldering.
      (5) not sure what the audio quality and level I should be expecting from the line-out
      (6) not sure how straightforward the audio library is

I tested the setup first with the simplest code of passing line-in to line-out, and it did not play any sound other than a bunch of noise. I tried playing music with my computer and I could hear faint music coming out of the amp when I turned up the volume to the max, so I assumed the pre-amp of my active bass is not powerful enough, and Teensy is just somehow eating all my signal. I tried setting the audio board level and line-out level to the highest, but nothing significantly changed. This prompted me to make or get an addtional pre-amp, and during the recitation I asked about this issue, which Quentin pointed me toward the common differential amplifier circuit using op-amps. That should work as well but as I was running tight with time, I prefer borrowing a pre-amp first to get a working version. Thankfully, Hari has a guitar multiple effect pedal which works fine as a bass pre-amp as well. And now the signal is passing through and I got very excited, couldn’t wait to start testing out different effects. It was weird though because the effects didn’t seem to be that clear, and even just turning the signal on and off with a mixer or by adjusting volume didn’t do anything. I tested it with so much confusion and nothing was making sense. I switched between playing music from my computer and from bass pre-amp and sometimes the result is just inconsistent and it was a mess. At the end of the night, it didn’t play any sound at all and it was late and I disappointedly went to bed.

The next day things are still not playing, and as I picked up the board to examine, it suddenly bursted with loud and clear audio signal. I spent more time identifying the problem, and I figured that there’s a place around line-in, line-out and ground pins that I have to touch with my finger to make the signal pass. There were no loose solder joints and I’ve tested all the pin connections with a multimeter earlier, so I assume the ground is noisy and needs a resistor connected to it as I assume I’m a 100k ohm resistor. I cheked that all the grounds are indeed connected together and still soldered a 100k ohm resistor to the ground pins the next day. It seemed to work but then it wasnt. I tried more touching of the pins and initially nothing happend, but then after more touching and pressing a bit the signal passed through. I tried a bunch of different pressing methods and places and identified that I have to both press it tight enough and also with my bare hand to make it work, which is very confusing. In addition to all these pressing-pin frustration, there was also another weird thing at some point where the audio is passing through no matter what, as I initially thought it’s working and started to mess with different effects again but realize nothing is happening, and the audio even passes through with a blank file uploaded to Teensy. (I didn’t do all the documentation immediately and I have probably messed up a bit on the chronological order of when these bugs happened, but the message is nothing is consistently working, and the bugs are all over the place) At the end of that day, which was Friday night, I at least identified clearly the pressing-hard-with-bare-hand issue around certain pins and was able to replicate the issue multiple times, and I decided to ask Anthony for help, which was one of the best decision I’ve made.

It was my first time there at the EECS lab, and I was impressed by the clean and spacious electronics working area. I set up my testing circuit and waited in line for help, and made sure I could still replicate the issue. Anthony really is super knowledgeable and extremely patient, I don’t really understand how he managed to do that when literally dozens of students are just constantly asking him for help with all kinds of questions. We examined the board, tested with oscilloscope, tried many many things and spent around an hour on that board trying to make sense of the pin-pressing issue, as it didn’t make sense to Anthony either. After some more time, he pointed out that there are some nicks on the board and is pretty close to some traces, but seems to be fine. And there is also a missing capacitor on the audio shield, which might be the issue but it’s hard to see where it’s connecting to, and it might also be intentionally left unconnected. After failing to address the issue with other explanations, we went back to this missing capacitor and Anthonny saw that it seems to connect to the left line-in pin, which would explain a lot. Rather than adding a capacitor back, an easier test would just be modifying the code to use the intact right line-in channel. And it absoluted worked! I was so happy when the audio is just consistently passing through without my pressing or doing anything funky with it, and in the end, it is such a straightforward issue. It is a nasty one though, as some random capacitance would give inconsistent result during testing, and since it sometimes worked, as well as the sound-synthesis through line-out is always working, I didn’t consider that the board is faulty. Anyways, I was able to move forward and I felt so confident that I will make this project work, as the rest should be easy and straightforward. It was not.

Wireless Communication (slightly smaller struggle #2)

I have tested with nRF24L01 to communicate between boards in the networking week, and it was pretty simple. I though it would be easy to control effects on the Teensy with instructions sent by my spike-generating board through nRF24L01, but it didn’t just work. I suspect it is the clashing of Teensy’s audio library & RF24 library, or a SPI bus conflict. This doesn’t make much sense on paper though given that there are 3 SPI buses on Teensy, and the audio shield talks with Teensy 4.0 through SPI0, and I used SPI1 for nRF24L01. The hard fact is whenever I have the radio receiver activated, no signal will be passing through, and interestingly I can hear very faint random radio from my bass amp. As it is Sunday already and time is really ticking, I decided not to delve deeper to debug the issue, and instead avoid SPI and RF all at once. Trying to avoid any modifcation to the spike-generating board, I figure that since it is already using Xiao ESP32C6, I actually have many wireless transmission options built-in. During the quick search to decide whether to use bluetooth or wifi, I stumbled across the ESPnow protocol that allows simple wireless communication between ESP boards, and there’s a thorough documentation on the Xiao’s website, which is very helpful when I’m running low on time. My plan is thus soldering a Xiao ESP32C6 to my Teensy and treat it as a wireless communication module, which will then pass down information through UART. This sounded pretty straightforward and shouldn’t affect any of the existing audio processing codes. However, after connecting them correctly, I still can’t send any information from ESP32’s TX to Teensy’s RX. I thus went to the EECS lab for the second time, and debugged this with Anthony with an oscilloscope again. The ground is definitely common since Teensy is powering the ESP32 through VBUS & GND pin, and UART is simple enough that there shouldn’t be much issue. After trying some very explicit configuring of the serial port such as declaring the pins used even it’s used as a default pin, we figured out that with Xiao ESP32C6, you can not just use the casual “Serial1.begin(115200); “, but it has to be “Serial1.begin(115200,SERIAL_8N1,17,16); “. After fixing this problem, the two boards are talkiing happily now. Anthony to the rescue once again.

The final working code for the recevier Xiao ESP32C6 is as follow. The code is a collaboration with ChatGPT.

// 2024.12.15 worked with sender_helloworld
// - have to get rid of all serial monitor print for it to work with & get powered from UART_Teensy_helloworld (since there's no serial monitor connected)

#include<Arduino.h>
#include <WiFi.h>
#include <esp_now.h>
#include <HardwareSerial.h>

#define ESPNOW_WIFI_CHANNEL 0
#define MAX_ESP_NOW_MAC_LEN 6
#define BAUD 115200
#define MAX_CHARACTERS_NUMBER 20
#define NO_PMK_KEY false

typedef uint8_t XIAO;
typedef int status;

static uint8_t XIAOS3_Sender_MAC_Address[MAX_ESP_NOW_MAC_LEN] = {0x54, 0x32, 0x04, 0x21, 0x67, 0x3c};

esp_now_peer_info_t peerInfo_sender;

typedef struct receiver_meesage_types{
  char Reveiver_device[MAX_CHARACTERS_NUMBER];
  char Reveiver_Trag[MAX_CHARACTERS_NUMBER];
}receiver_meesage_types;

receiver_meesage_types XIAOC6_RECEIVER_INFORATION;

typedef struct message_types{
  char Sender_device[MAX_CHARACTERS_NUMBER];
  char Sender_Trag[MAX_CHARACTERS_NUMBER];
}message_types;

message_types XIAOS3_SENDER_INFORATION;

void Receiver_MACAddress_requir();
void espnow_init();
void espnow_deinit();
void ReceiverXIAOC6_Recive_Data_cb(const uint8_t * mac, const uint8_t *incomingData, int len);
void ReceiverXIAOC6_Send_Data();
void ReceiverXIAOC6_Send_Data_cb(const XIAO *mac_addr,esp_now_send_status_t status);
void Association_SenderXIAOS3_peer();

void setup() {
  // Serial.begin(BAUD);
  // while(!Serial);

  Serial1.begin(115200,SERIAL_8N1,17,16); 
  pinMode(LED_BUILTIN, OUTPUT);
  // Serial.println("Xiao ESP32C6: UART initialized");

  Receiver_MACAddress_requir();
  espnow_init();

  esp_now_register_recv_cb(ReceiverXIAOC6_Recive_Data_cb);

  esp_now_register_send_cb(ReceiverXIAOC6_Send_Data_cb);
  Association_SenderXIAOS3_peer();  
}

void loop() {
  // ReceiverXIAOC6_Send_Data();
  // Serial1.println("loop from ESP");

  // LED blinking as indicator
  digitalWrite(LED_BUILTIN, HIGH);  
  delay(500); // Send every second
  digitalWrite(LED_BUILTIN, LOW);   
  delay(500);   
  // delay(1000);
}

void espnow_init(){
  status espnow_sign = esp_now_init();
  if(espnow_sign == ESP_OK)
  {
    // Serial.println("the esp now is successful init!");
  }else
  {
    // Serial.println("the esp now is failed init");
  }
}

void espnow_deinit(){
  status espnow_sign = esp_now_deinit();
  if(espnow_sign == ESP_OK){
    // Serial.println("the esp now is successful deinit!");
  }else
  {
    // Serial.println("the esp now is failed deinit!");
  }
}

void Receiver_MACAddress_requir(){
    WiFi.mode(WIFI_STA);
    WiFi.setChannel(ESPNOW_WIFI_CHANNEL);
    XIAO mac[MAX_ESP_NOW_MAC_LEN];
    while(!WiFi.STA.started()){
      // Serial.print(".");
      delay(100);
    }
      WiFi.macAddress(mac);
      // Serial.println();
      // Serial.printf("const uint8_t mac_self[6] = {0x%02x, 0x%02x, 0x%02x, 0x%02x, 0x%02x, 0x%02x};", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
      // Serial.println();
}

void ReceiverXIAOC6_Recive_Data_cb(const esp_now_recv_info *info, const uint8_t *incomingData, int len) {
  memcpy(&XIAOS3_SENDER_INFORATION, incomingData, sizeof(XIAOS3_SENDER_INFORATION));
  // Serial.print("Bytes received: ");
  // Serial.println(len);
  // Serial.print("Sender_device: ");
  // Serial.println(XIAOS3_SENDER_INFORATION.Sender_device);
  // Serial.print("Sender_Trag: ");
  // Serial.println(XIAOS3_SENDER_INFORATION.Sender_Trag);
  // Serial.println();
  if (strcmp(XIAOS3_SENDER_INFORATION.Sender_Trag, "bursts") == 0) {
    // Action for "burst" message
    // Serial.println("Received 'bursts' message");
    Serial1.println("bursts");
   
  }
  else if (strcmp(XIAOS3_SENDER_INFORATION.Sender_Trag, "spike") == 0) {
    // Action for "spike" message
    // Serial.println("Received 'spike' message");
    Serial1.println("spike");
    
  }
}

void ReceiverXIAOC6_Send_Data_cb(const XIAO *mac_addr,esp_now_send_status_t status){
  char macStr[18];
  // Serial.print("Packet to: ");
  snprintf(macStr, sizeof(macStr), "%02x:%02x:%02x:%02x:%02x:%02x",
           mac_addr[0], mac_addr[1], mac_addr[2], mac_addr[3], mac_addr[4], mac_addr[5]);
  // Serial.println(macStr);
  delay(500);
  // Serial.print(" send status:\t");
  // Serial.println(status == ESP_NOW_SEND_SUCCESS ? "Delivery Success" : "Delivery Fail");
  // Serial.println("");
}

void ReceiverXIAOC6_Send_Data(){
  
  strcpy(XIAOC6_RECEIVER_INFORATION.Reveiver_device, "XIAOC6"); 
  strcpy(XIAOC6_RECEIVER_INFORATION.Reveiver_Trag, "yes"); 

  esp_err_t XIAOC6_RECEIVER_INFORATION_data1 = esp_now_send(XIAOS3_Sender_MAC_Address, (uint8_t *)&XIAOC6_RECEIVER_INFORATION, sizeof(receiver_meesage_types));

  if (XIAOC6_RECEIVER_INFORATION_data1 == ESP_OK)
  {
    // Serial.println("Sent with success: XIAOC6_RECEIVER_INFORATION_data1");
  }
  delay(4000);
}

void Association_SenderXIAOS3_peer(){
  // Serial.println("Attempting to associate peer for XIAOC6...");
  peerInfo_sender.channel = ESPNOW_WIFI_CHANNEL;
  peerInfo_sender.encrypt = NO_PMK_KEY;

  memcpy(peerInfo_sender.peer_addr, XIAOS3_Sender_MAC_Address, 6);
  esp_err_t addPressStatus = esp_now_add_peer(&peerInfo_sender);
  if (addPressStatus != ESP_OK)
  {
    // Serial.print("Failed to add peer");
    // Serial.println(addPressStatus);
  }else
  {
    // Serial.println("Successful to add peer");
  }
}

Teensy Audio Effect

After all that it’s finally time for playing with fun effects. The audio library design tool is very fun, and although I was a bit confused initially, confounded by the missing capacitor issue with my board, it is actually very intuitive and easy to use. I tried multiple effects to find a combination that will work well with the randomly triggered spike timing, and the initial idea of just toggling signal on and off is clean but boring. I settled down on a delay effect that plays back a delayed signal after 50 ms and 80 ms whenever a spike is triggered, and a rectifier effect with boosted volume to create a distored blast of sound whenever there’s a burst firing detected by sudden motion. Since I’ve seen many people praising the freeverb effect of Teensy’s audio library, I hooked it on at the end of my signal chain just before line-out, and it sounded awesome, with a bit of the eerie, metallic vibe. After getting the sound that I liked, it really did what I had anticipated it to do, which is to inspire me melodically and rhythmytically to play something for this effect pedal. And that’s when I know this is probably achieving the sound that I like.

The code for Teensy 4.0, receiving UART triggering signal from Xiao ESP32C6:

// Setting 3: like setting 1 but with amp on burst, even better
#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>

// GUItool: begin automatically generated code
AudioInputI2S            i2s1;           //xy=77,270
AudioEffectDelay         delay1;         //xy=253,348
AudioEffectRectifier     rectify1;       //xy=280,498
AudioAmplifier           amp1;           //xy=484,464
AudioMixer4              mixer1;         //xy=612,277
AudioEffectFreeverb      freeverb1;      //xy=613,178
AudioOutputI2S           i2s2;           //xy=729,128
AudioConnection          patchCord1(i2s1, 1, mixer1, 0);
AudioConnection          patchCord2(i2s1, 1, delay1, 0);
AudioConnection          patchCord3(i2s1, 1, rectify1, 0);
AudioConnection          patchCord4(delay1, 0, mixer1, 1);
AudioConnection          patchCord5(delay1, 1, mixer1, 2);
AudioConnection          patchCord6(rectify1, amp1);
AudioConnection          patchCord7(amp1, 0, mixer1, 3);
AudioConnection          patchCord8(mixer1, freeverb1);
AudioConnection          patchCord9(freeverb1, 0, i2s2, 0);
AudioControlSGTL5000     sgtl5000_1;     //xy=222,165
// GUItool: end automatically generated code

// GUItool: end automatically generated code

void setup() {
  // Enable the audio shield
  AudioMemory(50);
  SPI.setMOSI(11);
  SPI.setSCK(13);
  
  sgtl5000_1.enable();
  sgtl5000_1.volume(0.5); // Set initial volume to 50%
  
  // Set input and output levels
  sgtl5000_1.inputSelect(AUDIO_INPUT_LINEIN);
  sgtl5000_1.lineOutLevel(13); // Set line-out level
  

  delay1.delay(0, 50);
  delay1.delay(1, 80);

  mixer1.gain(0, 0.5);  // Start with full gain
  mixer1.gain(1, 0);
  // mixer1.gain(2, 0);
  mixer1.gain(3, 0);
  Serial.println("mixer");

  amp1.gain(3);
  freeverb1.roomsize(0.7); // 1 is fun as well
  delay(100); 
  
  // Serial.begin(115200); // For debugging
  Serial4.begin(115200);  // UART4 on Teensy

}

void loop() {
  if (Serial4.available()) {
    String received = Serial4.readStringUntil('\n');
    received.trim(); // Remove leading/trailing whitespace, \n, \r

    if (received == "bursts") {
        // Serial.println("Bursts from ESP");
        mixer1.gain(0, 0);  
        mixer1.gain(3, 1); 
        delay(1000);
        mixer1.gain(0, 0.5);
        mixer1.gain(3, 0); 
    }
    else if (received == "spike") {
        // Serial.println("Spike from ESP");
        mixer1.gain(0, 0.5);  
        mixer1.gain(1, 1); 
        mixer1.gain(2, 1);
        delay(100);
        mixer1.gain(0, 0.5);
        mixer1.gain(1, 0);
        mixer1.gain(2, 0);
    }
    Serial.println(received);
  }
  // Serial.print("Audio memory usage: ");
  // Serial.println(AudioMemoryUsageMax());
}

Assembly & Final Integration

With all the electronics done, and everything functionally working, I need to do the final assembly. I had integration in mind this whole time so the tasks now was not too demanding. I have designed my spiking board to be compact and with no loose wire, and I decided to make a simple case that will hold it in place, and the case will be like a backpack for a mouse. I got a free 3D mouse model here as an .obj file, imported to Fusion as a mesh and reduced mesh faces to 5%, then transformed it into a solid body so that I’m more familiar and is easier to work with. I then sticked a box directly on the back of the mouse and sent it to Prusa Mk4 to print it with black PLA. I also CAD a mold out of it as I planned to make a silicone rubber mouse instead of a basic 3D printed one, but decided to print both the mold and the mouse itself so that I have at least a backup. As shown above, the boards have been giving me troubles so I ended up using this backup 3D print itself, which actually still works pretty well in my opinion. It was initially all black, but after seeing how my effect processing board turned out, I decided to add some googly eyes to it so that the two of them comes from roughly the same universe. I rode my bike to the large Target beside Fenway at 11 pm on Monday, all for the googly eyes. At Target, I also stumbled across a cheap 10 dolloar fake leather wallet with a strap, which actually fits perfectly as a backpack strap for my mouse. I superglued them right before 1:30 pm the next day, and that for me is the cherry on top.

For the effect processing unit, I already have the top lid made with glass fiber, and all I need is a base to go with it. I figured that I’ll need to hide a power bank underneath the lid as well, so I decided a wooden base with some thickness might be a ncie choice. Plus, I really enjoyed using ShopBot milling out plywoods during the “make something big” week. So I offset the original outline of my cell, and designed grooves for the lid to fit in and a hollow center for placing powerbank. It’s a small part and the milling itself took around 30 minutes including tool change. The part turned out nice, but I milled the contour on the wrong direction for the groove, which made it slightly too big for my lid, so I just fixed it manually with a Dremel rotary tool. For the 1/4” jack to screw into the lid, I needed to drill two holes in the glass fiber, which I did it in the basement CBA shop with the strong vacuum turned on. I also spray painted a few layers of a frosted-glass textrue paint to give it a better finish. After all these, I screwed the jacks in place, resoldered a pair of wires for line-in to accomodate the new placement angle, and now it’s done as well. However, after screwing in the two 1/4” jacks, I couldn’t help but notice it very much is a smilely face, so I taped a tiny mouth for it and now it looks so cute. This also prompted me to grab some googly eyes for my mouse cause they will look silly and happy together.

Reflection:

[Update later]