Jamming with neural spikes!
1. First thought of the final project (around September)
As a direct result of my interest in neuroscience, music, and playing electric bass, there are something that I’ve really wanted to build for a while. As I often work with in-vivo electrophysiology recording in mice, I have spent a long time looking at single-neuron firing when mice are doing all kinds of behaviors. Staring at the spike trains of these neurons are pretty mesmerizing, as you can imagine these neurons are somewhere in the mouse brain and are actually firing action potential right now, and we are able to record them through an interface in its brain and see the spikes. The spike trains also seem pretty random at first, but when you look at it long enough there seems to be patterns and motifs. We sometimes even play the spikes as audio to monitor the changes in firing rate as our ears are way more sensitive to frequency changes than our eyes.
This inspired me to explore the integration of neuronal firing and music, on which I currently have two perspectives. One is to make an instrument that has the common interface of the standard electrophysiology recording devices that we use during research, which I can then connect it with the electrodes on mice and create live music with neuronal signals. A spike could trigger a variety of sounds or serve to modulate other sounds, which should be able to encapsulate and embody the patterns among seemingly random neuronal firing. It might be able to connect with an EEG headset that will incorporate your brain signal as well, which can feel like you are jamming with the cute mice through brain signals. Another line of thinking lies in creating an effect pedal for electric bass from the neuronal firing spikes, which may be pre-loaded by a large dataset of spike train patterns and can modulate your bass signals in different ways.
2. The spike generating source (Week 2)
For obvious reasons I won’t involve real mice with electrode implants in this project (which is definitely not in our CAC protocol), and thus I am making a random spike generator board as a digital mouse to jam with me. I made this goal the task for PCB design and manufaturing weeks, which can be seen at the week 2 assignment. (Week 2 PCB Design) It will basically be generating random spikes, and the probability of firing a spike is determined by the proximity sensor, which reflects the mouse getting nervous and firing spikes more frequently when you approach it. Later on this firing rate can be modulated by the frequenct of my bass signal, and my bass signal can simultaneously be modulated by the spikes, acting like triggers. I messed up the footprint of the LCD screen though, and will have to update that.
3. The spike generating source v2 (Week 7)
After the Input Device week, I learned how to measure capacitance to sense if my hands are close to the board, which seems way cooler than using a proximity sensor and I plan to incorporate that to my design. (Week 7 Input Device)
4. Made the spike generator & LCD display (Week 8)
In the Week 8 Output Device week, I successfully hooked a LCD to my board and displayed the generated spike train and the firing probability. For the final project, I’ll just have to make the firing probability variable take in the readings of the distance measuring capacitance.
5. A rough diagram for my system (right befor midterm review)
The whole design has been in my head for a while and I’ve dialed back many functions so that I have a higher chance of actually making something, following the spiral development method. The rough system diagram looks like this:
The rough diagram
Tasks & Timeline
- Proximity sensing input device manufacture
-> designed & milled
-> solder & test input readings by 11/24 - Amplitude modulation circuit board design (most unsure, need help)
-> discuss with TA by 11/20
-> start design by 11/24 - Amplitude modulation circuit board manufacture
-> first mill by 11/27 - A 3D print or casted silicon containing the spiking unit as an abstract mouse
-> can enclose electronics once proximity sensing & trigger output is working
-> leaning towards casting with silicone for now
-> design the model by 11/27
-> make the mold by 12/04 - Integration
-> start testing everything together by 12/07
6. Update after Midterm review with Alfonso
After discussing the system with Alfonso, he suggested me to use Teensy 4.0 for the signal processing unit, as it is fast and has a intuitive audio effect library. I also decided that the spiking unit should send trigger time point wirelessly since it would be so much better with a control that you can hold in the hand and move around freely. Tying back to the networking week, I tested it with nRF24L01 and send IMU data sensed with the MPU6050. As shown in week 11 page, it worked out nicely.
7. Sending bursts of spike (12/3)
As I’m waiting for the Teensy 4.0 to arrive, I made a new, better integrated spiking unit with Xiao RP2040, MPU6050, nR24L01 and an OLED. The traces and milled board with soldered components look like this:
The edge The trace The board The board from the side
I find it satisfying how this utilized both sides and kept the footprint small. I then updated its code to read X,Y,Z acceleration, and set a threshold that when you accelerate the board over the threshold, it sends a burst of spikes to the receiver. The bursting duration is set from 0.5~2 sec depending on how much over the threshold the acceleration is. And it works! Very satisfying to see a steak of 1s bursting out when you shake the spiking unit. The current working code is shown below:
#include <SPI.h>
#include <nRF24L01.h>
#include <RF24.h>
#include <Wire.h>
#include <MPU6050_light.h>
// CE and CSN pins for nRF24L01
#define CE_PIN D2
#define CSN_PIN D7
// Capacitive sensing setup (disabled for now)
// #define PROXIMITY_PIN D0 // Pin used for capacitance sensing
RF24 radio(CE_PIN, CSN_PIN); // Create an RF24 object
const byte address[6] = "00001"; // Address of the receiver
// Initialize MPU6050
MPU6050 mpu(Wire);
unsigned long timer = 0;
// Variables
float frProb=1.0/2.5;
float thr = 1; // The threshold for considering it to be an abrupt movement
float burst_dur_range[2] = {0.5,2};
void setup() {
pinMode(LED_BUILTIN, OUTPUT); // Set the built-in LED pin as output, for monitoring transmission status
// Start I2C communication
Wire.begin();
// Start serial monitor for testing, disabled when using power bank otherwise there will be error
Serial.begin(115200);
while (!Serial); // Wait for Serial Monitor
Serial.println("Transmitter Starting...");
if (!radio.begin()) {
// Serial.println("nRF24L01 not connected!");
while (1); // Halt
}
radio.openWritingPipe(address); // Set the address of the receiver
radio.setPALevel(RF24_PA_HIGH); // Power Amplifier level
radio.setChannel(108); // Channel (0-125)
radio.setDataRate(RF24_1MBPS); // Data rate
radio.stopListening(); // Set the module as transmitter
byte status = mpu.begin();
Serial.print(F("MPU6050 status: "));
Serial.println(status);
while(status!=0){ } // stop everything if could not connect to MPU6050
// Serial.println(F("Calculating offsets, do not move MPU6050"));
delay(1000);
mpu.upsideDownMounting = true;
mpu.calcOffsets(); // gyro and accelero
float offset [3] = {mpu.getAccXoffset(), mpu.getAccYoffset() ,mpu.getAccZoffset()};
Serial.println("Done\n");
char buff[20];
sprintf(buff,"%f, %f, %f", offset[0],offset[1],offset[2]); // the offset will be calculated correctly only when the board is flat -> accz is 1 when not moving as it's the ratio to g
Serial.println(buff);
}
void loop() {
mpu.update();
float temp, value;
float angle [3] = {mpu.getAngleX(), mpu.getAngleY() ,mpu.getAngleZ()};
float acc [3] = {mpu.getAccX(), mpu.getAccY() ,mpu.getAccZ()-1};
float burst_dur = 0;
int counter = 0;
// Serial.print("X : ");
// Serial.print(acc[0]);
// Serial.print("\tY : ");
// Serial.print(acc[1]);
// Serial.print("\tZ : ");
// Serial.println(acc[2]);
// Send the message
value = max(max(abs(acc[0]), abs(acc[1])), abs(acc[2]));
Serial.println(value);
if (value > thr) {
burst_dur=burst_dur_range[0]+((value-thr)*(burst_dur_range[1]-burst_dur_range[0]))/(2.2-thr); // 2.2 is roughly the max acc you can achieve
counter=burst_dur*1000; // in sec.
while (counter>0){
Serial.print(1);
int spike=1;
radio.write(&spike, sizeof(spike));
counter--;
delay(1);
}
int spike=burst_dur*1000;
radio.write(&spike, sizeof(spike));
}
Serial.println(burst_dur);
// bool success = radio.write(&value, sizeof(value));
// if (success) {
// // Serial.println("Message sent successfully!");
// digitalWrite(LED_BUILTIN, HIGH); // Turn the LED on
// delay(100);
// digitalWrite(LED_BUILTIN, LOW); // Turn the LED off
// delay(100);
// } else {
// // Serial.println("Message failed to send!");
// }
delay(100); // Delay between messages
}