Computer Mouse as a Shoe
I started this week with the goal of making a shoe that could work as an infrared computer laser mouse. The idea is to always keep my hands on the keyboard while using my foot to control the mouse. This meant embedding an ESP32 (for Bluetooth functionality), an infrared optical sensor (like those found in gaming mice), and a haptic feedback driver and engine under the toes to simulate the click sensation—much like the "Taptic Engine" in Mac trackpads.
As I delved into this, I realized that most of the work involved interfacing the peripherals (infrared sensor, haptic engine) with the ESP32-S2 over I2C. However, the components weren't readily available in EDS, and Wokwi didn't support them either. While I considered simulating this with an LCD display over I2C, it felt like it would stray from the original intent of the project.
Piano Peripheral Tech Exploration
Instead, I decided to pivot to another concept I had in mind: creating a piano-playing automaton. This would be part of a broader set of modular piano peripherals, including:
- A piano-playing robot capable of interpreting MIDI files (or WAV files with FFT processing).
- An LED sheet that visualizes piano tutorials, akin to YouTube tutorials, guiding users through songs.
- An 88-laser system that projects lights through the piano and onto the ceiling, reacting to the music.
These modules would be independent, meaning you could use just the piano-playing automaton or combine all three for an enhanced experience. This idea was sparked by the untouched grand piano in my fraternity's library/party room, and I thought this could be a way to breathe new life into it.
First Steps: Piano-Playing Automaton
This week, I decided to focus on the first component: the piano-playing automaton. The mechanical design is still evolving, but my current idea involves using actuators or servos to map to the 88 keys of a piano. The device should be portable, with a mechanism to grip the piano's sides, and the actuators would be statically placed over the keys.
At EDS, JD shared a Mark Rober video showcasing a lot of features similar to what I wanted to implement, such as using FFT on audio files for piano automation.
For now, I chose to use LEDs to simulate the actuators in Wokwi. The ESP32-S2 has fewer GPIOs than needed for 88 LEDs, so I employed 74HC595 shift registers for multiplexing. With just three pins, I could control multiple LEDs by daisy-chaining the shift registers.
Implementation and Testing
Here's a video of my first test with 8 LEDs and a single shift register. The lights bounce left-to-right and back, then flash all at once:
The code used to drive that setup:
// GPIO pins to 74HC595
const int dataPin = 0; // DS
const int clockPin = 18; // SHCP
const int latchPin = 5; // STCP
void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
Serial.println("Hello, ESP32-S2!");
// initialize shift register control pins
pinMode(dataPin, OUTPUT);
pinMode(clockPin, OUTPUT);
pinMode(latchPin, OUTPUT);
// initialize shift register control pins to LOW
digitalWrite(dataPin, LOW);
digitalWrite(clockPin, LOW);
digitalWrite(latchPin, LOW);
}
void loop() {
// put your main code here, to run repeatedly:
for (int i = 0; i < 8; i++) {
shiftOutData(1 << i); // Light up one LED at a time
delay(200);
}
for (int i = 6; i >= 0; i--) {
shiftOutData(1 << i); // Move back the light
delay(200);
}
// Example Pattern: Light up all LEDs
shiftOutData(0xFF);
delay(1000);
// Example Pattern: Turn off all LEDs
shiftOutData(0x00);
delay(1000);
}
// Function to shift out data to 74HC595
void shiftOutData(byte data) {
// Begin by setting latch low to start sending data
digitalWrite(latchPin, LOW);
// Shift out the data byte, MSB first
for (int i = 7; i >= 0; i--) {
// Write bit by bit
digitalWrite(clockPin, LOW);
digitalWrite(dataPin, (data & (1 << i)) ? HIGH : LOW);
digitalWrite(clockPin, HIGH);
}
// After all bits are sent, set latch high to update the outputs
digitalWrite(latchPin, HIGH);
}
I then scaled up to 88 LEDs with 11 daisy-chained shift registers, resulting in a more complex but functional circuit:
And here's the code that controlled all 88 LEDs:
// Updated GPIO pins to 74HC595
const int dataPin = 19; // DS (Pin 14 of Shift Register 1)
const int clockPin = 18; // SH_CP (Clock Pin for all Shift Registers)
const int latchPin = 5; // ST_CP (Latch Pin for all Shift Registers)
const int numShiftRegisters = 11; // Total number of 74HC595 shift registers
// Array to hold the state of each shift register
byte shiftRegisterStates[numShiftRegisters] = {0};
void setup() {
// Initialize Serial Communication
Serial.begin(115200);
Serial.println("Hello, ESP32-S2!");
// Initialize Shift Register Control Pins
pinMode(dataPin, OUTPUT);
pinMode(clockPin, OUTPUT);
pinMode(latchPin, OUTPUT);
// Initialize control pins to LOW
digitalWrite(dataPin, LOW);
digitalWrite(clockPin, LOW);
digitalWrite(latchPin, LOW);
}
void loop() {
// Example Pattern: Shift a single LED across all 88 LEDs
// Shift Left
for (int i = 0; i < 88; i++) {
// Calculate which shift register and which bit
int srIndex = i / 8; // 0 to 10
int bitIndex = i % 8; // 0 to 7
// Clear previous state
memset(shiftRegisterStates, 0, sizeof(shiftRegisterStates));
// Set the current LED
if (srIndex < numShiftRegisters) {
shiftRegisterStates[srIndex] = (1 << bitIndex);
}
// Send the updated states to all shift registers
shiftOutMultiple(shiftRegisterStates, numShiftRegisters);
delay(100); // Adjust speed as needed
}
// Shift Right
for (int i = 86; i >= 0; i--) { // Start from second last to avoid out of range
int srIndex = i / 8;
int bitIndex = i % 8;
memset(shiftRegisterStates, 0, sizeof(shiftRegisterStates));
if (srIndex < numShiftRegisters) {
shiftRegisterStates[srIndex] = (1 << bitIndex);
}
shiftOutMultiple(shiftRegisterStates, numShiftRegisters);
delay(100);
}
// Example Pattern: Light up all LEDs
memset(shiftRegisterStates, 0xFF, sizeof(shiftRegisterStates));
shiftOutMultiple(shiftRegisterStates, numShiftRegisters);
delay(1000);
// Example Pattern: Turn off all LEDs
memset(shiftRegisterStates, 0x00, sizeof(shiftRegisterStates));
shiftOutMultiple(shiftRegisterStates, numShiftRegisters);
delay(1000);
}
// Function to shift out data to multiple 74HC595s
void shiftOutMultiple(byte data[], int numRegisters) {
digitalWrite(latchPin, LOW); // Begin data transmission
// Shift out all bytes from last to first shift register
for (int i = numRegisters - 1; i >= 0; i--) {
shiftOutData(data[i]);
}
digitalWrite(latchPin, HIGH); // Update outputs
}
// Function to shift out a single byte to 74HC595
void shiftOutData(byte data) {
// Shift out the data byte, MSB first
for (int i = 7; i >= 0; i--) {
// Write bit by bit
digitalWrite(clockPin, LOW);
digitalWrite(dataPin, (data & (1 << i)) ? HIGH : LOW);
digitalWrite(clockPin, HIGH);
}
}
Finally, I tested MIDI interaction by importing the "MIDI Library" in Wokwi and writing code to "play" Mozart’s "Twinkle, Twinkle, Little Star." Here’s the resulting video:
#include <MIDI.h>
// Updated GPIO pins to 74HC595
const int dataPin = 19; // DS (Pin 14 of Shift Register 1)
const int clockPin = 18; // SH_CP (Clock Pin for all Shift Registers)
const int latchPin = 5; // ST_CP (Latch Pin for all Shift Registers)
const int numShiftRegisters = 11; // Total number of 74HC595 shift registers
// Array to hold the state of each shift register
byte shiftRegisterStates[numShiftRegisters] = {0};
void setup() {
// Initialize Serial Communication
Serial.begin(115200);
Serial.println("Hello, ESP32-S2!");
// Initialize Shift Register Control Pins
pinMode(dataPin, OUTPUT);
pinMode(clockPin, OUTPUT);
pinMode(latchPin, OUTPUT);
// Initialize control pins to LOW
digitalWrite(dataPin, LOW);
digitalWrite(clockPin, LOW);
digitalWrite(latchPin, LOW);
}
void loop() {
playMozartMelody(); // Play the simulated Mozart melody
delay(5000); // Pause for 5 seconds before repeating
}
// Function to simulate a simple Mozart melody using MIDI Note On/Off events
void playMozartMelody() {
int melody[] = {60, 60, 67, 67, 69, 69, 67, // C4 C4 G4 G4 A4 A4 G4
65, 65, 64, 64, 62, 62, 60}; // F4 F4 E4 E4 D4 D4 C4
int noteDurations[] = {500, 500, 500, 500, 500, 500, 1000, // Durations for first phrase
500, 500, 500, 500, 500, 500, 2000}; // Durations for second phrase
int length = sizeof(melody) / sizeof(melody[0]);
for (int i = 0; i < length; i++) {
// Simulate Note On event
handleNoteOn(1, melody[i], 127); // Channel 1, velocity 127
delay(noteDurations[i]); // Hold the note for its duration
// Simulate Note Off event
handleNoteOff(1, melody[i], 0); // Channel 1, velocity 0
delay(100); // Short delay between notes
}
}
// Function to map MIDI note to LED index
int midiNoteToLED(int note) {
if (note < 21 || note > 108) return -1; // Out of range
return note - 21; // Maps MIDI note to LED index (0-87)
}
// Handle MIDI Note On Event
void handleNoteOn(byte channel, byte note, byte velocity) {
int ledNumber = midiNoteToLED(note);
if (ledNumber == -1) return; // Invalid note
// Calculate which shift register and which bit
int srIndex = ledNumber / 8; // 0 to 10
int bitIndex = ledNumber % 8; // 0 to 7
// Set the corresponding bit to turn on the LED
shiftRegisterStates[srIndex] |= (1 << bitIndex);
// Update the shift registers
shiftOutMultiple(shiftRegisterStates, numShiftRegisters);
}
// Handle MIDI Note Off Event
void handleNoteOff(byte channel, byte note, byte velocity) {
int ledNumber = midiNoteToLED(note);
if (ledNumber == -1) return; // Invalid note
// Calculate which shift register and which bit
int srIndex = ledNumber / 8; // 0 to 10
int bitIndex = ledNumber % 8; // 0 to 7
// Clear the corresponding bit to turn off the LED
shiftRegisterStates[srIndex] &= ~(1 << bitIndex);
// Update the shift registers
shiftOutMultiple(shiftRegisterStates, numShiftRegisters);
}
// Function to shift out data to multiple 74HC595s
void shiftOutMultiple(byte data[], int numRegisters) {
digitalWrite(latchPin, LOW); // Begin data transmission
// Shift out all bytes from last to first shift register
for (int i = numRegisters - 1; i >= 0; i--) {
shiftOutData(data[i]);
}
digitalWrite(latchPin, HIGH); // Update outputs
}
// Function to shift out a single byte to 74HC595
void shiftOutData(byte data) {
// Shift out the data byte, MSB first
for (int i = 7; i >= 0; i--) {
// Write bit by bit
digitalWrite(clockPin, LOW);
digitalWrite(dataPin, (data & (1 << i)) ? HIGH : LOW);
digitalWrite(clockPin, HIGH);
}
}
Check out the complete project here: Wokwi Project.
Special thanks to Anthony for helping me get an ESP32-S2!
Next Steps: Bluetooth MIDI Server Integration
As I move forward with this project, the goal is to replace the LED simulation with actual servos/actuators and build out a more comprehensive system that can handle real-time MIDI files cast over Bluetooth. This will involve setting up the ESP32-S2 as a Bluetooth MIDI server that can receive MIDI files sent from a web-based GUI or app. Here's the plan:
- Establish a Bluetooth connection between the web server (acting as a Bluetooth client) and the ESP32-S2 (acting as a Bluetooth MIDI server).
- Parse incoming MIDI messages and store them in a FIFO buffer on the ESP32-S2. This buffer will help manage the timing and execution of MIDI events in the order they were received.
- Drive the mechanical actuators using the parsed MIDI data to press the corresponding piano keys in real-time.
I could use the PCA9685 PWM driver board to control multiple servos, as it supports up to 16 channels and can be daisy-chained to cover all 88 keys. This board communicates over I2C, making it a suitable interface for the ESP32-S2, and it offers precise control over the actuators.
For the Bluetooth communication, the Web Bluetooth API will enable users to connect to the ESP32-S2 from any web browser, allowing for an intuitive way to cast MIDI files to the piano-playing automaton. This approach will make the system more flexible and user-friendly.
Reflection
This week was a solid step towards understanding the complexity and potential of this project. While I initially started with the idea of making a computer mouse shoe, pivoting to the piano-playing automaton allowed me to explore more intricate aspects of electronics, interfacing, and multiplexing. I learned how to efficiently use shift registers to expand the GPIO capabilities of the ESP32-S2, and the experience with Wokwi provided valuable insight into simulating a complex system.
Although there’s still a lot to tackle, particularly in integrating Bluetooth and building out the mechanical components, I’m excited about the potential this project holds. The ability to cast arbitrary MIDI files from a web server to an ESP32-S2 and have it play on a real piano will be a rewarding challenge, combining software, hardware, and music into one cohesive system.