HTMAA 2025 BLog

Week 14: Wildcard for Wildcut

Sun Chuanqi — Sun, 07 Dec 2025 00:00:00 GMT

In the wild card week, we get to choose from a selection of topics to dive deep with one of our TAs.

I wanted to celebrate everything we learned about time management from this class by engraving the cover of The Dark Side of The Moon, the album that includes "Time".

And then one day you find ten years have got behind you No one told you when to run, you missed the starting gun

— Pink Floyd, "Time"

Asset Preparation

Jiaming is our great TA who helped organize lab sessions and hands-on practice. We used the XTool F2 Ultra 60W MOPA Laser Cutter for engraving and the xTool MetalFab Laser Welder/CNC Cutter for cutting 3mm stainless steel.

I prepared 3 different designs to sample a variety of capabilities of the laser cutter.

Design	Focus
Dark Side of The Moon	Multi-color marking
Spectrogram	Raster image resolution
Nameplate	Typography details

Dark Side of The Moon

Iconic album cover of Pink Floyd's "The Dark Side of The Moon" (1973)

I learned that Master Oscillator Power Amplifier (MOPA) lasers can modulate the temperature at which metal oxidizes, producing different colors on stainless steel, aluminum, and titanium. I started with the original album cover and traced it with SVG.

In preparation for the lab session, I separated the traces by process:

Single line for incoming light and the prism outline
Gradient fill for the triangle inside the prism
Solid fill for the line beams exiting the prism

Decomposing the design by process

It turned out I over-prepared. Jiaming showed me how to separate elements with the XTool software. We just needed to select each part and apply different parameters.

In an earlier session, Edward had worked with Jiaming to characterize the color output as a function of power, speed, and pulse frequency. All I needed was choosing from their color palette, mainly in the bottom righ corner below:

Picking color from other people's characterization

After the first cut, the line stroke was not coming out. We updated the parameters and the rest of the engraving went smoothly.

Acceptable result after 2nd attempt

My design file messed up the gradient part inside the prism. It came out as a black triangle, losing the original gradient effect. The colors were limited but at least we had a rainbow-like pattern.

Macro lens close-up of the engraving

Under the macro lens, I saw strong aliasing patterns in the rainbow beams. I wonder if there is a way to avoid quantized laser movement.

Spectrogram of Neil's Saying

I wanted to test if we can store sound as an image engraved on metal.

I had separately vibe-coded a program for conversion between audio and spectrogram. The program was all AI generated with 50+ rounds of revision. Since I don't have the full history of the AI coding session, I won't claim credit for the code. For this project, I'm only using the tool to generate a spectrogram from sound and to verify the sound from the engraved spectrogram.

Luckily, we had gathered lots of quotes from Neil during the Machine Building Week. Let's pick a hot take!

Original quote from Neil

Original spectrogram generated from audio

Engraving didn't go well. All the gray levels were lost, and worse still, some of the darkest areas were inverted to light color.

Engraved spectrogram with unwanted artifacts

Upon a closer look, the engraving cut too deep. We should have used color marking process instead.

Close-up inspection revealed surface damage

I took a photo of the engraved metal, processed it in Figma, and decoded it into sound. The restoration steps:

Desaturate
Invert
Color burn (HEX #666666)
Reduce exposure

Manual restoration process

To make this project more interesting, I also used a large-language model with vision skills to restore the spectrogram. The AI restoration is similar to the manual process except I prompted AI to make most edits. AI was able to visually restore the formants lines, which in theory, contribute to a more natural utterance.

AI restoration process (chat log)

I uploaded both reconstructions to my custom spectrogram player and compared the results:

Original

Manual reconstruction from photo of engraving

AI reconstruction from photo of engraving

Neither reconstruction is very good. Given more time, I would attempt 2.5D engraving to capture the spectrogram in 3D and use computer vision to restore the depth. But, wait, did I just re-invent CDs?

Nameplate for Final Project

I wanted to design a nameplate for my final project. The 3mm thickness of the stainless steel was too thick for this purpose, but I proceeded anyway just to test how much detail we can get with typography.

Nameplate design with layers separated by process

We failed in creating small texts using SVG exported from Figma. The program tried to trace the outline of the text instead of engraving the internal strokes. We switched to XTool's built-in font and the same issue persisted. In addition, the Direct Current Symbol was too small for our process.

Had we had more time, I would play with different SVG font options until the machine can recognize the internal strokes.

Engraved nameplate

Zooming in, I saw crisp texts. But the MIT logo suffered the same aliasing issue as before.

MIT logo was aliased

Post-processing

Jiaming showed me how to cut the metal with the MetalFab laser cutter. It was the same process we practiced in Machine Cutting Week except for the use of a more powerful laser and the addition of air blasting for material removal.

The violent process of cutting metal with laser

After cutting, the molten metal edges need to be sanded down for safety and aesthetics. Jiaming showed me how to use the belt sander to remove the edges.

Before and after sanding, showing bottom side

Following Jiaming's demo, I sanded out the corners. The final results look great!

Final results of all 3 designs

Shout out to Jiaming for hands-on coaching, and Edward for sharing his characterization results.

Reflection

Color engraving is very trial-and-error based. It takes 2 hours just to map out a few possible color options. The parameters must be determined for the specific material and machine, and results may still vary.

I only had access to yellow, blue, purple colors on stainless steel. It would be vastly more interesting if we can access green and red colors as well.

Engraving raster image was very hard. I wish to have a process that can generate more gray levels. At the very minimum, I need to prevent inversions where darker designs become lighter due to over-burning.

AI was able to reconstruct somewhat intelligible spectrogram from a photo. This could become a new research topic or a multi-sensor AI benchmark.

Appendix

Week 13: An Audio Adventure

Sun Chuanqi — Wed, 03 Dec 2025 00:00:00 GMT

This week, I set out to bring my walkie-talkie device closer to an interactive demo. The goal was to write an application that interfaces a user with the input and output devices I made throughout the semester: the Operator (ESP32 with probe and buttons) and the Switchboard (ESP32 with TRRS sockets and LEDs). The application would orchestrate state management while providing a web UI for device management and debugging.

I grew tired of AI-generated frontend code with its Tailwind classes, React boilerplate, blue-purple gradients, oversized typography, and distracting animations. I decided to build as close to the web platform as possible: vanilla TypeScript, RxJS for reactive state, and lit-html for templating.

Reviving the Foundation

I started by reviving the walkie-talkie code from week 9. The immediate challenge was IP discovery. Both devices had hardcoded addresses that broke whenever the network changed.

Discovering ESP32 Address

The laptop could discover the ESP32's IP by inspecting rinfo.address from incoming UDP packets:

import * as dgram from "dgram";

const udpReceiver = dgram.createSocket("udp4");

udpReceiver.bind(8888);

udpReceiver.on("message", (msg, rinfo) => {
  // rinfo contains sender information
  const senderIp = rinfo.address; // Get sender's IP address
  const senderPort = rinfo.port; // Get sender's port

  console.log(`Received from ${senderIp}:${senderPort}`);
  console.log(`Data: ${msg.toString()}`);
});

Discovering Laptop Address

The reverse problem was harder. The ESP32 needed to know the laptop's IP address. I used the Node.js os.networkInterfaces() API to expose the laptop's address through an HTTP endpoint:

Server

import * as os from "os";
import express from "express";

const app = express();

app.get("/api/origin", (req, res) => {
  const interfaces = os.networkInterfaces();
  const ipv4 = Object.values(interfaces)
    .flat()
    .find((addr) => addr.family === "IPv4" && !addr.internal);

  res.json({ host: ipv4?.address || "localhost" });
});

app.listen(3000);

Web UI

const ipInput = document.getElementById("ipInput") as HTMLInputElement;
const fetchButton = document.getElementById("fetchButton") as HTMLButtonElement;

fetchButton.addEventListener("click", async () => {
  try {
    const response = await fetch("http://localhost:3000/api/origin");
    const data = await response.json();
    ipInput.value = data.host;
  } catch (error) {
    console.error("Failed to fetch origin:", error);
    ipInput.value = "Error fetching origin";
  }
});

The web UI allowed users to fetch the laptop IP. We still need to push it to the ESP32 somehow.

Initial UI for manually reading IP address

Automated Handshake Protocol

I borrowed the BLE networking code from the Networking week and enhanced it with an automated handshake protocol for the laptop and the ESP32 to exchange IP addresses that can be used for full duplex UDP streaming. As soon as ESP32 is paired over BLE, the following sequence occurs:

Web requests server address from Node.js
Server responds with its own IP
Web sends server address to Operator via BLE
Operator responds with its own IP
Web registers Operator address with server

web

// Fetch server IP on page load
const response = await fetch("http://localhost:3000/api/origin");
const data = await response.json();
serverAddressSpan.textContent = data.host;

// After Operator BLE connects, send server address
const message = `server:${url.hostname}:${url.port}`;
sendMessage(message);

// Receive operator address from device
if (message.startsWith("operator:")) {
  const address = message.substring(9);
  operatorAddressSpan.textContent = address;
  fetch(`http://localhost:3000/api/locate-operator?address=${encodeURIComponent(address)}`, {
    method: "POST",
  });
}

operator

// Receive server address from web
void handleRxMessage(String msg) {
  if (msg.startsWith("server:")) {
    String serverIp = msg.substring(7, msg.lastIndexOf(':'));
    int port = msg.substring(msg.lastIndexOf(':') + 1).toInt();
    // Store and use for UDP
    setupUDP(serverIp.c_str(), port);
  }
}

// Send operator address to web for registration
void sendOperatorAddress() {
  String myIP = WiFi.localIP().toString();
  sendBLE("operator:" + myIP);
}

On the UI, I displayed both addresses for verification. In addition, the probe state is streamed over BLE and shown in real-time.

Automated handshake and BLE data streaming

Adding the Switchboard UI

I added a Switchboard UI for connection testing, again, reusing the BLE communication patterns from the Networking week.

Switchboard connection panel

Real-time Server Push with SSE

I wanted the web UI to receive live updates from the server without polling. I added a Server-Sent Events (SSE) endpoint to push data to the browser with minimal latency:

// Server: SSE endpoint for pushing events to web
if (req.url === "/api/events") {
  res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-cache" });
  sseClients.push({ res });
  req.on("close", () => {
    sseClients = sseClients.filter((c) => c.res !== res);
  });
}

// Server: /api/speak endpoint - triggers speech synthesis
else if (req.url === "/api/speak") {
  const { text, voice } = JSON.parse(body);
  await synthesizeAndStreamSpeech(text, voice);
  emitServerEvent(text); // Push to SSE clients
}

// Web: Listen to SSE stream
const eventSource = new EventSource("http://localhost:3000/api/events");
eventSource.onmessage = (event) => {
  logDiv.textContent += `[${timestamp}] SSE: ${event.data}\n`;
};

// Web: Trigger speak from UI
speakBtn.addEventListener("click", async () => {
  const text = speakTextarea.value.trim();
  const response = await fetch("http://localhost:3000/api/speak", {
    method: "POST",
    body: JSON.stringify({ text }),
  });
});

The web UI could now trigger speech synthesis and receive real-time updates regarding any speech events from the server.

Speech synthesis controls with live event log

Taming Complexity

As features accumulated, the codebase became unwieldy. BLE communication lived in the web UI, but UDP communication lived in the Server.

Entangled architecture with BLE in web UI and UDP in server

I refactored nearly all the code to move BLE communication into the server, creating a modular architecture. Normally I would do this with AI. But the stake is too high: I'm running out of time for this week's project, and I knew I need a solid foundation to carry me through the final project. Reliability and maintainability became my priority, which meant no AI coding for this refactor.

Simplified architecture after refactoring

In the simplified architecture, I heavily relied on RxJS to create reactive data streams.

import { map, tap } from "rxjs";
import { HTTP_PORT, LAPTOP_UDP_RX_PORT } from "./config";
import { BLEDevice, opMac, swMac } from "./features/ble";
import { createButtonStateMachine } from "./features/buttons";
import { geminiResponse$, geminiTranscript$, handleConnectGemini, handleDisconnectGemini } from "./features/gemini-live";
import { createHttpServer } from "./features/http";
import {
  handleButtonsMessage,
  handleConnectOperator,
  handleDisconnectOperator,
  handleOpAddressMessage,
  handleProbeMessage,
  handleRequestOperatorAddress,
  logOperatorMessage,
  operatorAddress$,
  operatorButtons$,
  operatorProbeNum$,
} from "./features/operator";
import { silence$ } from "./features/silence-detection";
import { handleAudio, handleConnectSession, handleDisconnectSession, interrupt, triggerResponse } from "./features/simulation";
import { broadcast, handleSSE, newSseClient$ } from "./features/sse";
import { appState$, updateState } from "./features/state";
import { handleBlinkLED, handleConnectSwitchboard, handleDisconnectSwitchboard } from "./features/switchboard";
import { createUDPServer } from "./features/udp";

async function main() {
  const operator = new BLEDevice(opMac);
  const switchboard = new BLEDevice(swMac);

  createUDPServer([handleAudio()], LAPTOP_UDP_RX_PORT);

  createHttpServer(
    [
      handleSSE(),
      handleBlinkLED(switchboard),
      handleConnectSwitchboard(switchboard),
      handleDisconnectSwitchboard(switchboard),
      handleConnectOperator(operator),
      handleDisconnectOperator(operator),
      handleRequestOperatorAddress(operator),
      handleConnectSession(),
      handleDisconnectSession(),
      handleConnectGemini(),
      handleDisconnectGemini(),
    ],
    HTTP_PORT
  );

  appState$
    .pipe(
      map((state) => ({ state })),
      tap(broadcast)
    )
    .subscribe();

  newSseClient$.pipe(tap(() => broadcast({ state: appState$.value }))).subscribe();

  operator.message$.pipe(tap(logOperatorMessage), tap(handleProbeMessage()), tap(handleOpAddressMessage()), tap(handleButtonsMessage())).subscribe();
  operatorProbeNum$.pipe(tap((num) => updateState((state) => ({ ...state, probeNum: num })))).subscribe();
  operatorAddress$.pipe(tap((address) => updateState((state) => ({ ...state, opAddress: address })))).subscribe();
  operatorButtons$.pipe(tap((buttons) => updateState((state) => ({ ...state, btn1: buttons.btn1, btn2: buttons.btn2 })))).subscribe();

  const operataorButtons = createButtonStateMachine(operatorButtons$);

  operataorButtons.leaveIdle$.pipe(tap(interrupt)).subscribe();
  silence$.pipe(tap(triggerResponse)).subscribe();

  // Gemini Live API subscriptions
  geminiTranscript$.pipe(tap((text) => console.log(`🎤 Transcript: ${text}`))).subscribe();
  geminiResponse$.pipe(tap((text) => console.log(`🤖 Gemini: ${text}`))).subscribe();
}

main();

The web UI code followed the same modular pattern:

import { tap } from "rxjs";
import { appendDiagnosticsError, updateDiagnosticsState } from "./features/diagnostics";
import { initOperatorUI, updateOperatorUI } from "./features/operator";
import { initSimulationUI, updateSimulationUI } from "./features/simulation";
import { createSSEObservable } from "./features/sse";
import { state$, stateChange$ } from "./features/state";
import { initSwitchboardUI, updateSwitchboardUI } from "./features/switchboard";
import "./style.css";

initSwitchboardUI();
initOperatorUI();
initSimulationUI();

state$.pipe(tap(updateDiagnosticsState)).subscribe();

stateChange$.pipe(tap(updateSwitchboardUI), tap(updateOperatorUI), tap(updateSimulationUI)).subscribe();

export const sseEvents$ = createSSEObservable("http://localhost:3000/api/events");

sseEvents$.subscribe({
  next: (message) => {
    if (message.state) {
      state$.next(message.state);
    }
  },
  error: (error) => {
    appendDiagnosticsError(error);
  },
});

The UI now embodied the principle that view is a pure function of state. I rendered the JSON state of the server directly on the web UI. The rest of the interface was derived from that state.

Diagnostics panel showing raw server state

The Pivot

Hardware failure struck. The microphone randomly picked up high noise. The speaker had unstable contact and barely worked. With no time left to debug flaky audio hardware, I made a strategic decision: keep the custom input devices (probe and buttons), keep the custom output device (LEDs), but route audio through the computer's speakers. Based on the remaining functional hardware, I redesigned the application to be an interactive text adventure game.

Thanks to the modular refactor, this pivot was fast.

Gemini API generates branching story options
OpenAI TTS synthesizes speech for each option
LEDs illuminate when story options become available
Probe selects which option to preview (triggers audio playback)
Buttons commit the selection, advancing the story

The physical interface became a tangible story navigator. Probe a position to hear an option, press a button to choose your path.

Game Logic

The main server wires together the game flow:

textGenerated$.pipe(concatMap((index) => turnOnLED(switchboard, index))).subscribe();
operatorProbeNum$
  .pipe(
    filter((num) => num !== 7),
    tap((index) => previewOption(index))
  )
  .subscribe();
operatorProbeNum$
  .pipe(
    filter((num) => num === 7),
    tap(cancelAllSpeakerPlayback)
  )
  .subscribe();

operataorButtons.someButtonDown$
  .pipe(
    withLatestFrom(operatorProbeNum$),
    tap(cancelAllSpeakerPlayback),
    tap(() => turnOffAllLED(switchboard)),
    tap(([_, index]) => commitOption(index))
  )
  .subscribe();

The game logic module handles story generation and option management:

import { GoogleGenAI } from "@google/genai";
import { JSONParser } from "@streamparser/json";
import { BehaviorSubject, Subject } from "rxjs";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
import type { Handler } from "./http";
import { cancelAllSpeakerPlayback, playPcm16Buffer } from "./speaker";
import { appState$, updateState } from "./state";
import { GenerateOpenAISpeech } from "./tts";

const storyOptionsSchema = z.object({
  storyOptions: z.array(z.string().describe("A story beginning for a text adventure game.")),
});
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY! });

export const textGenerated$ = new Subject<number>();
export const assignments$ = new BehaviorSubject<{ index: number; text: string | null; audioBuffer: Promise<Buffer> | null }[]>([]);

let ongoingTasks = [] as AbortController[];

export async function previewOption(id: number) {
  const assignment = assignments$.value.find((a) => a.index === id);
  if (!assignment || assignment.text === null || assignment.audioBuffer === null) {
    console.error("Assignment not found or not ready");
    return;
  }

  cancelAllSpeakerPlayback();
  playPcm16Buffer(await assignment.audioBuffer);
}

export async function commitOption(id: number) {
  const assignment = assignments$.value.find((a) => a.index === id);
  if (!assignment || assignment.text === null) {
    console.error("Assignment not found or not ready");
    return;
  }

  killAllTasks();
  cancelAllSpeakerPlayback();

  // Commit the option to the story history
  updateState((state) => ({
    ...state,
    storyHistory: [...state.storyHistory, assignment.text!],
  }));

  // reset assignment slots
  assignments$.next(assignments$.value.map((a) => ({ ...a, text: null, audioBuffer: null })));

  const ac = new AbortController();
  ongoingTasks.push(ac);
  try {
    await generateOptionsInternal(ac, id);
  } finally {
    ongoingTasks = ongoingTasks.filter((task) => task !== ac);
  }
}

export function handleStartTextAdventures(): Handler {
  return async (req, res) => {
    if (req.method !== "POST" || req.url !== "/api/adventure/start") return false;

    killAllTasks();
    cancelAllSpeakerPlayback();
    assignments$.next([0, 1, 2, 3, 4, 5, 6].map((i) => ({ index: i, text: null, audioBuffer: null })));
    updateState((state) => ({ ...state, storyHistory: [] }));

    const ac = new AbortController();
    ongoingTasks.push(ac);

    try {
      await generateOptionsInternal(ac);
      res.writeHead(200);
      res.end();
    } finally {
      ongoingTasks = ongoingTasks.filter((task) => task !== ac);
    }

    return true;
  };
}

async function generateOptionsInternal(ac: AbortController, escapeIndex?: number) {
  const parser = new JSONParser();

  parser.onValue = (entry) => {
    if (typeof entry.key === "number" && typeof entry.value === "string") {
      console.log("Story option:", entry.value);

      const randomIndex = random(new Set(assignments$.value.filter((a) => a.text === null && a.index !== escapeIndex).map((a) => a.index)));
      if (randomIndex === null) {
        console.log("No available assignment slots, skip");
        return;
      }

      textGenerated$.next(randomIndex);
      assignments$.next(
        assignments$.value.map((a) =>
          a.index === randomIndex
            ? {
                ...a,
                text: entry.value as string,
                audioBuffer: GenerateOpenAISpeech(entry.value as string, ac.signal),
                visited: false,
              }
            : a
        )
      );
    }
  };

  const response = await ai.models.generateContentStream({
    model: "gemini-2.5-flash",
    contents: appState$.value.storyHistory.length
      ? `Based on the following story so far, generate 3 different story continuations for a text adventure game. Each option should be a one short verbal sentence with only a few words.

Story so far:
${appState$.value.storyHistory.join("\n")}
          `.trim()
      : `Generate 3 different story beginnings for a text adventure game. Each option should be a one short verbal sentence with only a few words.`.trim(),
    config: {
      responseMimeType: "application/json",
      responseJsonSchema: zodToJsonSchema(storyOptionsSchema as any),
      abortSignal: ac.signal,
    },
  });

  for await (const chunk of response) {
    const maybeOutput = chunk.candidates?.at(0)?.content?.parts?.at(0)?.text;
    if (!maybeOutput) continue;
    parser.write(maybeOutput);
  }
}

function killAllTasks() {
  ongoingTasks.forEach((ac) => ac.abort());
  ongoingTasks = [];
}

function random(set: Set<number>): number | null {
  if (set.size === 0) return null;
  const items = Array.from(set);
  return items[Math.floor(Math.random() * items.length)];
}

The final UI only displays the options chosen so far. The rest of the user interface is in the physical devices.

Final game UI with story plot

Let's go on an adventure!

Text adventure game in action

Reflection

The final system uses:

Node.js as HTTP and UDP server
node-ble for BLE communication with ESP32 devices
RxJS for functional reactive programming
lit-html for HTML templating
OpenAI SDK and Google Gen AI SDK for text and speech generation

The pivot in the middle of the project put the modular design to the test. I was able to modify the main logic without touching any communication code. What began as a voice communication tool transformed into an AI Dungeon. The architecture enabled rapid adaptation when hardware failed.

As a future improvement, I want to render each story decision point with a generated image that I can project into a room to bring more immersion to the game. If a room has multiple projectable surfaces, maybe each surface could represent a option that the player can "walk" into.

Appendix

Week 12: Network^3

Sun Chuanqi — Tue, 25 Nov 2025 00:00:00 GMT

Neil noted that I had already satisfied the networking requirement during the Input Device and Machine Building weeks. For this week, I made more progress on my final project without missing the opportunity in building something fun under the networking theme. I temporarily converted the hardware for my final project into a "whack-a-mole" game. In this setup, the switchboard lights up an LED, the user plugs in a phone jack to "whack" it, and then another LED lights up. By the end, I realized I had built three networks into a single project.

Network 1: Voltage as physical address

For my final project, I am building a walkie-talkie (the Operator) that plugs into a panel of phone jacks (the Switchboard). During Electronics fabrication week, I prototyped a 3-bit addressable interface using a TRRS connector. This week, I fabricated the phone jacks, complete with wiring, soldering, and mounting.

Soldering TRRS connector to ribbon wires

I used heat shrink tubes to reinforce the solder joints and prevent accidental shorts between adjacent pins. I repeatedly made the mistake of soldering the wire before adding the heat shrink tube. Because of the tube's size, I must add the tube first, solder the connection, slide the tube over the joint, and then shrink it with a heat gun. Eventually, I developed the muscle memory of "tube first, solder second."

Tube first, solder second, repeat after me...

I fell victim to the information denial trap observed in behavioral economics. This occurs when someone avoids learning information because they fear the consequences. I was on a happy streak soldering the TRRS jack wires and assumed that consistent soldering would suffice. As soon as I finished, I recalled Neil's warning that these connectors are "nasty" because the different terminals touch all conductive parts during insertion.

Here is my initial wiring configuration, which took over four hours to solder:

Switchboard
  - Tip: Address bit 0 (digital write high or ground)
  - Ring1: Ground
  - Ring2: Address bit 1
  - Sleeve: Address bit 2

Operator
  - Tip: Address bit 0 (digital read)
  - Ring1: Ground
  - Ring2: Address bit 1
  - Sleeve: Address bit 2

Let's visualize the different contact possibilities caused by the sliding motion:

TRRS
   TRRS 👈

TRRS
  TRRS 👈


TRRS
 TRRS 👈

TRRS
TRRS

When visualized in a grid, I saw the digital writes (Ring2, Sleeve) touching the ground (Ring1) causes a short:

	Tip (write)	Ring1 (GRD)	Ring2 (write)	Sleeve (write)
Tip (read)	✅	✅	✅	✅
Ring1 (GRD)		✅	⚠️	⚠️
Ring2 (read)			✅	✅
Sleeve (read)				✅

Realizing my mistake, I moved the ground to the tip so no other pins can touch it.

	Tip (GRD)	Ring1 (write)	Ring2 (write)	Sleeve (write)
Tip (GRD)	✅	✅	✅	✅
Ring1 (read)		✅	✅	✅
Ring2 (read)			✅	✅
Sleeve (read)				✅

Since I had already used heat shrink tubes to reinforce the pin headers, making these changes required removing the tubes and rearranging the wires. Fortunately, the ribbon wires allowed for rearrangement, though it took another two hours.

In the programming, the Switchboard is hard-wired to have a digital write high or ground on each address bit. I originally planned for 8 addresses using 3 bits, but I need to reserve an address for the "Unplugged" state. I ended up with 7 usable addresses (000 to 110), with 111 representing "Unplugged".

Here is the Switchboard code with digital write high on D6 pin, which supplies 3.3V to all the 1 address bits:

void setup() {
  pinMode(D6, OUTPUT);
  digitalWrite(D6, HIGH);
}

As seen on the PCB, each connector has six pins: two for the LED and four for the TRRS. Among the four TRRS pins, one is ground (second from the left in the left column, and second from the right in the right column).

The address bits are baked into the hardware

The Operator would read the address bits in a loop:

const int inputPins[] = { D3, D4, D5 };
const char* inputNames[] = { "D3", "D4", "D5" };
const int numInputs = 3;

void setup() {
  Serial.begin(115200);
  delay(50);

  for (int i = 0; i < numInputs; ++i) {
    pinMode(inputPins[i], INPUT_PULLUP);
    digitalWrite(inputPins[i], HIGH); // HIGH HIGH HIGH means unplugged
  }
}

void loop() {
  for (int i = 0; i < numInputs; ++i) {
    int v = digitalRead(inputPins[i]);
    Serial.print(inputNames[i]);
    Serial.print(": ");
    if (v == HIGH) {
      Serial.println("HIGH");
    } else {
      Serial.println("LOW");
    }
  }
  Serial.println();
  delay(50);
}

This was confirmed working in Electronics fabrication week.

Network 2: Mac and name as BLE address

I added LED lights to the Switchboard (see the Group assignment page for details). Since the TRRS connection provides only one-way communication from the Switchboard to the Operator, I needed a method for the Operator to send information back to change the LED states. A browser app was introduced to be the hub. The full data flow operates as follows:

The Operator reads a 3-bit address from the Switchboard.
The Operator sends the address to the browser app.
The browser app sends a new address to the Switchboard.
The Switchboard lights up the LED corresponding to that address.

I needed to associate the address of the phone jacks with the specific LED next to them. The wiring had changed over time, so the only guarantee was that they were distinct. I approached this empirically by probing them to determine the final addresses.

Probing for address, with the help from Adafruit TRRS Terminal Block

I placed a cheatsheet on the Switchboard case for easy reference:

Cheatsheet for LED number, LED digital pin, and TRRS bit address

Here is the key logic for lighting up an LED based on a BLE command received by the Switchboard:

#include <BLEDevice.h>
#include <BLEServer.h>
#include <BLEUtils.h>
#include <BLE2902.h>

#define UART_SERVICE_UUID "6e400001-b5a3-f393-e0a9-e50e24dcca9e"
#define UART_RX_UUID      "6e400002-b5a3-f393-e0a9-e50e24dcca9e"

BLEServer* pServer = NULL;
BLECharacteristic* pRxCharacteristic = NULL;
bool deviceConnected = false;

const int ledPins[] = {D0, D1, D2, D3, D7, D8, D9, D10};
const int numLeds = 8;

class MyServerCallbacks: public BLEServerCallbacks {
    void onConnect(BLEServer* pServer) {
        deviceConnected = true;
    }

    void onDisconnect(BLEServer* pServer) {
        deviceConnected = false;
    }
};

class MyRxCallbacks: public BLECharacteristicCallbacks {
    void onWrite(BLECharacteristic* pCharacteristic) {
        String rxValue = pCharacteristic->getValue();
        if (rxValue.length() > 0 && rxValue.startsWith("blink:")) {
            int ledIndex = rxValue.substring(6).toInt();
            if (ledIndex >= 0 && ledIndex < numLeds) {
                digitalWrite(ledPins[ledIndex], HIGH);
                delay(500);
                digitalWrite(ledPins[ledIndex], LOW);
            }
        }
    }
};

void setup() {
    for (int i = 0; i < numLeds; i++) {
        pinMode(ledPins[i], OUTPUT);
        digitalWrite(ledPins[i], LOW);
    }

    BLEDevice::init("Switchboard");

    pServer = BLEDevice::createServer();
    pServer->setCallbacks(new MyServerCallbacks());

    BLEService* pService = pServer->createService(UART_SERVICE_UUID);

    pRxCharacteristic = pService->createCharacteristic(
        UART_RX_UUID,
        BLECharacteristic::PROPERTY_WRITE
    );
    pRxCharacteristic->setCallbacks(new MyRxCallbacks());

    pService->start();

    BLEAdvertising* pAdvertising = BLEDevice::getAdvertising();
    pAdvertising->addServiceUUID(UART_SERVICE_UUID);
    pAdvertising->setScanResponse(false);
    pAdvertising->setMinPreferred(0x0);
    BLEDevice::startAdvertising();
}

void loop() {
    delay(10);
}

I noticed that one of the Bluetooth devices did not appear in the browser's device list. I discovered that the device name Switchboard became Switchbo. The Bluetooth library appears to shorten the name to 8 characters. There is a related issue reported by Android developers, though it is unclear if a similar issue lies with the ESP32 Bluetooth library or the Web Bluetooth API.

I fixed the device name overflow by shortening it:

deviceSw = await navigator.bluetooth.requestDevice({
-  filters: [{ name: "Switchboard" }],
+  filters: [{ name: "sw" }],
  optionalServices: [SERVICE_UUID],
});

The Operator reads the TRRS address and sends it to the browser via BLE:

#include <BLEDevice.h>
#include <BLEServer.h>
#include <BLEUtils.h>
#include <BLE2902.h>

#define UART_SERVICE_UUID "6e400001-b5a3-f393-e0a9-e50e24dcca9e"
#define UART_TX_UUID      "6e400003-b5a3-f393-e0a9-e50e24dcca9e"

BLEServer* pServer = NULL;
BLECharacteristic* pTxCharacteristic = NULL;
bool deviceConnected = false;
bool oldDeviceConnected = false;

const int inputPins[] = { D3, D4, D5 };
const int numInputs = 3;

class MyServerCallbacks: public BLEServerCallbacks {
    void onConnect(BLEServer* pServer) {
        deviceConnected = true;
    }

    void onDisconnect(BLEServer* pServer) {
        deviceConnected = false;
    }
};

void setup() {
  Serial.begin(115200);
  delay(50);

  for (int i = 0; i < numInputs; ++i) {
    pinMode(inputPins[i], INPUT_PULLUP);
  }

  BLEDevice::init("op");

  pServer = BLEDevice::createServer();
  pServer->setCallbacks(new MyServerCallbacks());

  BLEService* pService = pServer->createService(UART_SERVICE_UUID);

  pTxCharacteristic = pService->createCharacteristic(
      UART_TX_UUID,
      BLECharacteristic::PROPERTY_NOTIFY
  );
  pTxCharacteristic->addDescriptor(new BLE2902());

  pService->start();

  BLEAdvertising* pAdvertising = BLEDevice::getAdvertising();
  pAdvertising->addServiceUUID(UART_SERVICE_UUID);
  pAdvertising->setScanResponse(false);
  pAdvertising->setMinPreferred(0x0);
  BLEDevice::startAdvertising();
}

void loop() {
  if (!deviceConnected && oldDeviceConnected) {
    delay(500);
    pServer->startAdvertising();
    oldDeviceConnected = deviceConnected;
  }

  if (deviceConnected && !oldDeviceConnected) {
    oldDeviceConnected = deviceConnected;
  }

  String probe = "";
  for (int i = 0; i < numInputs; ++i) {
    int v = digitalRead(inputPins[i]);
    probe += (v == HIGH) ? "1" : "0";
  }

  if (deviceConnected) {
    pTxCharacteristic->setValue((uint8_t*)probe.c_str(), probe.length());
    pTxCharacteristic->notify();
  }

  delay(100);
}

In the web app, I needed to account for connection bounce caused by the sliding motion. I used the RxJS library to handle the debounce. The web app serves as the brain. It tracks the current LED, checks if the user probes the correct address, and sends a command to the Switchboard to turn off that LED and light up another.

import { Subject, debounceTime, distinctUntilChanged, BehaviorSubject } from "https://esm.sh/rxjs";

const SERVICE_UUID = "6e400001-b5a3-f393-e0a9-e50e24dcca9e";
const TX_CHAR_UUID = "6e400003-b5a3-f393-e0a9-e50e24dcca9e";
const RX_CHAR_UUID = "6e400002-b5a3-f393-e0a9-e50e24dcca9e";

let charTxSw = null;
let charRxOp = null;
let probeSubject = new Subject();
let currentTarget = 3;
let scoreSubject = new BehaviorSubject(0);

document.getElementById("connectBtnSw").addEventListener("click", async () => {
  const deviceSw = await navigator.bluetooth.requestDevice({
    filters: [{ name: "sw" }],
    optionalServices: [SERVICE_UUID],
  });
  const server = await deviceSw.gatt.connect();
  const service = await server.getPrimaryService(SERVICE_UUID);
  charTxSw = await service.getCharacteristic(RX_CHAR_UUID);
  const encoder = new TextEncoder();
  await charTxSw.writeValue(encoder.encode("off:all"));
  await charTxSw.writeValue(encoder.encode("on:3"));
});

document.getElementById("connectBtnOp").addEventListener("click", async () => {
  probeSubject = new Subject();
  scoreSubject = new BehaviorSubject(0);
  const deviceOp = await navigator.bluetooth.requestDevice({
    filters: [{ name: "op" }],
    optionalServices: [SERVICE_UUID],
  });
  const server = await deviceOp.gatt.connect();
  const service = await server.getPrimaryService(SERVICE_UUID);
  charRxOp = await service.getCharacteristic(TX_CHAR_UUID);
  await charRxOp.startNotifications();
  charRxOp.addEventListener("characteristicvaluechanged", (e) => {
    probeSubject.next(new TextDecoder().decode(e.target.value.buffer).trim());
  });
  probeSubject.pipe(distinctUntilChanged(), debounceTime(500)).subscribe((value) => {
    const num = parseInt(value, 2);
    if (num === currentTarget) {
      const encoder = new TextEncoder();
      charTxSw.writeValue(encoder.encode("off:" + currentTarget)).then(() => {
        let randomLed = Math.floor(Math.random() * 7);
        while (randomLed === currentTarget) randomLed = Math.floor(Math.random() * 7);
        charTxSw.writeValue(encoder.encode(`on:${randomLed}`)).then(() => {
          currentTarget = randomLed;
          scoreSubject.next(scoreSubject.value + 1);
        });
      });
    }
  });
});

const apiKeyInput = document.getElementById("apiKey");
apiKeyInput.value = localStorage.getItem("htmaa-matti-key") || "";
apiKeyInput.addEventListener("input", () => localStorage.setItem("htmaa-matti-key", apiKeyInput.value));

Network 3: URL as web address

To satisfy the group assignment requirement of networking with another project, I added logic for the browser to HTTP POST the current score to Matti's server. This, in turn, displays the score on his e-ink display.

+scoreSubject.subscribe((score) => {
+  const apiKey = document.getElementById("apiKey").value;
+  if (apiKey) {
+    fetch("https://api.mistermatti.com/htmaa-final/text", {
+      method: "POST",
+      headers: { "X-API-Key": apiKey, "Content-Type": "application/json" },
+      body: JSON.stringify({ text: score.toString() }),
+    });
+  }
+});

With this add-on, the browser posts to the server whenever the score updates. See Group assignment page for details.

Web app notifying remote server

Appendix

Week 11: Mob-making A Machine

Sun Chuanqi — Tue, 18 Nov 2025 00:00:00 GMT

This is a group project week. This document captures the tasks I was involved in. See the group project page for the full context.

Labubu

Miranda pitched the idea of building an icosahedron robot that can move based on IMU data. Each face would host a Labubu that punches out to propel the icosahedron in the desired direction. I'm too old to understand the cultural significance of Labubu, but the idea sounds cool. Towards the end of the project, we replaced Labubu with our professor Neil's face. This made the project much more relatable but also raised the stakes considerably.

Project Organization

The group divided into four sub-teams: MechE, Electronics, Software, and Creative. I focused on Software.

During the initial project planning, we discussed how to organize the codebase. People originally proposed a formal PR process and branch-based workflow. I wanted to emphasize people over technology and advised that sub-team leaders should be responsible for anticipating dependencies and conflicts with other teams. Communication should happen early. Team leaders should adapt their version control strategy based on their team members' skills and preferences.

Don't communicate by sharing memory; share memory by communicating.

— Go Language Design Principle

I proposed a controversial idea: organize code by people and duplicate code to surface full history at all times. Consider this folder structure:

- PersonA
  - sensing
  - networking
- PersonB
  - actuation
  - webUI
...

Fundamentally, this is a copy-paste version control system that most professional developers would cringe at. I advocated for this style because we needed to quickly branch and fork each other's ideas. I expected that 90% of the code in the beginning would be self-contained one-off experiments that wouldn't evolve later. Exposing everyone's work in the same branch at the same time provides several benefits:

Easy remixing of each other's code
People using AI can reference multiple components from different people and get help with integration
People's work becomes a natural source of documentation. History is not buried in git logs.
No merge conflicts because everyone works in their own folder

As a caveat, the person sharing code is responsible for discussing incoming breaking changes. The person consuming code is responsible for stating assumptions and expectations.

This organization worked well initially. During the second half of the project, we concluded that we needed a point of integration. Our final folder structure became:

- integration
  - controller
  - web
- PersonA
  ...
- PersonB
  ...

I fully understand this is not how git workflows are supposed to work. But for mob-like student projects, this organization helped us see each other's code without branching overhead. I would advocate for the same strategy in future projects.

Division of Labor

On the first night, Matti and I discussed task division. We wanted to create independent modules that could be worked on in parallel and reduce dependencies between modules by defining clear interfaces. This is what we came up with:

Sensing (C++)

Understand the format of IMU data from our sensor
Prepare data on the Xiao before sending to network

Networking (C++)

WiFi connection management
Track IP address of the remote control (PC)
Send IMU data from Xiao to remote control over UDP
Receive servo motor number from remote control to Xiao over UDP

Planning (JavaScript)

Use math to convert move front/back/left/right into the correct servo motor number

Actuation (C++)

Drive the motor based on the received servo motor number

UI (JavaScript/HTML/CSS)

Create a web UI
Visualize device orientation
Show buttons to move front/back/left/right
Show buttons to manually move individual servo motors

At the end of the project, a similar structure was reflected in our code. This reminded me of Conway's Law. We can use Conway's Law to our advantage by asking what kind of teams and collaborations we desire, then modularize our project to maximize that organizational structure.

Integration Folder Structure

Software Architecture

We agreed on a high level architecture: shift as much computation to the PC as possible. It's much easier to iterate and debug on the PC than on the Xiao. Running the server on PC offered several advantages:

Easier to debug
Avoid using ESP32 as Wifi access point so laptop can connect to the internet and use generative AI to accelerate coding
Acknowledged drawback: higher latency because of going through school WiFi

This concept was reflected in every subsequent decision. The Xiao's logic is simple:

Xiao sends low level IMU data to PC: quaternion WXYZ and accelerometer XYZ
Xiao takes a servo motor number and fully extends it, then retracts it based on predefined PWM sequence

The PC does the heavy lifting: interpreting IMU data, solving the geometric puzzle, calibrating, and determining which servos to move based on user commands.

Networking Proof of Concept

We started with UDP over WiFi because I already had a similar system working from Input/Output week for streaming voice. I implemented a few diagnostic programs to test UDP performance.

First, I wanted to understand the performance characteristics of Xiao ESP32 UDP over WiFi. I wrote a simple Node.js server that echoes any UDP message back to the sender:

const dgram = require("dgram");
const os = require("os");

const server = dgram.createSocket("udp4");

server.on("message", (msg, rinfo) => {
  console.log(`Received ${msg.length} bytes from ${rinfo.address}:${rinfo.port}`);
  try {
    const data = JSON.parse(msg.toString());
    console.log(`currentTime: ${data.currentTime}, latency: ${data.latency}`);
  } catch (e) {
    console.log(`Invalid JSON: ${msg.toString()}`);
  }
  // Send back the same message
  server.send(msg, 0, msg.length, rinfo.port, rinfo.address, (err) => {
    if (err) console.error("Error sending response:", err);
  });
});

server.on("listening", () => {
  const address = server.address();
  const interfaces = os.networkInterfaces();
  let localIP = "127.0.0.1"; // fallback
  for (let iface in interfaces) {
    for (let addr of interfaces[iface]) {
      if (addr.family === "IPv4" && !addr.internal) {
        localIP = addr.address;
        break;
      }
    }
    if (localIP !== "127.0.0.1") break;
  }
  console.log(`UDP server listening on ${localIP}:${address.port}`);
});

server.bind(41234); // Bind to port 41234

On the ESP32 side, I sent UDP packets in bursts and measured latency during peak load:

#include <WiFi.h>
#include <AsyncUDP.h>

const char* WIFI_SSID = "MLDEV";
const char* WIFI_PASSWORD = "";

AsyncUDP udp;
IPAddress targetIP(192, 168, 41, 229);
const unsigned int targetPort = 41234;
int packetNum = 0;
unsigned long lastLatency = 0;
int burstCount = 0;
const int BURST_SIZE = 1;

void setup() {
  Serial.begin(115200);

  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  unsigned long startTime = millis();
  while (WiFi.status() != WL_CONNECTED && millis() - startTime < 300000) {
    delay(500);
    Serial.print(".");
  }
  if (WiFi.status() != WL_CONNECTED) {
    Serial.println("WiFi Failed");
  }
  Serial.println("Connected to WiFi");
  Serial.print("IP Address: ");
  Serial.println(WiFi.localIP());

  if (udp.connect(targetIP, targetPort)) {
    Serial.println("UDP connected");
    udp.onPacket([](AsyncUDPPacket packet) {
      String msg = String((char*)packet.data(), packet.length());
      int colonPos = msg.indexOf("currentTime\":");
      if (colonPos != -1) {
        colonPos += 13;
        int commaPos = msg.indexOf(",", colonPos);
        if (commaPos != -1) {
          String timeStr = msg.substring(colonPos, commaPos);
          unsigned long sentTime = strtoul(timeStr.c_str(), NULL, 10);
          lastLatency = millis() - sentTime;
        }
      }
      packet.printf("Got %u bytes of data", packet.length());
    });
  }
}

void loop() {
  if (burstCount < BURST_SIZE) {
    packetNum++;
    burstCount++;
    String json = "{\"currentTime\":" + String(millis()) + ",\"packetNum\":" + String(packetNum) + ",\"latency\":" + String(lastLatency) + "}";
    udp.print(json);
    Serial.println("Sent: " + json);
  } else {
    Serial.println("Sleeping for 1 second...");
    delay(10);
    burstCount = 0;
  }
}

The testing revealed several characteristics:

Xiao UDP can send over 1000 Hz to a PC
The system crashes, presumably due to buffer overflow, when PC sends too much data
Typical latency: 30-50ms
Worst latency: 100-200ms
Best latency: 4-6ms
Xiao needs to sleep 1ms between sends. Otherwise, it will be blocked from reading packets

I realized that each person's laptop has a different IP address. I implemented a simple IP discovery protocol. People can announce their IP address on a server. The ESP32 polls the server to get the latest IP address of the laptop.

IP Discovery Tool

This tool was eventually taken offline due to switching from Wifi to Bluetooth. But I plan to use the technique for my final project in which IP discovery is still needed.

Interface First

To implement the web server without existing microcontroller code, I encouraged the team to define the networking contract first. The payloads contain Quaternion (w, x, y, z) and Accelerometer (ax, ay, az) readings in newline-delimited JSON:

esp32 -> laptop

{
  "w": 0.875,
  "x": 0.0243,
  "y": 0.0393,
  "z": -0.482,
  "ax": 17.5781,
  "ay": 3.1738,
  "az": 1025.3906
}

Each notification is parsed by bluetooth.js and forwarded to the UI; threejs-vis.js consumes the quaternion stream to animate the model.

laptop -> esp32

General format is JSON with "cmd" and "args" fields. For simplicity, "args" is a string. It is optional.

Move servo command

{ "cmd": "move_servo", "args": "5,12" }

Reset device command

{ "cmd": "reset" }

We discussed custom bit packing to reduce bandwidth consumption. I advocated for JSON for simplicity and agreed that we could optimize later if needed. The JSON protocol was eventually superseded by a custom op-code based binary protocol to conserve BLE bandwidth.

In retrospect, I would still advocate for the JSON protocol with more concise names. The bit packing optimization had marginal gain and made serialization/deserialization much more error prone.

Server Implementation

Yufeng ran the demo code for the Adafruit IMU board and started developing sensor data processing. Thanks to the data format contract, I was able to work in parallel. I mocked the IMU data in a separate Xiao and communicated with a Node.js server over UDP. This is my code that mocks the sensor data:

// Mock IMU sensor data
float gx = 0.0, gy = 0.0, gz = 0.0;  // Gyroscope in degrees
float mx = 0.0, my = 0.0, mz = 0.0;  // Compass/Magnetometer in degrees

void updateMockSensorData() {
  // Update mock IMU data with random changes
  gx += random(-10, 11) * 0.1;  // Change by -1.0 to +1.0 degrees
  gy += random(-10, 11) * 0.1;
  gz += random(-10, 11) * 0.1;
  mx += random(-10, 11) * 0.1;
  my += random(-10, 11) * 0.1;
  mz += random(-10, 11) * 0.1;

  // Keep values in reasonable ranges
  gx = constrain(gx, -180, 180);
  gy = constrain(gy, -180, 180);
  gz = constrain(gz, -180, 180);
  mx = constrain(mx, 0, 360);
  my = constrain(my, 0, 360);
  mz = constrain(mz, 0, 360);
}

String getSensorDataJSON() {
  // Create JSON array format: [gx,gy,gz,mx,my,mz]
  return "[" + String(gx, 2) + "," + String(gy, 2) + "," + String(gz, 2) + ","
             + String(mx, 2) + "," + String(my, 2) + "," + String(mz, 2) + "]";
}

Determine High Level Logic

Multiple people were adding pieces to the Xiao code. I started a refactoring effort to modularize the code so that the high level logic would be easier to understand. This is the pseudocode I came up with:

setup() {
  wifi = connect_wifi();
  laptop_ip = discover_laptop(wifi);

  on_message_received = (message) => {
    handle_reset(message);
    handle_servo_command(message);
  };

  handle_laptop_udp_message(wifi, laptop_ip, on_message_received);
}

loop() {
  sendor_data = read_imu_sensor();
  send_udp_message(wifi, laptop_ip, sendor_data);
}

I discussed the high level design with the group to ensure everyone shared the same understanding. In theory, the design allows future I/O to plug into the main program without needing to change other modules' code.

Sensor Integration Test

The EE team provided the electronics on Friday during the StudCom social tea hour. We uploaded our sketch and started testing.

Dumping IMU data to the web UI over UDP

Testing against interference in the crowded StudCom tea party

The results were concerning. In the Media Lab tea party with about 40 people, the device experienced 3+ seconds of latency. The WXYZ quaternion took 12+ seconds to stabilize.

Bad news: UDP plus WiFi clearly wouldn't work. Good news: we discovered this early enough. There was still time to pivot. Our communication wasn't tightly coupled to the sensor logic.

The Big Migration

After testing, I took Miranda's Bluetooth 2-way communication code and used Claude 4.5 Sonnet to migrate the WiFi plus UDP code to Bluetooth:

Plan step by step, we are going to swap out the wifi + UDP + WebSocket based communication between ESP32 and Node.js with a simpler Bluetooth BLE based communication between ESP32 and the Web page, using Web Bluetooth API.

The change will include at least the following:
1. Remove UDP and Wifi on both ESP32 and Node.js
2. Add BLE on both ESP32 and the web page
3. Remove IP discovery code
4. Treat the server folder as static. use simple npx command to serve the file and not worry about maintaining a node.js server

We already have working reference implemention in #file:bluetooth. You can use that code as skeleton.

Keep the ESP32 organized by files, similar to existing structure lib-xx-name.ino; delete files that are no longer in use.

Make sure to carefully map out the migration and execute it with a checklist.

Miraculously, the migration worked on the first try. AI is such a game changer.

Integrating Servo

Saetbyeol implemented the servo motor control. When I integrated her work, the servo could not respond to commands from the PC. We suspected I/O blocking. After investigation, we realized that the BLE communication loop was blocking:

  if (isBLEConnected()) {
    String sensorData = getSensorJSON();
    sendBLEMessage(sensorData); // <-- need to disable this line in order to send any command to the ESP32, why?
  }

The solution was to use a timer to send sensor data every 20ms instead of blocking the main loop. The 20ms interval was empirically determined. I wasn't happy with this solution, as the cool down period could vary depending on unknown factors, but we proceeded and decided to revisit it later if needed.

static unsigned long lastSendTime = 0;
const unsigned long SEND_INTERVAL_MS = 20;

void setupSensorTransmission() {
  lastSendTime = 0;
  Serial.println("Sensor transmission initialized");
}

void sendSensorDataIfReady() {
  if (!isBLEConnected()) {
    return;
  }

  unsigned long currentTime = millis();

  if (currentTime - lastSendTime >= SEND_INTERVAL_MS) {
    String sensorData = getSensorJSON();
    sendBLEMessage(sensorData);
    lastSendTime = currentTime;
  }
}

With this fix, we got the first full integration where sensors and servos are both working. Here is the celebratory dance:

Servos dancing while sensors streaming IMU data over BLE

Debugging MUX Issue

We tested driving multiple servos through the MUX PWM PCA9685 board. The code behaved erratically. We discovered two issues:

The MUX board was very sensitive. Touching it with a hand could trigger weird behavior.
Our code didn't fully implement the MUX behavior.

Matti directed us to run the Adafruit official MUX PWM PCA9685 library code. We confirmed that the board wiring was correct and all servo motors were functional. We solved the issue by using the Adafruit official library example code as a starting point.

Matti also discovered how to serial connect two MUX boards by soldering the address pin to set one board at 0x41 instead of 0x40.

Pads for changing the address of PCA9685 MUX Board

The base address is 0x40. The EE team would later solder A3 (0x08) for 0x48 and A5 (0x20) for 0x60. Had we known sooner, we could have matched their addressing scheme in our dev boards.

Stress Testing BLE Communication

We discovered that rapidly sending BLE messages would cause the connection to drop. Matti suggested using different TX characteristics for namespaced communication. This would save bandwidth by eliminating command names.

Systematic Testing of BLE in Web Bluetooth API

Sending characters at high frequency triggered an error:

sendInterval = setInterval(async () => {
  try {
    const encoder = new TextEncoder();
    const data = encoder.encode(message);
    await charTx.writeValue(data);
    log(`TX: ${message}`);
  } catch (error) {
    log(`SEND ERROR: ${error.message}`);
    stopSending();
  }
}, interval);

Error code from the Web Bluetooth API:

SEND ERROR: GATT operation already in progress.

This indicated that flow control was needed. On the browser side, we could throttle or buffer the messages. I reasoned through the options:

Throttling could help, but throughput is environment dependent. We would end up being very conservative and losing performance.
Buffering made more sense. We just needed to ensure only one transmission at a time.

I solved flow control with a queue-based scheduler. Using the RxJS mergeMap operator, I could easily toggle between single-thread mode and unrestricted concurrency mode:

const concurrency = useScheduler ? 1 : undefined;

const send$ = interval(intervalMs).pipe(
  takeUntil(stopBrowserSend$),
  tap(() => browserQueueSize++),
  mergeMap(async () => {
    const encoder = new TextEncoder();
    const data = encoder.encode(message);
    await charTx.writeValue(data);

    browserQueueSize--;
    messageSent$.next();
    log(`[Test 1] TX: ${message}`);
  }, concurrency)
);

Based on this idea, I implemented a comprehensive diagnostic tool to profile the BLE performance.

Performance Characterization

The tool allows users to measure latency and throughput for varying message sizes. I was able to confirm, the scheduler was able to max out the throughput without triggering errors.

BLE benchmarking in action

Best Case: Side by Side in Same Room

Bandwidth:

Browser to ESP32: 14 messages/sec
ESP32 to Browser: 100 messages/sec

Latency: 92ms average, min: 84ms, max: 140ms

Removing the antenna did not reduce performance at close range. However, as I walked away, performance dropped quickly. The connection was lost at 5 meters.

At Distance of 30 Meters Through One Glass Wall

Unable to establish a new connection at this distance, but could maintain a previously established connection to 30 meters.

Bandwidth:

Browser to ESP32: 8 messages/sec
ESP32 to Browser: 50 messages/sec

Latency: 250ms average, min: 89ms, max: 540ms

Realistic Usage: Inside Metal Icosahedron Structure at 10 Meters Distance

Bandwidth:

Browser to ESP32: 14 messages/sec
ESP32 to Browser: 100 messages/sec

Latency: 230ms average, min: 157ms, max: 332ms

Integrate Scheduler

The diagnostic tool used RxJS. For production, I wanted to avoid adding more libraries. I implemented a standalone scheduler that uses a queue to ensure single threaded execution of tasks. I wanted the scheduler to be hidden behind the bluetooth module so the caller wouldn't have to worry about scheduling. Multiple callers of the bluetooth module would be scheduled in a first-in, first-out manner.

Scheduler implementation:

class AsyncTaskScheduler {
  constructor() {
    this.queue = [];
    this.isProcessing = false;
  }

  /**
   * Add a task to the queue and process it
   * @param {Function} taskFn - Async function to execute
   * @returns {Promise} - Resolves when task completes
   */
  async enqueue(taskFn) {
    return new Promise((resolve, reject) => {
      this.queue.push({ taskFn, resolve, reject });
      this.processQueue();
    });
  }

  /**
   * Process the queue sequentially
   */
  async processQueue() {
    // If already processing, return
    if (this.isProcessing) {
      return;
    }

    // If queue is empty, return
    if (this.queue.length === 0) {
      return;
    }

    this.isProcessing = true;

    while (this.queue.length > 0) {
      const { taskFn, resolve, reject } = this.queue.shift();

      try {
        const result = await taskFn();
        resolve(result);
      } catch (error) {
        console.error("Task failed in scheduler:", error);
        reject(error);
        // Continue processing the rest of the queue despite the error
      }
    }

    this.isProcessing = false;
  }

  /**
   * Clear all pending tasks
   */
  clear() {
    // Reject all pending tasks
    while (this.queue.length > 0) {
      const { reject } = this.queue.shift();
      reject(new Error("Queue cleared"));
    }
  }

  /**
   * Get queue size
   * @returns {number} - Number of pending tasks
   */
  get size() {
    return this.queue.length;
  }

  /**
   * Check if scheduler is currently processing
   * @returns {boolean}
   */
  get busy() {
    return this.isProcessing;
  }
}

With the scheduler in place, I was able to dispatch commands at high speed without causing BLE transmission errors.

Reflection

The bearing of a child takes nine months, no matter how many women are assigned.

— Fred Brooks, The Mythical Man-Month

It's tempting to add more process and throw more people at problems. In this project, I found it most effective when two-people micro-teams pair programmed to solve one problem. I saw this working very well between Matti and Miranda, and between Saetbeyol and me. This reinforced the importance of decomposing problems into smaller chunks that can be solved by small teams in parallel.

This project also challenged our conventional wisdom about formal software development. Version control, testing, code review, CI/CD, TypeScript, ES6 modules, linting, formatting, and design patterns were all thrown out the window. Maybe for the better. I'm always fascinated by emerging practices under extreme constraints. In this project, I'm convinced that less is more when the timeline is short and the skill levels are diverse.

Appendix

Week 10: Neil's Law

Sun Chuanqi — Mon, 10 Nov 2025 00:00:00 GMT

Hofstadter's law: It always takes longer than you expect, even when you take into account Hofstadter's law.

Neil's law: You will cast the opposite of what you expect, even after learning about Neil's law.

Group Assignment

This week, I joined Kat's training session to learn about molding and casting. She showed us several examples covering molding, casting techniques, and materials. I also reviewed Safety Data Sheets (SDS) for two materials by Smooth-On: Oomoo 30 silicone rubber and Smooth-Cast 300 plastic. Understanding material properties and safety protocols is essential when working with these chemicals.

See additional documentation in the group notes.

Manual Particle Accelerator

Who doesn't like particle physics? Smashing particles together at near light speed could make cool images and win you a Nobel prize. Worried about creating a black hole? That's been debunked. In this project, you can safely accelerate a particle by hand!

I imagined a torus shaped device. It hurts my brain to imagine the positive/negative relationships involved in casting. I found sketching an effective way to develop the concept.

Concept sketch

After sketching out the positive/negative and hard/soft material relationships, I was ready to start fabrication. The design called for two shell components that, when closed like a clam, would form a torus in which you could accelerate a metal ball by shaking it. Similar to how self-winding watch feels.

The CAM Debacle

My initial plan was simple and naive: CNC wax mold → plastic part. I've been using Onshape for its Linux compatibility, but learning from Week 7 that using other people's Fusion 360 for CAM adds significant overhead, I decided to learn FreeCAD CAM this week to have a personalized workflow.

I studied two reference tutorials to understand the CAM capabilities:

Attempt 1: Onshape Modeling → FreeCAD CAM

I started by measuring the stock size so I could size the parts accordingly.

Stock measurement

Width: 76.45mm
Length: 87.45mm
Height: 38.21mm

I modeled the parts in Onshape and imported them into FreeCAD for toolpath generation. I found an effective workflow that uses a boolean operation to simulate the casting.

Simulated casting using the subtract boolean operation

A quick boolean operation against stock geometry immediately revealed a design flaw: I had edges without thickness, which would be impossible to mill.

Boolean operation reveals design flaw

After fixing the geometry, I managed to set up the machine job by creating tool bits based on example endmill shapes and modifying the JSON configuration, following this YouTube tutorial.

{
  "version": 2,
  "name": "1/8 inch Endmill",
  "shape": "endmill.fcstd",
  "parameter": {
    "CuttingEdgeHeight": "30.0000 mm",
    "Diameter": "3.1750 mm",
    "Length": "50.0000 mm",
    "ShankDiameter": "3.0000 mm"
  },
  "attribute": {}
}

However, a new problem emerged. The toolpath algorithm couldn't fit the bit in the narrow, ring-like pocket geometry.

Toolpath with excessive vertical movement

Increasing the ring width to be significantly greater than the tool diameter would solve the problem, but I didn't want to compromise the design. Additionally, FreeCAD's multiple toolpath feature proved very buggy. After generating the first toolpath, producing paths for the remaining material simply had no effect: subsequent toolpaths would start over from the surface of the stock.

Attempt 2: FreeCAD Modeling → FreeCAD CAM

Thinking the import process might be causing issues, I decided to model directly in FreeCAD.

Modeling in FreeCAD

The transition from Onshape to FreeCAD was rough but manageable given that they share a similar mental model. I finished the mold design and tested it in CAM, but the multiple toolpath issues persisted.

Toolpath missing half of the geometry

Attempt 3: Switching to Mods

Frustrated with FreeCAD's CAM limitations, I switched to Mods for toolpath generation, using the G-code/mill 2.5D stl program. Unfortunately, Mods couldn't generate sequential toolpaths that build on each other. Worse, I realized that my U-pipe geometry would require a ball endmill, and adding custom bit geometry would take too long to implement.

Excessive horizontal movement in Onshape model

Incomplete paths in FreeCAD model

Mods failed to generate proper toolpaths with issues similar to what I experienced in FreeCAD. At this point, I suspected fundamental problems with my modeling approach rather than toolpath algorithms.

Summary of Failed Attempts

CAD Software	CAM Software	Issues Encountered
Onshape	FreeCAD	Unnecessary vertical travels
FreeCAD	FreeCAD	Ibid, and missing toolpaths on half of the ring
Onshape	Mods	Unnecessary horizontal back-and-forth travels
FreeCAD	Mods	Ibid, and missing toolpaths on half of the ring

After spending considerable time troubleshooting, I needed to embrace pragmatism. In the spirit of supply-driven project management, I decided to move forward with something I could make real progress on: 3D printing the mother mold instead of CNC milling wax.

Table of Contents, Literally

Before diving into the technical details, I want to show the physical manifestation of this week's iterative process. I laid out all my artifacts on a table. Each row visualizes an iteration cycle from mother mold to a point where I either reached a milestone or learned from a mistake.

v1 through v7 of molding and casting using 3D printing

V1: Perfectly Making The Wrong Thing

First Iteration: PLA Mother Mold

I slightly modified the CNC model to be the mother mold, sliced it with PrusaSlicer, and printed it in PLA.

Modeling the mold

The print came out clean despite pronounced layer lines. I wanted to cast it first to see how bad the surface finish would be. Let's try Smooth-On Oomoo 30 silicone rubber.

Gathering materials for casting

Estimating the amount of silicone rubber needed using water

I noticed that part B has much higher density than part A. It makes sense to add A first so part B can sink and improve the mixing.

Before mixing, clear separation between part A and B

After mixing, the color is uniform

Using a small cup was a mistake. A larger cup would make mixing much easier.

Pouring is a battle against bubbles and requires perfect balance: too fast, you could spill or introduce bubbles. Too slow, it could drip and also cause bubbles. Too high, it could splash and make bubbles. Too low, the bubbles wouldn't be able to stretch and pop before entering the mold.

Pouring in action

The curing process requires keeping the mold undisturbed for 6 hours.

Curing

The result looks great, except... I cast the opposite of what I wanted. I knew Neil warned us in the lecture about making such a mistake. How on earth did I still manage to do it? In retrospect, I subconsciously believed that I could cast directly from the 3D printed mold. If I were able to cast hard material from the 3D printed mold, the outcome would be correct.

Good but wrong result

The result also confirmed the surface finish issue. The layer lines transferred to the silicone mold and would ultimately transfer to the final cast. I needed to smooth the surface, so I considered a few options:

Resin coating: Kat warned that resin inhibits silicone curing
Wax coating: Wax melts PLA, requiring a switch to PETG
Acetone vapor smoothing: Only works for ABS, not PLA or PETG
Sanding and polishing: Labor intensive and inconsistent results

By elimination, I decided to try wax coating with PETG for the next iteration.

V2: PETG with Wax Coating

I started V2 before realizing the positive/negative issue. Since the goal was to characterize surface treatment methods, I proceeded with the PETG print with the wrong geometry. At least I would be able to compare identical geometries across different surface treatments.

3D printed PETG mold

I used a heat gun to melt wax pellets and brushed them onto the PETG mold.

Wax pellets

Brushing wax onto the PETG mold

The final step involved applying heat to re-melt the wax in the mold and drain the excess. Unfortunately, I warped the PETG during the drain process when the material was still hot and pliable.

Warped PETG mold

Despite the warping, I proceeded to cast the silicone mold. The results were disappointing. The wax coating couldn't eliminate the layer lines, and worse, it destroyed the sharp edges that were critical to my design.

Left: PLA without surface treatment. Right: PETG with wax coating.

V3-V5: Pushing The Limit of PLA

I switched back to PLA and recalled a feature from the 3D printing assignment: ironing. This could potentially smooth the top surface without requiring post-processing. I explored several tweaks to optimize the print quality:

Added ironing at 10% flow and 0.1mm gap for surface smoothing
Reduced layer height to 0.05mm
Switched to concentric infill

Setup for ironing in PrusaSlicer

The concentric infill change had an unexpected benefit of significantly speeding up the print. My geometry is circular, so concentric infill minimizes travel moves compared to the default rectilinear pattern.

I tested various ironing parameters to find the optimal settings:

Setting up different processes in one job

2 by 2 grid of results

Position	Setting	Result
Top left	Monotonic line infill	Baseline
Top right	Concentric infill	Cleaner, faster
Bottom left	Ironing: 0.15mm spacing, 15% flow	Good improvement
Bottom right	Ironing: 0.1mm spacing, 10% flow	Best surface finish

In summary, everything we need to know to improve 3D printing is already captured in the culinary wisdom:

Low and slow.

-- Texas BBQ pitmasters

Low layer height and slow ironing did the trick. I observed much better interior layers thanks to concentric infill:

Ironing and concentric infill in action

Here is the full mold using the ironing and concentric infill settings. Unfortunately, the 3D printer had some issues extruding consistently despite my tuning of the temperature. It was not as good as my characterization test from another printer.

Ironed mold, sub-optimal surface and rough wall texture

When casting the silicone mold from the new PLA mother mold, I used a glass plate to press down the backside of the rubber as it cured, creating a flat surface to make the final casting easier to level.

Leveling the surface

This made a perfectly flat rubber surface, but it also created a vacuum suction that made it very difficult to remove it from the glass. I would not recommend this technique to others.

Due to the rough wall texture, the silicone mold adhered strongly to the PLA mother mold. During demolding, I tore the wall apart from the rubber base.

Damaged silicone mold after demolding

The right-most gutter made demolding very difficult

I wanted to proceed with the final casting to gain more experience and reveal other potential issues.

3D printed support ring to hold the damaged mold in shape

With the support ring, I was able to cast the final parts. The Smooth-On Smooth-Cast 300 plastic came with the instruction that you should stir or shake the bottles before mixing. That was a very bad idea. After shaking, the mixture was full of bubbles that would not go away. I had to switch to another bottle while letting those bubbles dissipate over time.

Casting the final parts despite the damaged mold (left)

I attempted to remove bubbles using a vacuum chamber, but it backfired. The vacuum caused surface roughness on the bottom side of the mold that would be visible in the final part.

Using vacuum chamber to remove bubbles

This marks the first production of the final parts. Good news is that the geometry is correct this time.

Design validated

To make the molds ready for assembly, I deburred the edges and sanded down rough surfaces.

Deburring the casted parts

For fun, I hand-painted some graphic details with a sharpie. It was clear that vacuum processing was not only unnecessary but also detrimental to surface quality.

Smooth side without vacuum processing

Rough side due to vacuum processing

The two sides didn't fit very well due to an inconsistent interface. I had to belt-sand the edges and deburr the groove to make them fit. I also repeatedly coupled and decoupled the two parts until they finally fit smoothly.

How hard should I shake to reach 99% the speed of light?

V6-V7: Details

During these final iterations, I made the following improvements based on everything observed in previous versions:

Add chamfer to mother mold for easier demolding
Increase silicone base thickness to prevent tearing
Add fillets to interface for easier assembly
Reduced coupling surface for easier assembly
Use the highest quality PLA printer available in the lab

Simplified model with chamfers and fillets

Using the highest level setting, it took 5 hours to print the mold. It was the highest quality print I've ever made in this class.

Incredible surface finish

Demolding and casting was smooth-sailing thanks to all the mistakes and lessons learned in previous iterations. The only issue is that due to the improved print quality, the fit between the two parts became a bit loose. I would have to adjust the model for future versions.

Mother mold, silicone mold, and final cast assembly

I deeply enjoyed this project. It could even become a final project if I add a digital frequency analyzer and laser-cut graphic masks.

Reflections

Sequential Workflows Demand Pipelining

The mother mold → silicone mold → plastic cast workflow is inherently sequential, where any failure causes significant rework. However, by staggering multiple iterations in parallel, I could manage the long iteration cycles effectively and continuously integrate feedback into the next iterations. If I had worked in a strictly single-threaded manner, this project would have taken more than 60 hours.

Emotional Factors in Decision Making

During one 3D print that was near completion, I realized I should have added fillets to the interface geometry. But I couldn't bring myself to stop the job. It felt as if I was terminating a life. I had fallen victim to the sunk cost fallacy as well as anthropomorphizing objects.

Don't Trust Yourself

I mistakenly flipped the positive/negative relationship, even after being fully aware of Neil's warning from the lecture. This taught me that sometimes it's better to have others check your work than to trust your own brain, especially when you've been staring at the same design for hours.

Appendix

Steps to Reproduce FreeCAM CAM Toolpath Issue

Download model.FCStd, open it in FreeCAD (v1.0.2)
Switch to the CAM workbench
Create a CAM Job for the female-part solid, with the following changes from the default
- stock: Create Box
- tool: 1/8 inch Endmill (download toolbit.zip and add 1/8" endmill to job)
Select Model-female-part.Face1 (the top facing surface of the outer-most gutter ring on the solid) and create 3D pocket toolpath, with the following changes from default
- Operation/Pattern: Offset
- Min Travel: checked
- Use Rest Machining: checked
Apply and visualize the toolpath

Expected: progressively deepening circular motion along the outer-most ring

Actual:

Missing toolpaths on half of the ring
Excessive vertical movements
Unwanted toolpaths for inner rings

FreeCAD screenshot of the 3 issues

Week 9: Machine Melody

Sun Chuanqi — Mon, 03 Nov 2025 00:00:00 GMT

Group Assignment

This week's theme is output devices. Since my final project involves a speaker, I invited our group to measure the power consumption of a speaker. We characterized the speaker to understand its how power consumption might be influenced. The detailed findings are documented in our group notes.

Ceci explaining multimeter setup

Give AI A Voice

In the previous week, I built a wireless microphone system that streams audio from an ESP32 to my laptop, where OpenAI generates speech responses. However, the generated voice was only playing on my laptop. This week, I wanted to complete the loop by streaming the synthesized voice back to the ESP32, creating a true wireless voice interaction system.

Playing Sine Wave

I started by testing basic audio output using the Arduino Audio Tools example code to play a sine wave through the I2S DAC. This simple test would verify that my speaker hardware and connections were working correctly. The following program generates and plays a simple sine wave tone:

#include "AudioTools.h"

#define I2S_BCLK D0
#define I2S_DOUT D1
#define I2S_LRC  D2
#define I2S_DIN  D10

const int frequency = 440;
const int sampleRate = 44100;

AudioInfo info(sampleRate, 2, 16);
SineWaveGenerator<int16_t> sineWave(4000); // sine wave with max amplitude of 4000
GeneratedSoundStream<int16_t> sound(sineWave); // stream generated from sine wave
I2SStream out;
StreamCopy copier(out, sound); // copies sound into i2s

void setup() {
  Serial.begin(115200);

  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Info);

  Serial.println("Starting I2S...");
  auto config = out.defaultConfig(TX_MODE);
  config.copyFrom(info);
  config.pin_bck = I2S_BCLK;
  config.pin_ws = I2S_LRC;
  config.pin_data = I2S_DIN;
  out.begin(config);

  sineWave.begin(info, frequency);
  Serial.println("Started sine wave playback");
}

void loop() {
  copier.copy();
}

When I ran this code, I heard loud clicking sounds instead of a clean tone. I systematically experimented with different sampling rates to identify the source of the artifacts:

Sample Rate (kHz)	Result
44.1	Pulsing sound artifact
44	Same artifact
40	Reduced artifact
32	No artifact
22	No artifact

I made several key observations during testing. Opening the serial port seemed to correlate positively with noise artifacts. Higher sampling frequencies also correlated with more artifacts, though this could be confounded by the fact that higher sampling rates mean higher data transmission rates.

To identify the root cause, I conducted follow-up experiments:

Hypothesis 1: The microphone on the device was causing interference since it shares BCLK and LRC lines with the speaker.
- Experiment: Unplugging the microphone should reduce or remove noise.
- Result: No effect observed.
Hypothesis 2: Serial port communication was causing interference.
- Experiment: Switching from a data-passing USB-C cable to a power-only cable, and removing all serial communication in the code.
- Result: Noise reduced but not eliminated.

Conclusion: Over-sampling could cause noise artifacts, and serial communication could exacerbate the issue. For clean audio output, I needed to use lower sampling rates and minimize serial communication during playback.

Stream Sine Wave from Computer to Device

With local audio playback working, the next step was to stream audio from my computer to the ESP32. I started with the sample code for receiving audio over UDP. On the server side, I wrote Node.js code to generate and stream a sine wave. This setup would help me verify the network streaming pipeline before adding the complexity of OpenAI voice synthesis.

Client (other logic omitted for brevity)

#include "AudioTools.h"
#include "AudioTools/Communication/UDPStream.h"

// ... pin definitions ...

const char *ssid = "";
const char *password = "";

const int SAMPLE_RATE = 22000;
const int CHANNELS = 1;
const int BITS_PER_SAMPLE = 16;
const int UDP_PORT = 8888;

AudioInfo info(SAMPLE_RATE, CHANNELS, BITS_PER_SAMPLE);
I2SStream i2s;           // I2S output to speaker
UDPStream udp(ssid, password);
StreamCopy copier(i2s, udp, 1024); // copy UDP stream to I2S

void setup() {
  // ... WiFi connection ...

  // Start I2S with custom pinout for speaker output
  auto i2sCfg = i2s.defaultConfig(TX_MODE);
  i2sCfg.copyFrom(info);
  i2sCfg.pin_bck = I2S_BCLK;
  i2sCfg.pin_ws = I2S_LRC;
  i2sCfg.pin_data = I2S_DIN;
  i2sCfg.i2s_format = I2S_STD_FORMAT;
  i2s.begin(i2sCfg);

  // Start UDP receiver
  udp.begin(UDP_PORT);
}

void loop() {
  copier.copy();
}

Server (other logic omitted for brevity)

const SAMPLE_RATE = 22000;
const PACKET_SIZE = 1024; // bytes per UDP packet
const FREQUENCY = 440; // Hz (A4 note)

function startSineWaveStream() {
  let phase = 0;
  const samplesPerPacket = PACKET_SIZE / 2; // 2 bytes per sample (16-bit)
  const phaseIncrement = (2 * Math.PI * FREQUENCY) / SAMPLE_RATE;

  setInterval(
    () => {
      const result = generateSineWaveBuffer(phase, samplesPerPacket, phaseIncrement);
      phase = result.phase;
      sendAudioPacket(result.buffer);
    },
    (samplesPerPacket / SAMPLE_RATE) * 1000
  );
}

function generateSineWaveBuffer(phase, samplesPerPacket, phaseIncrement) {
  const buffer = Buffer.alloc(PACKET_SIZE);

  for (let i = 0; i < samplesPerPacket; i++) {
    const sample = Math.sin(phase) * 0.3; // 30% amplitude to avoid clipping
    const pcm16Value = Math.round(sample * 32767);
    buffer.writeInt16LE(pcm16Value, i * 2);
    phase += phaseIncrement;
    if (phase >= 2 * Math.PI) {
      phase -= 2 * Math.PI;
    }
  }

  return { buffer, phase };
}

When I tested this setup, I noticed clicking sound artifacts again. Since we had already eliminated serial communication as a significant source of noise, networking became the prime suspect. I experimented with different UDP packet sizes to find the sweet spot:

Packet Size (bytes)	Result
128	Continuous clicking
256	Continuous clicking, less frequent
512	A few clicks every second
1024	No clicks

Conclusion: UDP buffer size directly affects noise artifacts. Larger buffer sizes reduce artifacts significantly.

As a side note, the continous clicking reminds me of a Geiger counter. Idea for a future project!

Upon reflection, I realized I only adjusted the UDP buffer size. There are additional parameters I could experiment with in future iterations:

I2S buffer size (defaults to 6)
I2S buffer count (defaults to 512)

The default parameters are defined in AudioToolsConfig.h. It's interesting that noise occurs when the UDP packet size is smaller than or equal to the I2S buffer size. This relationship warrants further investigation given more time.

Transcribing Audio and Streaming Voice from Computer to Device

With the basic UDP streaming working, I moved on to integrating the complete voice interaction loop. I updated the client to handle push-to-talk button functionality. This logic is similar to what I implemented in the previous week for audio input, but now it also handles audio output for playback.

Voice interaction demo

Client (other logic omitted for brevity)

I2SStream i2sMic;
I2SStream i2sSpeaker;
UDPStream udpSend(WIFI_SSID, WIFI_PASSWORD);
UDPStream udpReceive(WIFI_SSID, WIFI_PASSWORD);

StreamCopy transmitCopier(throttle, i2sMic);
StreamCopy receiveCopier(i2sSpeaker, udpReceive, 1024);

int debounceCounter = 0;
bool isTransmitting = false;

void loop() {
  bool buttonPressed = (digitalRead(BTN_PTT1) == LOW || digitalRead(BTN_PTT2) == LOW);

  if (buttonPressed) {
    debounceCounter++;
    if (debounceCounter >= DEBOUNCE_THRESHOLD) {
      isTransmitting = true;
    }
  } else {
    debounceCounter--;
    if (debounceCounter <= -DEBOUNCE_THRESHOLD) {
      isTransmitting = false;
    }
  }

  if (isTransmitting) {
    transmitCopier.copy();
  }

  receiveCopier.copy();
}

On the server side, I implemented the logic to handle OpenAI's real-time API for transcription and response generation, then stream the synthesized speech back to the ESP32.

Server (other logic omitted for brevity):

const STATE = {
  SILENT: "silent",
  SPEAKING: "speaking",
};

let currentState = STATE.SILENT;
let audioBuffer = [];
let lastPacketTime = null;

function detectSilence() {
  if (currentState === STATE.SPEAKING && lastPacketTime) {
    const timeSinceLastPacket = Date.now() - lastPacketTime;
    if (timeSinceLastPacket > SILENCE_TIMEOUT_MS) {
      transitionToSilentAndProcessAudio();
    }
  }
}

function streamAudioChunkToRealtime(audioChunk) {
  const base64Audio = audioChunk.toString("base64");
  const event = {
    type: "input_audio_buffer.append",
    audio: base64Audio,
  };
  realtimeWs.send(JSON.stringify(event));
}

async function commitAudioAndRequestResponse() {
  realtimeWs.send(JSON.stringify({ type: "input_audio_buffer.commit" }));
  realtimeWs.send(JSON.stringify({ type: "response.create", response: { modalities: ["text"] } }));
  realtimeWs.send(JSON.stringify({ type: "input_audio_buffer.clear" }));
}

// Convert TTS to PCM and stream to ESP32
async function synthesizeAndStreamSpeech(text) {
  const response = await fetch("https://api.openai.com/v1/audio/speech", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "gpt-4o-mini-tts",
      voice: "ash",
      input: text,
      instructions: "Low coarse seasoned veteran from war time, military radio operator voice with no emotion. Speak fast with urgency.",
      response_format: "wav",
    }),
  });

  const wavBuffer = Buffer.from(await response.arrayBuffer());
  const pcmBuffer = await convertWavToPCM16(wavBuffer);
  await streamAudioToUDP(pcmBuffer);
}

// Send audio packets with timing control
async function streamAudioToUDP(pcmBuffer) {
  const totalPackets = Math.ceil(pcmBuffer.length / PACKET_SIZE);

  for (let i = 0; i < totalPackets; i++) {
    const packet = pcmBuffer.slice(i * PACKET_SIZE, (i + 1) * PACKET_SIZE);
    await sendAudioPacketToESP32(packet);

    // Wait to match playback speed
    const delayMs = (PACKET_SIZE / 2 / SAMPLE_RATE) * 1000;
    await sleep(delayMs);
  }
}

This completes the full voice interaction loop. The system can now receive spoken input from the ESP32, process it with OpenAI's API, and stream the synthesized voice response back to play through the speaker.

Sonic Fidget Spinner

I've always been fascinated by Musical Roads that let drivers play music by driving over specially designed rumble strips. There are many demos like this one on the internet showing how the vibrations create melodies. I wanted to build a miniature version controlled by a rotary encoder, offering a similar tactile experience. This would be something I could fidget with during boring Zoom calls.

Rotary Encoder

I received feedback that I did too much networking work during Input Device week, so this was an opportunity to add more input device exploration. I picked up a rotary encoder from my lab's spare parts bin and started studying its mechanism by watching a YouTube tutorial.

The rotary encoder works by generating signals as you rotate the shaft. You can determine both the direction and speed of rotation. I implemented a basic test program using the Rotary Encoder library. My version is simplified from the full example code:

#include "RotaryEncoder.h"

// Define the pins connected to the encoder
#define PIN_ENCODER_A D6
#define PIN_ENCODER_B D7

RotaryEncoder encoder(PIN_ENCODER_A, PIN_ENCODER_B);

void checkPosition() {
  encoder.tick(); // Call tick() to check the state
}

void setup() {
  Serial.begin(115200);
  Serial.println("Rotary Encoder Example");

  // Attach interrupts for encoder pins
  attachInterrupt(digitalPinToInterrupt(PIN_ENCODER_A), checkPosition, CHANGE);
  attachInterrupt(digitalPinToInterrupt(PIN_ENCODER_B), checkPosition, CHANGE);
}

void loop() {
  // Read encoder position
  int newPosition = encoder.getPosition();
  static int lastPosition = 0;

  if (newPosition != lastPosition) {
    Serial.print("Encoder Position: ");
    Serial.println(newPosition);
    lastPosition = newPosition;
  }

  delay(10); // Debounce or prevent overwhelming serial
}

This simple program confirmed that the encoder was working correctly and gave me real-time position feedback through the serial monitor.

Trigger Mechanism

With the encoder working, I needed to decide how to map rotation to musical notes. I considered two approaches:

Approach	Pros	Cons
Note per click	Simple implementation, precise triggering	No control over note duration
Consecutive clicks to start and stop a note	More control over note duration	More complex, especially timing logic

I chose the second approach because it would allow for more expressive playing. With debouncing logic, I could track when a group of consecutive clicks starts and stops:

// ... setup and encoder initialization ...

int counter = 0;
unsigned long lastChangeTime = 0;
bool isChanging = false;
bool groupStarted = false;

void loop() {
  int newPosition = encoder.getPosition();
  static int lastPosition = 0;

  // Detect position change
  if (newPosition != lastPosition) {
    lastPosition = newPosition;
    lastChangeTime = millis();
    isChanging = true;
    if (!groupStarted) {
      Serial.println("Position change group started");
      groupStarted = true;
    }
  }

  // Detect end of group (100ms timeout)
  if (isChanging && (millis() - lastChangeTime > 100)) {
    counter++;
    Serial.print("Position change group ended, Counter: ");
    Serial.println(counter);
    isChanging = false;
    groupStarted = false;
  }
}

The 100ms timeout worked well for detecting when the user paused between rotation gestures. Each continuous rotation motion was now being counted as a single unit. Increasing the timeout would cause a lingering effect and makes the device feel lagging.

Audio Synthesis

With the rotation detection working, it was time to add sound. I started with the simple approach of playing the same tone for each note, merging the audio playback code from the walkie-talkie with the encoder logic:

#include "AudioTools.h"

// ... I2S pin definitions ...

const int sampleRate = 22000;
AudioInfo info(sampleRate, 1, 16);
SineWaveGenerator<int16_t> sineWave(16000);
GeneratedSoundStream<int16_t> sound(sineWave);
I2SStream out;
StreamCopy copier(out, sound);
bool playing = false;

void setup() {
  // ... encoder setup ...

  // Setup I2S for audio output
  auto config = out.defaultConfig(TX_MODE);
  config.copyFrom(info);
  config.pin_bck = I2S_BCLK;
  config.pin_ws = I2S_LRC;
  config.pin_data = I2S_DIN;
  out.begin(config);
  sineWave.begin(info, 440);  // 440 Hz (A4 note)
}

void loop() {
  // ... encoder position detection ...

  if (newPosition != lastPosition) {
    // ... debouncing logic ...
    if (!groupStarted) {
      playing = true;  // Start playing when group starts
      groupStarted = true;
    }
  }

  if (isChanging && (millis() - lastChangeTime > 100)) {
    playing = false;  // Stop playing when group ends
    // ... rest of debouncing ...
  }

  // Only copy audio when playing
  if (playing) {
    copier.copy();
  }
}

This worked well as a proof of concept. The speaker would play a tone when I started rotating and stop when I paused. Now it was time to make it musical.

Soundtrack

To create an actual melody, I mapped out "Ode to Joy" to musical notes and their frequencies. I chose this piece because it's simple, recognizable, and works well with discrete note triggers. I added a pointer to track the current position in the melody:

// Note-to-frequency mapping
float getFreq(char note) {
  switch (note) {
    case 'g': return 392.00 / 2;
    case 'C': return 261.63;
    case 'D': return 293.66;
    case 'E': return 329.63;
    case 'F': return 349.23;
    case 'G': return 392.00;
    default: return 0;
  }
}

// Ode to Joy melody
const char song[62] = {
  'E','E','F','G','G','F','E','D','C','C','D','E','E','D','D',
  'E','E','F','G','G','F','E','D','C','C','D','E','D','C','C',
  'D','D','E','C','D','E','F','E','C','D','E','F','E','D','C','D','g',
  'E','E','F','G','G','F','E','D','C','C','D','E','D','C','C'
};

int currentNote = 0;

void loop() {
  // ... encoder position detection ...

  if (newPosition != lastPosition) {
    // ... debouncing logic ...
    if (!groupStarted) {
      playing = true;
      // Play next note in sequence
      char note = song[currentNote % 63];
      float freq = getFreq(note);
      sineWave.setFrequency(freq);
      currentNote++;
      groupStarted = true;
    }
  }

  // ... rest of loop ...
}

Now each rotation gesture would play the next note in "Ode to Joy", creating a tactile musical experience. The device was starting to feel like a real instrument.

A Not-So-Mary B-side

As I tested the device, an idea struck me: what if rotating backwards would play a different melody, like the B-side of a vinyl record? I decided to add "Mary Had a Little Lamb" as the B-side melody. It's an equally joyful tune that would complement "Ode to Joy".

However, implementing bidirectional melodies revealed a problem. The encoder would occasionally bounce with an unexpected double flip: it changes direction for one click, and immediately changes back for the next click. I had to defer the direction detection after debouncing so as to ignore these spurious direction changes.

// Two melodies for bidirectional playback
const char odeToJoy[62] = {
  'E','E','F','G','G','F','E','D','C','C','D','E','E','D','D',
  'E','E','F','G','G','F','E','D','C','C','D','E','D','C','C',
  'D','D','E','C','D','E','F','E','C','D','E','F','E','D','C','D','g',
  'E','E','F','G','G','F','E','D','C','C','D','E','D','C','C'
};

const char littleLamb[26] = {
  'E','D','C','D','E','E','E','D','D','D','E','G','G','E','D','C','D','E','E','E','C','D','D','E','D','C'
};

int currentNote = 0;
int lastDirection = 0;  // Track rotation direction: 1 = forward, -1 = backward

void loop() {
  static int lastPosition = 0;
  int newPosition = encoder.getPosition();

  if (newPosition != lastPosition) {
    // Determine rotation direction
    int dir = (newPosition > lastPosition) ? 1 : -1;
    lastPosition = newPosition;
    lastChangeTime = millis();

    // Detect direction change (only when not currently changing to avoid bounce)
    if (!isChanging && dir != lastDirection && lastDirection != 0) {
      currentNote = 0;  // Reset counter on direction change
      lastDirection = dir;
      Serial.println("Direction changed, resetting counter");
    }

    // Set initial direction
    if (lastDirection == 0) {
      lastDirection = dir;
    }

    // Play appropriate melody based on direction
    if (!isChanging) {
      char note;
      if (lastDirection == 1) {
        note = odeToJoy[currentNote % 62];
      } else if (lastDirection == -1) {
        note = littleLamb[currentNote % 26];
      }

      float freq = getFreq(note);
      sineWave.setFrequency(freq);

      isChanging = true;
      playing = true;
    }
  }

  // Stop playing after debounce timeout
  if (isChanging && (millis() - lastChangeTime > DEBOUNCE_TIME)) {
    isChanging = false;
    playing = false;
    currentNote++;  // Advance to next note
  }

  if (playing) {
    copier.copy();
  }
}

I was able to play both melodies. There were still occasional misfires of direction change, but it was good enough for a one-off performance.

Playing Mary Had A Little Lamb on the B-side

If I had more time, I would focus on single soundtrack but use the change of rotation direction to delimit tones. This would allow user to quickly play two notes without waiting for the debounce timeout.

After Thought

My development board for the XIAO ESP32 was the unsung hero of this week. The breakout board I created in earlier weeks allowed me to explore both the wireless voice interaction system and the sonic fidget spinner without fabricating new PCBs. I was especially glad that I had mapped out all the pins to female headers during the initial design. This was a perfect example of upfront investment in modular design paying dividends later.

Appendix

Week 8: First Contact

Sun Chuanqi — Mon, 27 Oct 2025 00:00:00 GMT

Group Assignment

I joined Typer and Jacqueline to characterize input devices in our lab. We started by probing a photo transistor to understand its behavior. Our key finding was that the resting resistance is quite high, and covering the sensor with a hand has only a modest effect. However, shining a flashlight on the sensor causes a significant drop in resistance, confirming its sensitivity to light.

To gain more hands-on experience, I decided to probe the microphone breakout board I fabricated in Week 6 with Saleae Logic 8. Wiring up the probes proved tricky due to the small soldering joints on the board.

Probe setup

Setup Analyzer for I2S protocol

I couldn't find a clock signal generator in the lab, so I programmed the Xiao to initialize the I2S device and generate the necessary clock signals for testing.

This is the minimum code to drive the I2S device:

// Simple I2S microphone CSV output
// Custom pinout: D0 = BCLK, D1 = DOUT, D2 = LRC

#include "AudioTools.h"

AudioInfo info(16000, 1, 16);  // 16kHz, mono, 16-bit
I2SStream i2sStream;
CsvOutput<int16_t> csvOutput(Serial);
StreamCopy copier(csvOutput, i2sStream);

void setup() {
  Serial.begin(115200);
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Info);

  // Configure I2S for custom pinout
  auto cfg = i2sStream.defaultConfig(RX_MODE);
  cfg.copyFrom(info);
  cfg.pin_bck = D0;   // BCLK
  cfg.pin_data = D1;  // DOUT
  cfg.pin_ws = D2;    // LRC
  cfg.i2s_format = I2S_STD_FORMAT;

  i2sStream.begin(cfg);
  csvOutput.begin(info);
}

void loop() {
  copier.copy();
}

I confirmed from the serial plotter that the device was outputting data. The software supports both I2S and PCM protocols, but the decoding was not very useful for analysis purposes.

Making a Wireless Microphone

This week is built on top of boards made in week 6. I planned to add speech recognition to the hand-held device with wireless data streaming to my computer for generative AI interaction.

There is a similar project for a wireless microphone from a previous year. The project documentation seemed incomplete but it confirmed my intuition that separating the microphone PCB from the main board would give me the freedom to design the case more ergonomically.

Networking Test with Sine Wave

I started with an off-the-shelf Adafruit ICS-43434 microphone and Phil Schatzmann's examples in the Arduino Audio Tools library. My first goal was to stream a basic sine wave over WiFi. I took Phil's sample code as is and confirmed that sound was working over WiFi. During testing, I found that the antenna seems necessary for a stable connection. I had success in previous tests without using an antenna, but for some reason, the antenna became necessary in this setup.

/**
 * @file streams-generator-server_wav.ino
 *
 * See: https://github.com/pschatzmann/arduino-audio-tools/blob/main/examples/examples-communication/http-server/streams-generator-webserver_wav/streams-generator-webserver_wav.ino
 * This sketch generates a test sine wave. The result is provided as WAV stream which can be listened to in a Web Browser
 *
 * @author Phil Schatzmann
 * @copyright GPLv3
 *
 */

#include "AudioTools.h"
#include "AudioTools/Communication/AudioHttp.h"

// WIFI
const char *ssid = "REPLACE_WITH_SSID";
const char *password = "REPLACE_WITH_REAL_PASSWORD";

AudioWAVServer server(ssid, password);

// Sound Generation
const int sample_rate = 10000;
const int channels = 1;

SineWaveGenerator<int16_t> sineWave;            // Subclass of SoundGenerator with max amplitude of 32000
GeneratedSoundStream<int16_t> in(sineWave);     // Stream generated from sine wave


void setup() {
  Serial.begin(115200);;;
  AudioLogger::instance().begin(Serial,AudioLogger::Info);

  // start server
  server.begin(in, sample_rate, channels);

  // start generation of sound
  sineWave.begin(channels, sample_rate, N_B4);
  in.begin();

  Serial.print("Will sleep");
  // sleep for 5 seconds first
  delay(5000);

  Serial.print("Server URL: http://");
  Serial.print(WiFi.localIP());
}


// copy the data
void loop() {
  server.copy();
}

HTTP Streaming

In this setup, the microcontroller works as the server, exposing a WAV stream. My computer works as the client, receiving and playing the WAV stream. I connected the I2S microphone to the ESP32 and configured it to stream audio data over HTTP.

#include "AudioTools.h"
#include "AudioTools/Communication/AudioHttp.h"

// WiFi credentials
const char *ssid = "REPLACE_WITH_SSID";
const char *password = "REPLACE_WITH_PASSWORD";

// I2S and Audio
AudioInfo info(22000, 1, 16);  // 22kHz, mono, 16-bit
I2SStream i2sStream;           // Access I2S as stream
ConverterFillLeftAndRight<int16_t> filler(LeftIsEmpty);
AudioWAVServer server(ssid, password);

void setup() {
  Serial.begin(115200);
  delay(100);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  // Connect to WiFi
  Serial.println("\nConnecting to WiFi...");
  WiFi.begin(ssid, password);

  int attempts = 0;
  while (WiFi.status() != WL_CONNECTED && attempts < 20) {
    delay(500);
    Serial.print(".");
    attempts++;
  }

  if (WiFi.status() != WL_CONNECTED) {
    Serial.println("\nFailed to connect to WiFi");
    return;
  }

  Serial.println("\nWiFi connected!");
  Serial.print("Device IP: ");
  Serial.println(WiFi.localIP());

  Serial.println("Starting I2S...");
  auto cfg = i2sStream.defaultConfig(RX_MODE);
  cfg.copyFrom(info);
  cfg.pin_bck = D0;   // BCLK
  cfg.pin_data = D1;  // DOUT
  cfg.pin_ws = D2;    // LRC
  cfg.i2s_format = I2S_STD_FORMAT;

  if (!i2sStream.begin(cfg)) {
    Serial.println("Failed to initialize I2S");
    return;
  }
  Serial.println("I2S initialized successfully");

  // Start WAV server
  Serial.println("Starting WAV server...");
  server.begin(i2sStream, info, &filler);
  Serial.print("Server URL: http://");
  Serial.print(WiFi.localIP());
}

void loop() {
  server.copy();
}

After testing this approach, I observed several issues. The latency was inconsistent, ranging from 1 second to 5 seconds. The sound quality was also inconsistent, sometimes good, sometimes poor. My suspicion is that the network condition affected the streaming performance.

Exploring MP3 Encoding

I wanted to test MP3 encoding to reduce bandwidth usage and potentially improve robustness against network issues. The theory was that compressed audio would be more resilient to network fluctuations.

#include "AudioTools.h"
#include "AudioTools/AudioCodecs/CodecMP3LAME.h"
#include "AudioTools/Communication/AudioHttp.h"

// WiFi credentials
const char *ssid = "REPLACE_WITH_SSID";
const char *password = "REPLACE_WITH_PASSWORD";

// I2S and Audio
AudioInfo info(16000, 1, 16);  // 16kHz, mono, 16-bit
I2SStream i2sStream;           // Access I2S as stream
MP3EncoderLAME mp3;
AudioEncoderServer server(&mp3, ssid, password);

void setup() {
  Serial.begin(115200);
  delay(100);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  // Connect to WiFi
  Serial.println("\nConnecting to WiFi...");
  WiFi.begin(ssid, password);

  int attempts = 0;
  while (WiFi.status() != WL_CONNECTED && attempts < 20) {
    delay(500);
    Serial.print(".");
    attempts++;
  }

  if (WiFi.status() != WL_CONNECTED) {
    Serial.println("\nFailed to connect to WiFi");
    return;
  }

  Serial.println("\nWiFi connected!");
  Serial.print("Device IP: ");
  Serial.println(WiFi.localIP());

  // Configure I2S with custom pinout
  Serial.println("Starting I2S...");
  auto cfg = i2sStream.defaultConfig(RX_MODE);
  cfg.copyFrom(info);
  cfg.pin_bck = D0;   // BCLK
  cfg.pin_data = D1;  // DOUT
  cfg.pin_ws = D2;    // LRC
  cfg.i2s_format = I2S_STD_FORMAT;

  if (!i2sStream.begin(cfg)) {
    Serial.println("Failed to initialize I2S");
    return;
  }
  Serial.println("I2S initialized successfully");

  // Start MP3 server
  Serial.println("Starting MP3 server...");
  server.begin(i2sStream, info);
  Serial.print("Server URL: http://");
  Serial.print(WiFi.localIP());
}

void loop() {
  server.doLoop();
}

Running this code caused a memory allocation error:

[Error] lame.c : 2792 - calloc(1,85840) -> 0x0
available MALLOC_CAP_8BIT: 114676 / MALLOC_CAP_32BIT: 114676  / MALLOC_CAP_SPIRAM: 0

It turns out MP3 encoding is quite memory intensive. While it may reduce network bandwidth usage, it does not fit the memory constraints of the ESP32-C3. This was a dead end for optimization.

UDP Streaming Experiment

If I couldn't address the audio codec limitation, could I tackle the network issue itself? I knew that the HTTP protocol has built-in error correction and guarantees delivery through two-way communication, which may introduce significant overhead. UDP is a connectionless protocol that does not guarantee delivery, but it has lower latency and overhead. I don't worry about occasional packet loss in audio streaming, as it often goes unnoticed by the human ear. I decided to go back to basics and test whether I could stream a sine wave over UDP.


#include "AudioTools.h"
#include "AudioTools/Communication/UDPStream.h"


// WiFi credentials
const char *ssid = "REPLACE_WITH_SSID";
const char *password = "REPLACE_WITH_PASSWORD";


AudioInfo info(22000, 1, 16);
SineWaveGenerator<int16_t> sineWave(32000);
GeneratedSoundStream<int16_t> sound(sineWave);
UDPStream udp(ssid, password);
Throttle throttle(udp);
IPAddress udpAddress(192, 168, 41, 106);
const int udpPort = 8888;
StreamCopy copier(throttle, sound);

void setup() {
  Serial.begin(115200);
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Info);

  sineWave.begin(info, N_B4);

  // Define udp address and port
  udp.begin(udpAddress, udpPort);

  auto cfg = throttle.defaultConfig();
  cfg.copyFrom(info);
  throttle.begin(cfg);

  Serial.println("started...");
  Serial.print("Device IP: ");
  Serial.println(WiFi.localIP());
  Serial.print("Sending to: ");
  Serial.print(udpAddress);
  Serial.print(":");
  Serial.println(udpPort);
}

void loop() {
  copier.copy();
}

The challenge with UDP is that it cannot be directly played in a browser like HTTP streams. I needed to implement a simple UDP client. Having worked with JavaScript extensively, I chose to use Node.js for the client implementation. This program receives audio data over UDP and pipes it directly into FFmpeg for playback.

const dgram = require("dgram");
const { spawn } = require("child_process");

const SAMPLE_RATE = 22000;
const CHANNELS = 1;
const BITS_PER_SAMPLE = 16;

const UDP_PORT = 8888;

const server = dgram.createSocket("udp4");

// FFmpeg process to play audio
let ffmpegPlayer = null;

// Statistics
let packetsReceived = 0;
let bytesReceived = 0;
let lastStatsTime = Date.now();

function startAudioPlayer() {
  console.log("Starting audio player...");

  // Use ffmpeg to play raw PCM audio
  ffmpegPlayer = spawn("ffmpeg", [
    "-f",
    "s16le", // signed 16-bit little-endian
    "-ar",
    SAMPLE_RATE.toString(), // sample rate
    "-ac",
    CHANNELS.toString(), // channels
    "-i",
    "pipe:0", // input from stdin
    "-f",
    "alsa", // Use ALSA for Linux audio output
    "default", // Default audio output device
  ]);

  ffmpegPlayer.stderr.on("data", (data) => {
    // ffmpeg outputs info to stderr, only log errors
    const message = data.toString();
    if (message.includes("error") || message.includes("Error")) {
      console.error("FFmpeg error:", message);
    }
  });

  ffmpegPlayer.on("error", (err) => {
    console.error("FFmpeg process error:", err);
  });

  ffmpegPlayer.on("close", (code) => {
    console.log(`FFmpeg process exited with code ${code}`);
  });

  console.log("✓ Audio player started");
}

// Handle incoming UDP packets
server.on("message", (msg, rinfo) => {
  if (!ffmpegPlayer) {
    startAudioPlayer();
  }

  // Write audio data to ffmpeg stdin
  if (ffmpegPlayer && !ffmpegPlayer.killed) {
    ffmpegPlayer.stdin.write(msg);
  }

  // Update statistics
  packetsReceived++;
  bytesReceived += msg.length;

  // Log statistics every 5 seconds
  const now = Date.now();
  if (now - lastStatsTime > 5000) {
    const elapsed = (now - lastStatsTime) / 1000;
    const packetsPerSec = (packetsReceived / elapsed).toFixed(1);
    const kbytesPerSec = (bytesReceived / elapsed / 1024).toFixed(2);

    console.log(`Stats: ${packetsPerSec} packets/s, ${kbytesPerSec} KB/s from ${rinfo.address}:${rinfo.port}`);

    packetsReceived = 0;
    bytesReceived = 0;
    lastStatsTime = now;
  }
});

server.on("error", (err) => {
  console.error(`Server error:\n${err.stack}`);
  server.close();
});

server.on("listening", () => {
  const address = server.address();
  const interfaces = require("os").networkInterfaces();
  const addresses = [];

  for (const name of Object.keys(interfaces)) {
    for (const iface of interfaces[name]) {
      if (iface.family === "IPv4" && !iface.internal) {
        addresses.push(iface.address);
      }
    }
  }

  console.log("\n==============================================");
  console.log("ESP32 Audio UDP Receiver");
  console.log("==============================================");
  console.log(`UDP Server listening on port ${address.port}`);
  console.log(`Sample Rate: ${SAMPLE_RATE} Hz`);
  console.log(`Channels: ${CHANNELS} (mono)`);
  console.log(`Bits per sample: ${BITS_PER_SAMPLE}`);
  console.log("\nListening on:");
  console.log(`  Local:   ${address.address}:${address.port}`);
  addresses.forEach((addr) => {
    console.log(`  Network: ${addr}:${address.port}`);
  });
  console.log("\nWaiting for ESP32 to send audio...");
  console.log("==============================================\n");
});

// Start UDP server
server.bind(UDP_PORT);

// Graceful shutdown
process.on("SIGINT", () => {
  console.log("\n\nShutting down...");

  if (ffmpegPlayer && !ffmpegPlayer.killed) {
    ffmpegPlayer.stdin.end();
    ffmpegPlayer.kill("SIGTERM");
  }

  server.close(() => {
    console.log("Server closed");
    process.exit(0);
  });
});

I confirmed the sine wave was working with this setup. The UDP approach showed promise for reducing latency.

UDP Microphone Streaming

Next, I modified the microcontroller code to stream I2S data over UDP instead of generating a sine wave. UDP does not have flow control, so I needed to implement the throttle pattern recommended in the AudioTools example to prevent overwhelming the network.

#include "AudioTools.h"
#include "AudioTools/Communication/UDPStream.h"


// WiFi credentials
const char *ssid = "";
const char *password = "";

AudioInfo info(22000, 1, 16);  // 22kHz, mono, 16-bit
I2SStream i2sStream;           // Access I2S as stream
ConverterFillLeftAndRight<int16_t> filler(LeftIsEmpty); // fill both channels
UDPStream udp(ssid, password);
Throttle throttle(udp);
IPAddress udpAddress(192, 168, 41, 106);  // Broadcast address
const int udpPort = 8888;
StreamCopy copier(throttle, i2sStream);  // copies I2S microphone input into UDP

void setup() {
  Serial.begin(115200);
  delay(100);
  AudioToolsLogger.begin(Serial, AudioToolsLogLevel::Info);

  // Connect to WiFi
  Serial.println("\nConnecting to WiFi...");
  WiFi.begin(ssid, password);

  int attempts = 0;
  while (WiFi.status() != WL_CONNECTED && attempts < 20) {
    delay(500);
    Serial.print(".");
    attempts++;
  }

  if (WiFi.status() != WL_CONNECTED) {
    Serial.println("\nFailed to connect to WiFi");
    return;
  }

  Serial.println("\nWiFi connected!");
  Serial.print("Device IP: ");
  Serial.println(WiFi.localIP());

  Serial.println("Starting I2S...");
  auto i2sCfg = i2sStream.defaultConfig(RX_MODE);
  i2sCfg.copyFrom(info);
  i2sCfg.pin_bck = D0;   // BCLK
  i2sCfg.pin_data = D1;  // DOUT
  i2sCfg.pin_ws = D2;    // LRC
  i2sCfg.i2s_format = I2S_STD_FORMAT;

  if (!i2sStream.begin(i2sCfg)) {
    Serial.println("Failed to initialize I2S");
    return;
  }
  Serial.println("I2S initialized successfully");

  // Define udp address and port
  udp.begin(udpAddress, udpPort);

  auto throttleCfg = throttle.defaultConfig();
  throttleCfg.copyFrom(info);
  throttle.begin(throttleCfg);

  Serial.println("Started streaming...");
  Serial.print("Sending to: ");
  Serial.print(udpAddress);
  Serial.print(":");
  Serial.println(udpPort);
}

void loop() {
  copier.copy();
}

The Node.js client remained unchanged from the sine wave test, which allowed me to evaluate the latency and compare it with the HTTP protocol version. I was able to achieve much more consistent latency with 1-2 seconds delay. On special occasions, I achieved near real-time performance, but it was unclear how to reproduce it consistently. I suspect it still depends on network conditions.

Latency test with UDP

OpenAI Transcription

Next, I modified the Node.js code to transcribe the audio input using OpenAI's Whisper API. The FFmpeg playback was kept for debugging purposes and also provided an audible reference on where the latency occurs—whether it's before or during the transcription.

The key design decision was to use multi-part form data to stream audio chunks as soon as they are available. Since we don't have a way to delimit the audio stream automatically, I used an arbitrary 5-second interval to send chunks. I verified that OpenAI was able to respond with the transcribed text.

const dgram = require("dgram");
const { spawn } = require("child_process");

const SAMPLE_RATE = 22000;
const CHANNELS = 1;
const BITS_PER_SAMPLE = 16;
const UDP_PORT = 8888;

const OPENAI_API_KEY = process.env.OPENAI_API_KEY; // Make you set the environment variable with your own key
const TRANSCRIPTION_INTERVAL = 5000; // ms

const server = dgram.createSocket("udp4");

let ffmpegPlayer = null;
let audioBuffer = [];
let lastTranscriptionTime = Date.now();
let isTranscribing = false;
let packetsReceived = 0;
let bytesReceived = 0;
let lastStatsTime = Date.now();

function startAudioPlayer() {
  console.log("Starting audio player...");

  ffmpegPlayer = spawn("ffmpeg", ["-f", "s16le", "-ar", SAMPLE_RATE.toString(), "-ac", CHANNELS.toString(), "-i", "pipe:0", "-f", "alsa", "default"]);

  ffmpegPlayer.stderr.on("data", (data) => {
    const message = data.toString();
    if (message.includes("error") || message.includes("Error")) {
      console.error("FFmpeg error:", message);
    }
  });

  ffmpegPlayer.on("error", (err) => {
    console.error("FFmpeg process error:", err);
  });

  ffmpegPlayer.on("close", (code) => {
    console.log(`FFmpeg process exited with code ${code}`);
  });

  console.log("✓ Audio player started");
}

async function createWavFromPCM(pcmData) {
  const dataSize = pcmData.length;
  const fileSize = 44 + dataSize;
  const header = Buffer.alloc(44);

  header.write("RIFF", 0);
  header.writeUInt32LE(fileSize - 8, 4);
  header.write("WAVE", 8);

  header.write("fmt ", 12);
  header.writeUInt32LE(16, 16); // fmt chunk size
  header.writeUInt16LE(1, 20); // audio format (1 = PCM)
  header.writeUInt16LE(CHANNELS, 22);
  header.writeUInt32LE(SAMPLE_RATE, 24);
  header.writeUInt32LE((SAMPLE_RATE * CHANNELS * BITS_PER_SAMPLE) / 8, 28); // byte rate
  header.writeUInt16LE((CHANNELS * BITS_PER_SAMPLE) / 8, 32); // block align
  header.writeUInt16LE(BITS_PER_SAMPLE, 34);

  // data chunk
  header.write("data", 36);
  header.writeUInt32LE(dataSize, 40);

  return Buffer.concat([header, pcmData]);
}

async function transcribeAudio(audioData) {
  if (!OPENAI_API_KEY) {
    console.error("⚠️  OPENAI_API_KEY not set. Skipping transcription.");
    return;
  }

  if (audioData.length === 0) {
    console.log("⚠️  No audio data to transcribe");
    return;
  }

  console.log(`🎤 Transcribing ${(audioData.length / 1024).toFixed(2)} KB of audio...`);

  try {
    // Convert PCM to WAV
    const wavData = await createWavFromPCM(audioData);

    // Build multipart/form-data with boundary (similar to web implementation)
    const boundary = "----WebKitFormBoundary" + Math.random().toString(36).slice(2);
    const CRLF = "\r\n";

    // Build the multipart form data manually
    const preamble =
      `--${boundary}${CRLF}` +
      `Content-Disposition: form-data; name="model"${CRLF}${CRLF}` +
      `whisper-1${CRLF}` +
      `--${boundary}${CRLF}` +
      `Content-Disposition: form-data; name="language"${CRLF}${CRLF}` +
      `en${CRLF}` +
      `--${boundary}${CRLF}` +
      `Content-Disposition: form-data; name="file"; filename="audio.wav"${CRLF}` +
      `Content-Type: audio/wav${CRLF}${CRLF}`;

    const epilogue = `${CRLF}--${boundary}--${CRLF}`;

    // Create a ReadableStream from the data
    const { Readable } = require("stream");
    const bodyStream = Readable.from(
      (async function* () {
        yield Buffer.from(preamble, "utf-8");
        yield wavData;
        yield Buffer.from(epilogue, "utf-8");
      })()
    );

    // Make request using fetch
    const response = await fetch("https://api.openai.com/v1/audio/transcriptions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${OPENAI_API_KEY}`,
        "Content-Type": `multipart/form-data; boundary=${boundary}`,
      },
      body: bodyStream,
      duplex: "half",
    });

    if (!response.ok) {
      const errorText = await response.text();
      throw new Error(`HTTP ${response.status}: ${errorText}`);
    }

    const result = await response.json();
    console.log(`\n📝 Transcription: "${result.text}"\n`);
  } catch (error) {
    console.error("❌ Transcription error:", error.message);
  }
}

async function processTranscriptionQueue() {
  if (isTranscribing || audioBuffer.length === 0) {
    return;
  }

  const now = Date.now();
  if (now - lastTranscriptionTime < TRANSCRIPTION_INTERVAL) {
    return;
  }

  isTranscribing = true;
  lastTranscriptionTime = now;

  // Get accumulated audio data
  const audioData = Buffer.concat(audioBuffer);
  audioBuffer = [];

  // Transcribe in background
  transcribeAudio(audioData).finally(() => {
    isTranscribing = false;
  });
}

// Handle incoming UDP packets
server.on("message", (msg, rinfo) => {
  if (!ffmpegPlayer) {
    startAudioPlayer();
  }

  // Write audio data to ffmpeg stdin
  if (ffmpegPlayer && !ffmpegPlayer.killed) {
    ffmpegPlayer.stdin.write(msg);
  }

  // Add to transcription buffer
  audioBuffer.push(Buffer.from(msg));

  // Update statistics
  packetsReceived++;
  bytesReceived += msg.length;

  // Check if it's time to transcribe
  processTranscriptionQueue();

  // Log statistics every 5 seconds
  const now = Date.now();
  if (now - lastStatsTime > 5000) {
    const elapsed = (now - lastStatsTime) / 1000;
    const packetsPerSec = (packetsReceived / elapsed).toFixed(1);
    const kbytesPerSec = (bytesReceived / elapsed / 1024).toFixed(2);
    const bufferSize = (audioBuffer.reduce((sum, buf) => sum + buf.length, 0) / 1024).toFixed(2);

    console.log(`📊 Stats: ${packetsPerSec} packets/s, ${kbytesPerSec} KB/s, buffer: ${bufferSize} KB`);

    packetsReceived = 0;
    bytesReceived = 0;
    lastStatsTime = now;
  }
});

server.on("error", (err) => {
  console.error(`Server error:\n${err.stack}`);
  server.close();
});

server.on("listening", () => {
  const address = server.address();
  const interfaces = require("os").networkInterfaces();
  const addresses = [];

  for (const name of Object.keys(interfaces)) {
    for (const iface of interfaces[name]) {
      if (iface.family === "IPv4" && !iface.internal) {
        addresses.push(iface.address);
      }
    }
  }

  console.log("\n==============================================");
  console.log("ESP32 Audio UDP Receiver with Transcription");
  console.log("==============================================");
  console.log(`UDP Server listening on port ${address.port}`);
  console.log(`Sample Rate: ${SAMPLE_RATE} Hz`);
  console.log(`Channels: ${CHANNELS} (mono)`);
  console.log(`Bits per sample: ${BITS_PER_SAMPLE}`);
  console.log(`Transcription interval: ${TRANSCRIPTION_INTERVAL / 1000}s`);
  console.log(`OpenAI API Key: ${OPENAI_API_KEY ? "✓ Set" : "✗ Not set"}`);
  console.log("\nListening on:");
  console.log(`  Local:   ${address.address}:${address.port}`);
  addresses.forEach((addr) => {
    console.log(`  Network: ${addr}:${address.port}`);
  });
  console.log("\nWaiting for ESP32 to send audio...");
  console.log("==============================================\n");
});

// Start UDP server
server.bind(UDP_PORT);

// Graceful shutdown
process.on("SIGINT", () => {
  console.log("\n\nShutting down...");

  if (ffmpegPlayer && !ffmpegPlayer.killed) {
    ffmpegPlayer.stdin.end();
    ffmpegPlayer.kill("SIGTERM");
  }

  server.close(() => {
    console.log("Server closed");
    process.exit(0);
  });
});

Push to Talk

The 5-second interval auto-send was a temporary solution for testing transcription. For a more practical implementation, I used one of the buttons on my Operator Board to signal the beginning and ending of speech. When the button is pressed, the microcontroller starts recording audio. When the button is released, it stops recording and sends the audio for transcription. This push-to-talk approach is similar to how walkie-talkies work.

I found this button in my lab

Here are the key sections in the code related to the push-to-talk functionality.

// ...

// Debounce settings
const int DEBOUNCE_THRESHOLD = 5;
int buttonCounter = 0;
bool buttonState = HIGH;
bool lastButtonState = HIGH;

// ...

void setup() {
  // ...

  // Configure D8 and D9 as input with pull-up resistors
  pinMode(D8, INPUT_PULLUP);
  pinMode(D9, INPUT_PULLUP);

  // ...
}

void loop() {
  // Read combined button state (LOW if either button is pressed)
  int buttonReading = (digitalRead(D8) == LOW || digitalRead(D9) == LOW) ? LOW : HIGH;

  // Debounce combined button
  if (buttonReading == LOW) {
    buttonCounter++;
    if (buttonCounter >= DEBOUNCE_THRESHOLD) {
      buttonState = LOW;
      buttonCounter = DEBOUNCE_THRESHOLD; // Cap the counter
    }
  } else {
    buttonCounter--;
    if (buttonCounter <= -DEBOUNCE_THRESHOLD) {
      buttonState = HIGH;
      buttonCounter = -DEBOUNCE_THRESHOLD; // Cap the counter
    }
  }

  // Log state changes
  if (buttonState != lastButtonState) {
    if (buttonState == LOW) {
      Serial.println("Speaking...");
    } else {
      Serial.println("Sent");
    }
    lastButtonState = buttonState;
  }

  // Transmit audio only if button is pressed
  if (buttonState == LOW) {
    copier.copy();
  }
}

On the Node.js side, I implemented a state machine to handle the audio stream more intelligently. The system starts in a silent state. Upon receiving audio packets, it transitions to a speaking state and begins streaming audio to OpenAI. After detecting sustained silence, it transitions back to the silent state and wraps up the transcription request.

// ...

// State machine
const STATE = {
  SILENT: "silent",
  SPEAKING: "speaking",
};
let currentState = STATE.SILENT;
let audioBuffer = [];
let lastPacketTime = null;
let silenceCheckInterval = null;
let isTranscribing = false;

// ...

function transitionToSpeaking() {
  if (currentState !== STATE.SPEAKING) {
    console.log("🎤 Speaking...");
    currentState = STATE.SPEAKING;
    audioBuffer = [];
  }
}

async function transitionToSilent() {
  if (currentState !== STATE.SILENT) {
    console.log("📤 Sent");
    currentState = STATE.SILENT;

    // Transcribe the accumulated audio
    if (audioBuffer.length > 0 && !isTranscribing) {
      const audioData = Buffer.concat(audioBuffer);
      audioBuffer = [];
      await transcribeAudio(audioData);
    }
  }
}

function checkForSilence() {
  if (currentState === STATE.SPEAKING && lastPacketTime) {
    const timeSinceLastPacket = Date.now() - lastPacketTime;
    if (timeSinceLastPacket > SILENCE_TIMEOUT) {
      transitionToSilent();
    }
  }
}

// ...

// Handle incoming UDP packets
server.on("message", (msg, rinfo) => {
  // ...

  // Transition to speaking state on first packet
  transitionToSpeaking();

  // ...

  // Add to transcription buffer
  audioBuffer.push(Buffer.from(msg));

  // ...
});

// ...

server.on("listening", () => {
  // ...

  // Start silence checker
  silenceCheckInterval = setInterval(checkForSilence, 100);
});

// ...

Realtime Voice Synthesis

In this final version, I jumped ahead slightly and implemented response synthesis. Instead of going through transcription separately, I directly prompt the model with the audio stream and manually trigger a response upon detecting silence. The playback currently happens on the computer and will be streamed to the microcontroller in next week's Output Device assignment.

The microcontroller side required a minor change. According to OpenAI's documentation, the Realtime API requires 24kHz sampling rate.

- AudioInfo info(22000, 1, 16);  // 22kHz, mono, 16-bit
+ AudioInfo info(24000, 1, 16);  // 24kHz, mono, 16-bit

The Node.js server code required a substantial overhaul. The architecture changed from a simple HTTP transcription flow to a WebSocket-based real-time conversation system.

Before:

UDP packet -> FFmpeg -> STT -> TTS -> Audio playback

After:

UDP packet -> OpenAI WebSocket -> TTS -> Audio playback

I engineered a prompt for AI to migrate my previous implementation to generate voice responses. The prompt referenced a markdown file that contains the full content of the Realtime Models Prompting guide and Realtime conversations guide.

const dgram = require("dgram");
const { spawn } = require("child_process");
const WebSocket = require("ws");

const SAMPLE_RATE = 24000;
const CHANNELS = 1;
const BITS_PER_SAMPLE = 16;
const UDP_PORT = 8888;
const SILENCE_TIMEOUT_MS = 1000;
const STATS_INTERVAL_MS = 5000;
const SILENCE_CHECK_INTERVAL_MS = 100;

const STATE = {
  SILENT: "silent",
  SPEAKING: "speaking",
};

const server = dgram.createSocket("udp4");

let currentState = STATE.SILENT;
let audioBuffer = [];
let lastPacketTime = null;
let silenceCheckInterval = null;
let isProcessing = false;
let packetsReceived = 0;
let bytesReceived = 0;
let lastStatsTime = Date.now();
let realtimeWs = null;
let sessionReady = false;

startServer();

function startServer() {
  connectToRealtimeAPI();
  server.bind(UDP_PORT);
  server.on("listening", handleServerListening);
  server.on("message", handleIncomingAudioPacket);
  server.on("error", handleServerError);
  process.on("SIGINT", handleGracefulShutdown);
}

function handleServerListening() {
  const address = server.address();
  logServerStartup(address);
  silenceCheckInterval = setInterval(detectSilence, SILENCE_CHECK_INTERVAL_MS);
}

function handleIncomingAudioPacket(msg, rinfo) {
  beginSpeakingStateIfNeeded();
  lastPacketTime = Date.now();
  audioBuffer.push(Buffer.from(msg));

  // Stream audio to Realtime API immediately if session is ready
  if (sessionReady && realtimeWs && realtimeWs.readyState === WebSocket.OPEN) {
    streamAudioChunk(msg);
  }

  updateStatistics(msg.length);
  logStatisticsIfIntervalElapsed();
}

function handleServerError(err) {
  console.error(`Server error:\n${err.stack}`);
  server.close();
}

function handleGracefulShutdown() {
  console.log("\n\nShutting down...");
  if (silenceCheckInterval) {
    clearInterval(silenceCheckInterval);
  }
  if (realtimeWs) {
    realtimeWs.close();
  }
  server.close(() => {
    console.log("Server closed");
    process.exit(0);
  });
}

function connectToRealtimeAPI() {
  if (!process.env.OPENAI_API_KEY) {
    console.error("⚠️  OPENAI_API_KEY not set. Cannot connect to Realtime API.");
    process.exit(1);
  }

  const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime";
  const headers = {
    Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    "OpenAI-Beta": "realtime=v1",
  };

  console.log("🔌 Connecting to OpenAI Realtime API...");
  realtimeWs = new WebSocket(url, { headers });

  realtimeWs.on("open", handleRealtimeOpen);
  realtimeWs.on("message", handleRealtimeMessage);
  realtimeWs.on("close", handleRealtimeClose);
}

function handleRealtimeOpen() {
  console.log("✓ Connected to Realtime API");
}

function handleRealtimeMessage(data) {
  try {
    const event = JSON.parse(data.toString());

    switch (event.type) {
      case "session.created":
        console.log("✓ Session created");
        configureSession();
        break;

      case "session.updated":
        console.log("✓ Session configured");
        sessionReady = true;
        break;

      case "response.output_text.delta":
        // Text being generated in chunks (optional logging)
        process.stdout.write(event.delta);
        break;

      case "response.done":
        console.log("✓ Response complete");
        handleResponseComplete(event);
        break;
    }
  } catch (error) {
    console.error("❌ Error parsing Realtime message:", error.message);
  }
}

function handleRealtimeClose() {
  console.log("🔌 Realtime connection closed. Reconnecting...");
  sessionReady = false;
  setTimeout(connectToRealtimeAPI, 2000);
}

function configureSession() {
  const sessionConfig = {
    type: "session.update",
    session: {
      modalities: ["text"], // Only text output, no audio
      instructions: "Respond to user speech in the voice of a HAM radio operator. One short spoken phrase response only.",
      voice: "ash",
      input_audio_format: "pcm16",
      output_audio_format: "pcm16",
      turn_detection: null, // Disable VAD - we handle silence detection manually
    },
  };

  realtimeWs.send(JSON.stringify(sessionConfig));
}

function beginSpeakingStateIfNeeded() {
  if (currentState !== STATE.SPEAKING) {
    console.log("🎤 Speaking...");
    currentState = STATE.SPEAKING;
    audioBuffer = [];
  }
}

function detectSilence() {
  if (currentState === STATE.SPEAKING && lastPacketTime) {
    const timeSinceLastPacket = Date.now() - lastPacketTime;
    if (timeSinceLastPacket > SILENCE_TIMEOUT_MS) {
      transitionToSilentAndProcessAudio();
    }
  }
}

async function transitionToSilentAndProcessAudio() {
  if (currentState !== STATE.SILENT) {
    console.log("📤 Sent");
    currentState = STATE.SILENT;

    if (audioBuffer.length > 0 && !isProcessing && sessionReady) {
      isProcessing = true;
      audioBuffer = [];
      await commitAudioAndRequestResponse();
    }
  }
}

function streamAudioChunk(audioChunk) {
  // Convert PCM16 buffer to base64 and send to Realtime API
  const base64Audio = audioChunk.toString("base64");
  const event = {
    type: "input_audio_buffer.append",
    audio: base64Audio,
  };
  realtimeWs.send(JSON.stringify(event));
}

async function commitAudioAndRequestResponse() {
  console.log(`🔄 Committing audio buffer and requesting response...`);

  try {
    realtimeWs.send(JSON.stringify({ type: "input_audio_buffer.commit" }));
    realtimeWs.send(JSON.stringify({ type: "response.create", response: { modalities: ["text"] } }));
    realtimeWs.send(JSON.stringify({ type: "input_audio_buffer.clear" }));
  } catch (error) {
    console.error("❌ Error requesting response:", error.message);
    isProcessing = false;
  }
}

function handleResponseComplete(event) {
  // Extract text from response
  const response = event.response;
  let responseText = "";

  if (response && response.output) {
    for (const item of response.output) {
      if (item.type === "message" && item.content) {
        for (const content of item.content) {
          if (content.type === "text") {
            responseText = content.text;
            break;
          }
        }
      }
      if (responseText) break;
    }
  }

  if (responseText) {
    console.log(`💬 Final response: "${responseText}"`);
    speakTextAloud(responseText);
  } else {
    console.log("⚠️  No text response received");
  }

  isProcessing = false;
}

async function speakTextAloud(text) {
  console.log(`🔊 Playing TTS for: "${text}"`);

  try {
    // Use OpenAI REST API for TTS since Realtime API is text-only mode
    const response = await fetch("https://api.openai.com/v1/audio/speech", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model: "gpt-4o-mini-tts",
        voice: "ash",
        input: text,
        instructions: "Low coarse seasoned veteran from war time, military radio operator voice with no emotion. Speak fast with urgency.",
        response_format: "wav",
      }),
    });

    if (!response.ok) {
      throw new Error(`TTS API error: ${response.status}`);
    }

    const buffer = Buffer.from(await response.arrayBuffer());
    await playAudioBufferThroughSpeakers(buffer);
  } catch (error) {
    console.error("❌ TTS error:", error.message);
  }
}

async function playAudioBufferThroughSpeakers(buffer) {
  const ffplay = spawn("ffplay", ["-nodisp", "-autoexit", "-loglevel", "quiet", "-i", "pipe:0"]);

  ffplay.on("error", (err) => {
    console.error("❌ ffplay error:", err.message);
  });

  ffplay.on("close", (code) => {
    if (code === 0) {
      console.log("✓ TTS playback completed");
    } else {
      console.error(`❌ ffplay exited with code ${code}`);
    }
  });

  ffplay.stdin.write(buffer);
  ffplay.stdin.end();
}

function updateStatistics(messageLength) {
  packetsReceived++;
  bytesReceived += messageLength;
}

function logStatisticsIfIntervalElapsed() {
  const now = Date.now();
  if (now - lastStatsTime > STATS_INTERVAL_MS) {
    const elapsed = (now - lastStatsTime) / 1000;
    const packetsPerSec = (packetsReceived / elapsed).toFixed(1);
    const kbytesPerSec = (bytesReceived / elapsed / 1024).toFixed(2);
    const bufferSize = (audioBuffer.reduce((sum, buf) => sum + buf.length, 0) / 1024).toFixed(2);

    console.log(`📊 Stats: ${packetsPerSec} packets/s, ${kbytesPerSec} KB/s, buffer: ${bufferSize} KB`);

    packetsReceived = 0;
    bytesReceived = 0;
    lastStatsTime = now;
  }
}

function logServerStartup(address) {
  const networkAddresses = getNetworkAddresses();

  console.log(`UDP Server listening on port ${address.port}`);
  console.log("\nListening on:");
  console.log(`  Local:   ${address.address}:${address.port}`);
  networkAddresses.forEach((addr) => {
    console.log(`  Network: ${addr}:${address.port}`);
  });
}

function getNetworkAddresses() {
  const interfaces = require("os").networkInterfaces();
  const addresses = [];

  for (const name of Object.keys(interfaces)) {
    for (const iface of interfaces[name]) {
      if (iface.family === "IPv4" && !iface.internal) {
        addresses.push(iface.address);
      }
    }
  }

  return addresses;
}

The final implementation cuts down response latency to about 3 seconds. The speech synthesis sounds quite natural, creating a convincing conversational experience. This prototype successfully demonstrates the core functionality needed for my final project's speech input.

Wrap up with an immersive demo. USB-C is only for power-supply.

PCB Fabrication Revisited

In addition to the Adafruit ICS-43434 I2S breakout board, I reattempted fabricating my own microphone PCB after burning my first one in week 6.

I redesigned the PCB with several improvements. I rerouted the traces to match the pintout of the Adafruit ICS-43434 breakout board so it can be a drop-in replacement. I switched from Through-Hole to SMD components for easier soldering, planning to bend the legs of the TH pins to make them SMD-like. I also added rounded edges and mounting holes to make the board consistent with other components in my system.

Redesigned microphone PCB

3D view of the redesigned microphone PCB

I decided to try a simpler process for laser cutting the PCB:

First mill the holes and edge cut, with tabs to hold the PCB in place.
Laser cut the traces. Then remove the tabs.

Milling was successful

Unfortunately, the vacuum system for all the laser cutters in the shop was broken. I had to postpone the cutting. I realized my board needs another iteration anyway:

I increased the dimension of the board to fit the M3 mounting holes, but this caused the board to extend beyond the footprint of the main Switchboard. I need to shrink it back.
I found rivets for making vias. I can switch to real PTH mounted headers instead of bending the legs to make them surface mount.

Appedix

Week 7: A Very Dangerous Machine

Sun Chuanqi — Sat, 18 Oct 2025 00:00:00 GMT

This week's schedule was a bit special. Due to limited machine time for the large-format CNC, I was only able to characterize the machine on the same day as my cutting session. This meant I essentially did the group assignment with my TA Dan, as we iterated on our designs together right by the machine.

Project Context

I've been racing mountain bikes since 2015, but I've never quite mastered the technique of manualing. Manualing is the skill of lifting the front wheel off the ground and balancing on the rear wheel while riding, which is crucial for clearing obstacles on the trail as well as for the occasional show-off. A great reference video is How To Manual Like A Pro – MTB Skills. To help riders learn this skill in a controlled environment, people have created "Manual Machines". I decided to build my own using the skills from this week.

Gathering Data

The first step was to get the critical dimensions of my bike. I found a geometry chart on the manufacturer's website, which gave me the wheelbase (distance between the front and rear axles) and tire diameter. I also manually measured the tire width, which came out to be 55 mm, slightly narrower than the 56-57 mm stated in the official spec.

Bike Geometry

Paper Prototyping

I started with a quick hand sketch to get a feel for the form and how the 3D parts would relate to each other.

Sketching the Design

With the practice from Week 2, I was shocked at how quickly I could move from a sketch to a physical prototype. I used the laser cutter to make a small-scale model. After a quick test, I found that 85% power at 70 mm/s speed with a 0.12 mm kerf worked well for the cardboard.

Characterize and Cut

The initial assembly felt wobbly. I realized I could tilt the joints, using gravity to create more stable connections. This small-scale model was invaluable for identifying weak points in the design.

Initial Paper Prototype

I iterated on the fastening mechanism to improve stability. In the final version, I introduced through-holes, slanted joints, and a press-fit "tail bone".

Final Paper Prototype

Scale Up

When I started the paper prototype, I didn't know what material we would be using for the final build. I had assumed 2x4 lumber, but later found out we would be using 4x8 ft sheets of Oriented Strand Board (OSB). This change in material forced me to rethink the design to work with sheet stock.

I scaled up the model in Onshape, using a picture of my bike as an underlay to trace its geometry and ensure a perfect fit.

Designing the Full Scale Model

Special thanks to Dan, who helped me set up the machine and run the job. I did all my CAD work in Onshape, which unfortunately does not have a free CAM solution for students. Dan generously let me use his Fusion 360 license to generate the toolpaths.

During the first CAM simulation, I realized I had misunderstood how dogbones work. I had placed the center of the dogbone circle at the corner of the part, creating a bottleneck that the drill bit couldn't navigate.

Dogbone Incorrect

The key insight was that the goal is to create the smallest possible cut that allows the mating part to fit perfectly, which doesn't require the circle's center to be at the corner.

Identifying the bottleneck

The correct placement of the dogbone is to make the corner coincide with the circle.

Dogbone Corrected

Due to limited machine time, I went with the practical decision of simply increasing my circle's radius until the bottleneck was wide enough.

Parametric Design Saved My Day

Thanks to Parametric Design, I could iterate quickly. This resulted in a 9 mm circle at each right corner angle of my shapes. This approach is less efficient and slightly undermines the structure.

Verifying Toolpath After Dogbone Fix

With the dogbones issue addressed, I generated the toolpaths using 2D Contour cuts:

Job 1: A single pass down to -0.2 inches.
Job 2: A multi-pass cut from -0.2 inches down to -0.45 inches, intended to cut all the way through.

Milling in Progress

However, during the milling, Job 2 didn't cut all the way through the material.

It should have cut through here...

After the job, I manually measured and calculated the remaining thickness, and created a third job to cut through the rest of the material.

Measuring Remaining Thickness

This final pass successfully cut the parts out. Comparing my parameters with others, I realized my mistake: I had set the stock height to 11.2 mm (thinnest measurement) while others used 12 mm (thickest measurement). I believe this discrepancy is why the machine didn't cut through on the first attempt.

Job 3 Milled Through

Parameter	Job 1	Job 2	Job 3
Clearance Height
From	Retract height	Retract height	Retract height
Offset	0.4 in	0.4 in	0.4 in
Retract Height
From	Stock top	Stock top	Stock top
Offset	0.2 in	0.2 in	0.2 in
Feed Height
From	Top height	Top height	Top height
Offset	0.2 in	0.2 in	0.2 in
Top Height
From	Stock top	Stock top	Stock top
Offset	0 in	-0.2 in	-0.45 in
Bottom Height
From	Stock top	Stock top	Stock top
Offset	-0.2 in	-0.45 in	-0.56 in
Multiple Depths
Enabled	No	Yes	No
Maximum Roughing Stepdown	N/A	0.2 in	N/A

Assemble and Enhance

I used tabs to keep the parts in place during milling, and removing them wasn't trivial. I used a Milwaukee oscillating multi-tool with a bi-metal blade to cut the tabs and flatten the edges.

Removing Tabs

A quick safety note: wearing gloves is essential when working with OSB. A wood splinter cut straight through my glove and stabbed my thumb. Had I not been wearing the glove, the injury could have been much worse.

The first assembly on the floor went smoothly. My calculations for the material thickness were perfect, and all the joints had a snug fit without any glue or fasteners.

First Assembly on Floor

However, adding the bike revealed two problems: I had miscalculated the tire diameter, and the middle section felt a bit wobbly.

First Assembly with Bike

The tire fit can be addressed with spacers. The weak middle can be reinforced with brackets. I measured the gap between the tire and the wood to determine the size of the needed spacers.

Measuring Gap

Due to limited machine time, I designed reinforcements that I could cut manually on a band saw.

Handle Calculating the Redesign

I learned how to use the band saw by observing and asking a classmate. Special shout-out to Charlie who happened to be using the band saw in the shop and generously demoed his technique to me. The key is to cut from the side with loose wood chips, so the saw's downward motion holds the OSB against the table and minimizes splintering.

Cutting Reinforcements with Band Saw

I used a Milwaukee Straight Die Grinder to smooth the corners that the band saw left rough.

Detailing with Die Grinder

One of the reinforcements was a bit loose, a result of a slight error in my design. This taught me that it's better to design for a tight fit and then iteratively sand or file it down until it fits perfectly. Adding the spacers and reinforcements significantly improved the structure's rigidity.

Final Assembly with Spacers and Reinforcements (Colored Blue)

For the safety tether, I tied a rope using a flexible knot, anchored by a washer. The rope was from an unknown source and seemed weak. A proper design would use rated cargo tie-down straps with a secure buckle mechanism.

Rope Anchor, with Fraying Core

Testing

With the machine fully assembled, it was time to test. Thank you Quincy and Yuhan for photography, video, and physical support.

Preparation: Helmet and gloves on. Removing seat post for extra clearance.
Static Test: I started by placing one foot on the ground, rotating the bike backward, holding the brake, and mounting. The structure felt solid.

Functional Test

Limit Test: I asked my friend to support me while I rotated as far back as possible to test the strength of the rope tether.

Limit Test

Full Test: Finally, I attempted to bring the bike up from the ground into a manual without any assistance. I noticed that the device was jerking forward each time I brought the bike up. Future improvements should include rubber feet to increase friction with the ground.

Final Test

The machine was a big success, but my demo needs practice. I promptly landed on my butt, learning the lesson the "hard way". Sometimes, the tools you create can be more dangerous than the tools that created them.

Key Learnings

Parametric Design for Joints: Separate parameters for material thickness and slot size. Use the thickest measurement for material, but make slots slightly smaller for press-fit joints.
Measure Physical Objects: Always measure the actual object directly rather than relying on images or specifications to avoid dimensional errors.
Understand Materials Before Design: Survey available materials before starting your design to avoid unnecessary redesigns.
Stay Cautious During Testing: Maintain caution during physical testing, even when things work well. Don't let excitement lead to unnecessary risks.
Use Integrated CAD/CAM Software: Using separate software for CAD (Onshape) and CAM (Fusion 360) added significant overhead in exporting and importing files. An integrated solution would streamline the workflow and reduce potential errors in translation.

Appendix

Week 6: Wish I Were The Steves

Sun Chuanqi — Mon, 13 Oct 2025 00:00:00 GMT

The legend goes that Steve Jobs and Steve Wozniak hand-built the first Apple-1 computers in their garage. This week, I learned how to hand-build something simpler, but in the same spirit: a custom microcontroller board.

Group assignment

Our group assignment was to characterize the design rules for our Carvera PCB milling machine. We started by reading the excellent tutorial by our TA Quentin, then started producing a test PCB to understand the machine's capabilities. Characterizing the Carvera PCB milling machine

Our first run resulted in large amounts of burrs along the edges of the traces. We learned this was due to a damaged endmill and subsequently learned how to replace the bit. The machine is able to consistently handle traces down to 0.2mm. Burrs everywhere

Later, I attended a second session with Kristof where we debugged the milling machine after it entered a halted state. We documented our debugging process and findings in our group note.

Fabricating

For my final project, I designed two main components: a hand-held unit I call "the Operator" and a main body unit called "the Switchboard." Having completed the circuit simulations in the previous week, I already had initial designs for both boards. This week, my focus shifted to the physical milling and fabrication process.

The Operator

The Operator is a dev board for the Xiao-ESP32-C3 that connects a Microphone, DAC/Amp, TRRS jack, and a couple of switches. Milling the first version of the Operator board, a single-sided design, was straightforward.

In KiCad, export the PCB design as Gerber files and drill files. Export Gerber files

Export drill files

Upload both files at the same time to Quentin's gerber2img tool to convert the copper layer, edge cut layer, and drill files into PNG images.

For the exterior cut, I used the "Black and white" setting and checked "Fill edge cut" to create a solid outline including the drill holes.
For the traces, I unchecked this option.

Use Neil's modsproject with the Carvera mill 2D PCB program to generate the final G-code for the milling machine. The tool is self-explanatory. Just follow the pipeline to proceed.

The initial milling produced some burrs, but after a bit of sanding, the results were very good.

Before sanding

After sanding

However, as I gathered components for stuffing the board, I saw my mistake: we didn't have any surface-mount pin connector sockets in the lab. I could either bend the legs of the through-hole connectors or redesign the board. This was a key lesson: check component availability and stay flexible during the design process.

Bending the legs of the through-hole connectors was not ideal: they added extra height and also undermined the reliability of the connection. I also took TA feedback into account and decided to redesign the board as a two-sided PCB with a ground plane on the back, which would also simplify routing.

Two-sided PCB design

Thanks to this change, I learned how to fill a ground plane in KiCad and produced my new design.

Ground plane filling

This led to my second mistake, which I thankfully caught before milling. When I switched from surface-mount to through-hole components, I didn't consider the soldering process. I had assumed we would have plated through-holes (PTH), allowing me to solder on either side. Without PTH, I would have to solder components on the opposite side of the board and use vias to connect traces between the front and back. This required another complete redesign to account for these manufacturing constraints. Lesson learned: think about the manufacturing process during design.

You cannot solder under the plastic

In the final redesign, I added vias, M2 mounting holes, and rounded edges for a more polished board.

Final two-sided PCB design

Milling a two-sided PCB proved to be a challenge. While there is a clever trick using symmetry to create a fixture for perfect alignment, I opted for a manual calibration method. I milled the first side, measured the machine's offset, and calculated the new origin for the second side.

My first attempt failed because I forgot to mirror the backside image and didn't include the tab offset for both sides, resulting in total misalignment.

Everything that could have gone wrong did go wrong

The second attempt, however, was a success. The holes aligned perfectly.

Good alignment from manually calculating the offset

Here are the calculations I used to align the origins:

// Backside milled first
bottom_offset: 5 mm
entered_left_offset: 3.9265 mm
expected_right_offset: 3.9265 mm
actual_left_offset: 5.51 mm
actual_right_offset: 4.74 mm

// Frontside calculation
bottom_offset: 5 mm
entered_left_offset: 3.156 mm = 4.74 - (5.51 - 3.9265)

Soldering this board was much harder than expected. The non-plated through-holes had limited grip on the solder, even with flux. I had to reflow several connections multiple times to ensure a solid connection.

Checking with the TAs, I received the advice to use a higher temperature or to preheat the pins longer. They diagnosed me with a "cold solder joint" issue.

The Switchboard

The Switchboard is an array of TRRS jacks, each with an LED to indicate status.

For the Switchboard PCB, I made almost all the same mistakes as with the Operator. My original design also assumed plated through-holes. Luckily, the Switchboard is a single-sided board, so I could move all the components to the other side and adjust the circuit without needing vias.

Original design without considering soldering

However, I then made two more errors:

I forgot to mirror the component footprints after flipping them to the other side.
I used the wrong pins for the connectors after flipping the design and only realized it after milling.

Corrected design with mirrored footprints and correct pins

I didn't notice the mirroring problem until I had finished soldering all the resistors for the LEDs. At that point, I decided to capitalize on the mistake and use the partially assembled board to test the LED circuit. I placed an LED on the pads without solder and used a multimeter to light it up. It worked!

Testing LED circuit

Bonus: Laser cutting my own ICS-43434 breakout board

I needed a breakout board for the ICS-43434 microphone, similar to this one from Adafruit. I decided to try fabricating it with a laser cutter.

I followed a YouTube tutorial to set up the machine, but his settings UI was different from mine. Instead of speed (mm/s), I had dot duration (microseconds). I proceeded with a test cut with varying duration values. The results were terrible: some parts of the copper were burned away, but the surface wasn't fully removed. The laser also seemed to have reduced power towards the edges of the cutting area.

Failed test

I soon realized the problem was the image format. Switching to SVG files exposed the correct settings. The gerber2img tool didn't export SVGs, and KiCad's native SVG export produced the wrong shapes when opened in xTool. I ended up using the Adobe PNG to Vector converter and then manually cleaned up the SVG in Figma.

I settled on the following workflow:

Drill alignment holes with the PCB mill first.
Use the laser cutter to ablate the copper for the traces, using the holes for alignment.
Use the mill again to cut the board's outline.

I scaled the SVG to match the measured dimension in KiCad before applying it on top of the drill holes. Alignment

I first made the mistake that the image was outdated after I moved the stock. This caused the laser to engrave on the bed.

Engraving on the bed

The xTool V1 Ultra's camera alignment was slightly off. Even with careful alignment in the software, the first few results were shifted.

The perfect alignment turned out to be offset by 0.3mm on both axes

I also dialed in the machine settings. It took a total of 20 passes to completely remove the copper.

10 passes each, processed twice

After applying a manual offset, my fourth attempt produced a very good result and was ready for the edge cut. The tolerance for edge cutting is wide. I visually aligned the outline using the Carvera's laser scan. It worked perfectly. Laser scan for edge cut

The fourth attempt was good enough for milling

In retrospect, I could have drilled the holes and cut the outline in one go before laser cutting. That would save a trip to the mill.

I also checked with TA Quentin that the xTool software has its own vectorize tool. I planned to use that next time instead of using Adobe and Figma.

Soldering the ICS-43434 was nightmarish. The component has tiny pads on its underside, which are unreachable with a soldering iron. The laser-etched traces were so thin they could barely hold any solder.

How to solder this?

I used a heat gun, hoping surface tension would do the work, but the heat began to damage the microphone's plastic casing. An attempt to heat the board from the bottom nearly burned the PCB before the solder melted. There was a warning during the lecture that FR1 was a poor heat conductor so I should have avoided this in the first place. After many tries, I managed to get it positioned with no shorts between the pads, but I have yet to test if the microphone actually works.

Heating up the PCB from the bottom resulted in damage

I tested with a multimeter and confirmed there was no short. But it was unclear whether the microphone survived the heat damage.

Microphone breakout, soldered

At this point, I realized that the pinout on my board is not the same as the Adafruit breakout board which my Operator board assumes. The only way for it to be compatible is to use jumper wires to sort out the connections. This was not ideal, but I had no choice now as it was already 3 AM.

Later, I received advice from TA Quentin that I could use solder paste. The process looks like this:

Apply solder paste to the pads
Heat from above to let the solder paste flow into the trace
Place the component and heat again from above and let surface tension do the work

This could be something to try for v2.

Integration Test

These were all the components I made this week. From top, clockwise:

ICS-43434 breakout board (laser cut)
Switchboard (milled, single-sided)
Operator (milled, two-sided)

All the components fabricated this week

Let's assemble and test. Since we plan to have dedicated I/O and networking weeks, I focused only on basic connectivity.

I programmed the Switchboard to supply digital signals to each TRRS socket, and programmed the Operator to read the digital signal and determine which pins have high voltage. To simulate the final project scenario, I used a real TRRS male and female connector pair.

Assembly with real TRRS connectors

I was able to confirm from the serial output that all the TRRS sockets are giving out distinct high/low voltage patterns.

Digital readings from TRRS sockets

Key observations:

Unplugging/replugging caused the reading to be high momentarily
It took 1-2 seconds for the reading to stabilize after plugging in

I switched the digital pins to use PULL_UP mode, and the floating HIGH issue disappeared. I also realized that I need to reserve the LOW LOW LOW reading for unplugged state, therefore, I only have 7 distinct states for 8 TRRS sockets.

Key lessons

This week was a gauntlet of trial and error, but the lessons were invaluable.

Heeding the instructions in class. could have prevented the burning of the ICS-43434 breakout board.
Check component availability before designing. A simple stock check can save a complete redesign.
Think about the manufacturing process during design. Constraints like non-plated through-holes fundamentally change how a board must be laid out.
Double-check orientations, especially when two-sided PCBs are involved. Mirroring is easy to forget and fatal to a design.
Homemade through-holes do not solder as easily as commercial PTH. If possible, consider using surface-mount components for easier assembly.
Don't rely on intuition. I found myself inclined to make the same mistake over and over. A checklist or a more rigorous verification process is essential.

Appendix

Week 5: Into the Matrix

Sun Chuanqi — Sat, 04 Oct 2025 00:00:00 GMT

Group assignment

For the group assignment, Alan Han explained power supply, multimeter, oscilloscope, function generator, and logic analyzer. I used my own project to study the logic analyzer. The device used MAX98357A as DAC and amp but had an issue producing sound. During the debugging session with the TAs (Nikhil and Alan), we ruled out power issue by observing stable power voltage. We also ruled out connection issue as all the lines: clock, channel, and data were normal. We found low voltage on speaker. Eventually, we realized the bug was software. The diagnostic tool was helping in eliminating hardware issues.

During the testing, we noticed that the plugging in/out of probes can interfere with the I2S signal. Presumably because of the motion of the probes causing inductance change, which in turn, distorted the clock or data line.

Hooking up logic analyzer probes

We used saleae Logic 2 analyzer with the companion software. The tool can visualize both digital and analog signals and allowed us to choose a protocol and decode the data. Very helpful.

Visualize data

Simulation

I wanted to simulate the hand-held device of my final project. For simulation, I wanted to study the idea of addressing unique 3.5mm audio jacks using binary encoding.

TRRS jack

One pin for ground, 3 pins for 2^3 = 8 unique addresses. I used an 8-pin DIP switch to simulate the 3-bit binary input. In production, the input will be determined by the jack that the user plugs in. The jack's identity only matters in the software, but to make the simulation fun, I added an LCD display to show which jack is plugged in. Since it's a walkie talkie, I added a push button to show TTS, and a slide switch to toggle between modes: programming vs. interaction. I used wokwi, which provided the Xiao-ESP32-C3 board that my project uses. For production, there won't be an LCD, but will be an I2C DAC/amp and a I2C ADC for audio input/output. I will prototype them for the actual PCB design. If there are enough I/O pins, I will consider a second button so user can squeeze both to enable broadcast mode.

Simulate the hand-held device

I also wanted to simulate the main body device. It has two roles: lighting up an LED for a specific jack, where the computer will control which jack is active and use wifi to activate it, and setting the 3-bit address for each jack, which can be hardwired directly from Xiao's 3.3v output. To make the simulation more interesting, I cycled the LEDs in order, just to show I can address them by software.

Simulate the main device

Designing the Operator

To prepare KiCad with the symbols, footprints, and 3D models, I followed the documentation to install our course library. In addition, depending on the installation method, you may need to manually add additional libraries from the official repo. When installing with Flatpak, the official 3D models were already included.

I wanted to start the PCB design towards my final project, using KiCad. The schematic design is very simple: layout the Xiao-ESP32-C3, I2S Microphone, I2S DAC/Amp, a TRRS jack, and a couple of switches.

Schematic design

As soon as I transitioned to the PCB design, I realized the problem: I hoped to use the MAX98357A and ICS-43434 both on their breakout boards, not as bare ICs.

PCB design missing breakout boards

Also, I don't want to solder them onto the same board as the Xiao. As a consumer, I enjoy electronics that come with user repairable parts. So I decided to pivot: design a development board that helps me route the pins from the Xiao to the headers where I can plug in the breakout boards. If I accidentally damage any component, I want to make it easy to swap out the broken part.

Starting over, I put down the schematic design using pin sockets and loaded them into PCB editor. When I opened the 3D view, I realized that I chose the wrong connector.

Schematic design with the wrong sockets

PCB design with the wrong sockets

I wanted female socket, but got the male pin instead. Switching to female socket, the symbol exists, but they are missing footprint. Luckily, I was able to find a similar footprint. Assigning the footprint to the symbol allowed me to open the PCB editor based on schematic design.

The symbol is missing footprint

Manually assign footprint

Schematic design with the right sockets

3D view confirmed socket type

Just as I thought I was approaching the finish line, I recalled to run the constraint checker. I realized I had chosen the wrong track width. Instead of 0.2mm, I should have chosen 0.4mm.

I should have selected track width sooner

Set minimum clearance

It took me a while to manually fix all the wires. I also marked up the area for the speaker, just to get a sense of scale.

PCB design with the correct sockets

Feeling a bit uncertain, I went to the office of our TA Alfonso and got the immediate bad news: I didn't run constraint checker after changing the track width. Running the constraint checker again, more problems appeared. Because I laid out the wires using 0.2mm track, now the wires are touching the keepout area. After several rounds of manual fixing, I felt that it was impossible to clear the errors without crossing at least one wire. Alfonso agreed and encouraged me to try using 0 ohm resistors as a jumper.

Crossing the wire

Adding a jumper was not as straightforward as I thought. I had to go back to add the symbol, then redo almost the entire PCB design due to position shifting. The resistor jumper requires additional clearance, forcing me to push out the speaker area.

Jumper in schematic

3D view with the jumper

Designing the Switchboard

After designing the hand-held device PCB, I felt more comfortable with the tool and quickly designed the main body PCB. For each TRRS jack, I added an LED as status indicator. I "hardwired" the 3-bit address from Xiao to each jack. I used resistors to protect the LEDs from high voltage.

Main body schematic

Main body PCB design

Main body 3D view

Key lessons learned: Apply design rules before starting PCB layout. Check constraints often. Do not lay the wires too early. They are not easy to change.