Complete transcript of Cursor AI assistance for updating Week 11 individual assignment documentation, including MIT Wi-Fi instructions, Networked System 3 (Long Range Radio Board Design), and Network Address Tables
Full Conversation
UserUnknown
general guidelines:
— for each image or video add a caption and description.
— make all the text professional and succinct without losing key details
— make the information as organized and as aesthetically pleasing as possible
— don't just take my notes from the prompt, refine it before putting it on the page
-- weave in text description between images when you update so it all makes sense together, i gave you the names of the images or videos to grep with each section
specific tasks:
-- update week11 individual assignment camera wifi livestream with MIT Wifi instructions by refining these notes below into step by step process, including the image week11/media/mit-wifi-page.png
just use 'MIT' wifi network with the password they generate for you
To get your password:
https://wifi.mit.edu/
Click 'Using a device not 802.1X compatible? Click here'
Login with Touchstone with your kerberos
It should give you a QR code and password, you can have it emailed to yourself, generate a new one, etc.
This is the page you want to get the password from (see attached image)
Use this password in the code
const char* ssid = "MIT";
const char* password = "insert generated password from steps above;
-- update week11 individual assignment with a new subsection 'Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic' after 'Networked System 2: Camera Livestream & Edge AI (Wi-Fi)' right before 'Network Address Tables'
fill out the same template as Networked System 1 and Networked System 2 but add an inspiration heading called Invisible String right before design & build
explain what the invisible string is and include an amazon link. explain that in the Invisible string workbook there's an activity for making an invisible string chain of hearts. :-)
I designed a board for the XIAO ESP32S3 & Wio-SX1262 Kit for Meshtastic & LoRa (kit link: https://www.seeedstudio.com/Wio-SX1262-with-XIAO-ESP32S3-p-5982.html?srsltid=AfmBOoqHXFeWGqG6YRbaOwUdHwVEy7ljrBPbhOkdhf4rmtafwbQo9GdE , usage link: https://wiki.seeedstudio.com/wio_sx1262_with_xiao_esp32s3_kit/ ) that is connected to the pulse oximeter I'm using for my final project https://a.co/d/eGLIsBh (schematic in https://a.co/d/hXnFRRo for some reason amazon_schematic.png and amazon_schematic_2.png + analog_devices_schematic.png from https://www.analog.com/media/en/technical-documentation/data-sheets/max30102.pdf ). this was done with the same steps as in Week 4 https://fab.cba.mit.edu/classes/863.25/people/SaleemAldajani/week4.html
I milled the board using the Othermill in 38-501 as per the electronics production protocol in week 5 https://fab.cba.mit.edu/classes/863.25/people/SaleemAldajani/week5.html#training
milling on Bantam Tools and milled boards:
milling-invisible-string.jpeg
milled-boards.jpeg
I soldered through-hole components with more advanced techniques using a board stand either holding the board vertically or horizontally depending on the board positions, embed these images
soldering techniques:
desoldering-headers.jpeg
soldering-screen.jpeg
soldering-esp.jpeg
presoldering configuration and checking schematic to make sure connections ok:
throughholes-invisible-string.jpeg
config-invisible-string.jpeg
fixing milling errors by probing shorts with continuiting mode and exacto knife cutting any hairs:
milling-error-hair-copper.jpeg side by side with design-where-to-cut-2.jpeg
milling_issue_hair_copper
in the end i had to slash off the common ground on the interrupt side so RD and IRD are isolated from the rest of the GND (might even have to isolate RD and IRD for more accurate readings)
'Network Type & Protocol'
from meshtastic iPhone app:
#Node Number 3665045004
User Id !da74260c
Device name: Meshtastic 260c
fill out anything else from the websites I linked for the ESP32S3 Wio-SX1262 module
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
1. MIT Wi‑Fi connection instructions — Added a step-by-step section in Networked System 2 with:
- Instructions for generating a password from wifi.mit.edu
- Code example showing how to use the password
- Image of the MIT Wi‑Fi password generation page
2. Networked System 3: Long Range Radio Board Design — Added a section after Networked System 2, including:
- Invisible String inspiration — Explanation of the concept with links to the book series and workbook
- Design & Build — PCB design process with images:
- Component selection and library management
- PCB layout and distance measurement
- Schematic design and connections
- Routing and design rule check
- Final design files (schematic, PCB, 3D rendering)
- PCB milling and fabrication
- Soldering and assembly techniques
- Milling error correction
- Network Type & Protocol — LoRa/Meshtastic details with device screenshots and videos
- Input & Output Devices — MAX30102 sensor and OLED display with working images/videos
- Connections — Technical connection details
- Design Files — Links to all design files
3. Network Address Tables — Added a new table for the LoRa Meshtastic (Invisible String) system with:
- Node Number: 3665045004
- User ID: !da74260c
- Device Name: Meshtastic 260c
- Complete device information
All images and videos include captions and descriptions. The text is organized and formatted to match the existing page style.
UserUnknown
link to Network Address Tables in each Networked System 1 and Networked System 2 and Networked System 3 sections
add a second row of highlights that link to exactly where everything showing the image/video
first column: Invisible string Final PCB layout side by side with 3D CAD rendering
second column: Milled PCBs side by side with Configuration check
third column: Meshtastic device screen side by side with Meshtastic connection process
AssistantUnknown
Adding links to Network Address Tables in each Networked System section and creating a second row of highlights. Reviewing the structure:
Reading the highlights section to understand the structure:
Adding links to Network Address Tables in each Networked System section and creating a second highlights row. Finding where to add the ID for the Network Address Tables section:
Finding the exact locations of the images to link to:
Adding IDs to the Network Address Tables section and the relevant figures, then adding links in each Networked System section, and creating the second highlights row:
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
See Network Address Tables for complete device addressing information including MAC addresses for both player devices.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
See Network Address Tables for complete device addressing information including MAC addresses for both player devices.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
See Network Address Tables for complete device addressing information including MAC and IP addresses for the camera system.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
See Network Address Tables for complete device addressing information including MAC addresses for both player devices.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
See Network Address Tables for complete device addressing information including MAC and IP addresses for the camera system.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
See Network Address Tables for complete device addressing information including Node Number, User ID, and Device Name for the Meshtastic network.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
See Network Address Tables for complete device addressing information including MAC addresses for both player devices.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
See Network Address Tables for complete device addressing information including MAC and IP addresses for the camera system.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
See Network Address Tables for complete device addressing information including Node Number, User ID, and Device Name for the Meshtastic network.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
1. Added links to Network Address Tables in each Networked System section:
- Networked System 1: Added link in Connections section
- Networked System 2: Added link in Connections section
- Networked System 3: Added link in Connections section
2. Added IDs to Network Address Tables section and relevant figures for direct linking
3. Created second row of highlights with the requested layout:
- First column: "Invisible String Final PCB Layout" — Final PCB layout side by side with 3D CAD rendering
- Second column: "Milled PCBs & Configuration" — Milled PCBs side by side with Configuration check
- Third column: "Meshtastic Device & Connection" — Meshtastic device screen side by side with Meshtastic connection process video
All links point to the exact locations where the images/videos are displayed using anchor IDs. The highlights row follows the same styling as the first row and links directly to the relevant sections in the Networked System 3 documentation.
UserUnknown
the second row of highlights are a fourth column to the first row somehow, fix this
AssistantUnknown
Checking the highlights section structure to fix the layout:
The first row's closing
is missing, so Row 2 is nested inside Row 1. Closing Row 1 before Row 2:
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
See Network Address Tables for complete device addressing information including MAC addresses for both player devices.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
See Network Address Tables for complete device addressing information including MAC and IP addresses for the camera system.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
See Network Address Tables for complete device addressing information including Node Number, User ID, and Device Name for the Meshtastic network.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
was missing, so Row 2 was nested inside Row 1. Added the closing tag so Row 2 now appears as a separate row below Row 1, not as a fourth column. The highlights section now displays two rows of three columns each.
UserUnknown
add white space at the top and bottom of the left image in Invisible String Final PCB Layout so the full layout can be seen
AssistantUnknown
Adding top and bottom padding to the left image in the "Invisible String Final PCB Layout" highlight so the full layout is visible:
Updating the left image to use object-fit: contain and adding padding so the full layout is visible:
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
See Network Address Tables for complete device addressing information including MAC addresses for both player devices.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
See Network Address Tables for complete device addressing information including MAC and IP addresses for the camera system.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
See Network Address Tables for complete device addressing information including Node Number, User ID, and Device Name for the Meshtastic network.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
Updated the left image in the "Invisible String Final PCB Layout" highlight:
1. Changed object-fit from cover to contain so the full layout is visible
2. Added padding:8px 0 for top and bottom spacing
3. Added background:white to provide white space around the image
The full PCB layout is now visible with white space at the top and bottom, while the right image (3D CAD rendering) remains unchanged.
UserUnknown
white space on left and right of the right image in Invisible String Final PCB Layout same thing
AssistantUnknown
Adding left and right padding to the right image (3D CAD rendering) so the full image is visible:
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes: How to Integrate (Almost) Any System
Understanding failure modes is critical for robust system design. See the class system integration page for detailed failure mode documentation and links.
Design: PCs vs Macs (Box vs Designed Products)
The contrast between functional boxes (PCs) and designed products (Macs) illustrates the importance of user experience design. This connects to "How to Design (Almost) Anything" and "How to Make (Almost) Anything Usable" classes.
Dr. Jan Borchers's Session: Usability
hci.rwth-aachen.de/fabusability — Computer science, design, and psychology intersect in usability research. Usability focuses on designing systems that work great for people, whether in projects, startups, or products like FabFame.
Golden Rules of Usability
Simplicity — Keep it simple, avoid feature creep. Consider user (price tag and feature list), task (reduce complexity to fit the task), and context.
Visibility and Feedback — Immediately see current state, available features, and how to access them.
Gestalt Laws — How we perceive units/groups through distance/proximity, shapes, and similarity.
Natural Mapping — Physical arrangement matches physical movement (e.g., up/down buttons, sliders, rotational dials). Additive dimensions (power, heat, light, water level, volume). Haptic feedback enables operation without looking (e.g., car safety: physical buttons vs touch screen).
User's Language — Use terminology familiar to users.
Avoid Modes — Prevent mode confusion.
Principle of Least Surprise — Systems should behave as users expect.
Dialogue, Not Monologue — Interactive, responsive systems.
Tolerate Errors — Provide constructive feedback and error recovery.
Visual Design — Use nice color schemes (e.g., color.adobe.com).
Process Tips
Design is iterative and agile
Observe and ask first, then start solving
Design: Search the solution space
Prototype: Implementations for feedback
Analyze: Observe and ask for constructive interaction
Analyze: Retrospective testing
Iterate to expand and focus solution
Literature: The Design of Everyday Things (intro to human computer interaction). See hci.rwth-aachen.de/fabusability for more resources.
Packaging
Route wiring appropriately and mount circuit boards in a box—don't just throw them in! See examples from Gianfranco Paluto on Fab Academy. Consider mechanism alignment and surface finish.
Testing
Prevent defects (QA) and detect defects (QC). Methods include shake and drop tests, power cycling, environmental testing, and random inputs.
Mechanical — Static vs dynamic failure (e.g., Tacoma Narrows Bridge)
Wiring — Ground mecca necessary where all grounds meet. Can't just daisy chain power and ground; otherwise devices get different grounds.
Components — Component-level failures
Power — Robust against wrong connections
Software — Buffer overflow is a serious vulnerability. Security through obscurity is insufficient.
Scaling — O(?) complexity, phase transitions in complexity
Manufacturing — Production-related failures
Collaboration — Team and communication failures
Fail Fast!!! — Feed-forward vs feedback development. Murphy's Law (rocket test engineer): "Anything that can go wrong, will go wrong." This means you have to anticipate how it'll fail and prevent it.
Repair
Drop/Shake — Test robustness
Modularity — Debug module at a time
Widlarize — Bob Widlar's approach: smash it with a hammer (as a broken system) to understand failure modes
Lifecycle
Right to Repair — Design for repairability
Disassembly — Consider how systems come apart
End-of-Life — Plan for system retirement and recycling
Assignment: System Integration for Final Project
Design and document the system integration for your final project:
What are the modules? (Hardware and software)
CAD for how everything comes together
Architecture for modularity: how all hardware comes together
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
See Network Address Tables for complete device addressing information including MAC addresses for both player devices.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
See Network Address Tables for complete device addressing information including MAC and IP addresses for the camera system.
Connecting to MIT Wi-Fi Network
To connect the ESP32-S3 camera system to the MIT Wi-Fi network, you need to generate a device-specific password. The MIT network uses WPA2-PSK authentication for devices that are not 802.1X compatible.
Networked System 3: Long Range Radio Board Design, Fabrication, and Connection to Meshtastic
A custom PCB design integrating the XIAO ESP32-S3 with the Wio-SX1262 LoRa radio module for Meshtastic communication, connected to a MAX30102 pulse oximeter sensor. The system enables long-range wireless communication using LoRa (Long Range) radio technology and the Meshtastic mesh networking protocol for decentralized device-to-device messaging.
Invisible String
This project draws inspiration from the Invisible String book series by Patrice Karst, which explores the concept of invisible connections that link us to the people we love, even when we're far apart. The book series and workbook include activities for making an invisible string chain of hearts, symbolizing these connections. This long-range radio system embodies that concept by creating invisible wireless links between devices across distances, enabling communication and data sharing through the Meshtastic mesh network.
The MAX30102 sensor integrates red and infrared LEDs, a photodetector, and low-noise electronics for heart rate and blood oxygen (SpO2) measurement. Reference schematics were sourced from the Amazon product page and the Analog Devices MAX30102 datasheet.
Component Selection and Library Management
Component selection required careful library management in Fusion 360 Electronics. I started by identifying the correct through-hole header variants needed for the XIAO ESP32-S3 and Wio-SX1262 modules, ensuring proper pin spacing and mechanical compatibility.
Component selection processInitial component selection showing taped version for reference during design.Part verificationIdentifying incorrect part selections to ensure proper component matching.
Adding componentsAdding new components to the schematic from Fusion 360 component library.Header library searchSearching Fusion 360 library for appropriate header components.
Header requirementsIdentifying specific header types needed for XIAO ESP32-S3 and Wio-SX1262 modules.Through-hole header selectionSelecting appropriate through-hole header variant for module connections.
Complete component assemblyAll components placed in schematic: XIAO ESP32-S3, Wio-SX1262, and MAX30102 pulse oximeter sensor.
PCB Layout and Distance Measurement
Accurate spacing between components was critical for proper module alignment. I used physical measurements from breadboard prototypes and Fusion 360's measurement tools to ensure correct header placement and pin spacing.
Header distance measurementMeasuring physical distance between headers to ensure proper module spacing on PCB.Breadboard prototypingUsing breadboard to verify component spacing and connection requirements before PCB design.
Schematic Design and Component Connections
The schematic integrates the MAX30102 sensor with pull-up resistors for I²C communication, connects power and ground planes, and routes all necessary signals between the ESP32-S3, Wio-SX1262, and sensor modules.
PCB layout with componentsInitial PCB layout showing component placement before routing connections.Pull-up resistors for I²CAdding pull-up resistors to I²C bus (SDA and SCL) for proper communication with MAX30102 sensor.
Routing and Design Rule Check
PCB routing required careful attention to signal integrity, especially for I²C lines and power distribution. I iterated from initial routing to clean, labeled connections, then performed design rule checks to ensure manufacturability.
Initial routingFirst pass at routing connections between components, showing basic connectivity.Refined routing with labelsCleaned up routing with proper net labels and organized trace layout for better readability.
Design rule checkDesign rule check (DRC) verification ensuring all traces meet minimum width, spacing, and via requirements for manufacturing.
Final Design Files
The final design includes a complete schematic, PCB layout, and 3D CAD rendering showing the assembled board with all modules and components.
Final schematic designComplete schematic showing all component connections, power distribution, and I²C pull-up resistors.Final PCB layoutCompleted PCB layout with all traces routed, ground planes filled, and components properly positioned.
3D CAD rendering3D visualization of the assembled board showing XIAO ESP32-S3, Wio-SX1262, and MAX30102 sensor module placement.
PCB Milling and Fabrication
The board was milled using the Othermill (Bantam Tools) in room 38-501, following the electronics production protocol established in Week 5. The milling process required careful tool selection, proper depth settings, and multiple passes for traces and isolation.
PCB milling processMilling the invisible string PCB on Bantam Tools Othermill, showing tool path and material removal.Milled PCBsCompleted milled boards showing clean traces, isolation cuts, and mounting holes ready for component assembly.
Soldering and Assembly
Through-hole component soldering required advanced techniques using a board stand, positioning the board vertically or horizontally depending on component location. This approach improved access to pads and ensured consistent solder joints.
Desoldering headersRemoving incorrectly placed headers using desoldering techniques and board stand for stability.Soldering with board standUsing board stand in horizontal position for soldering screen connections with improved access and visibility.
Soldering ESP32-S3 moduleSoldering XIAO ESP32-S3 module headers with board positioned for optimal access to all pins.
Through-hole components installedAll through-hole headers and components soldered in place, ready for module installation.Configuration checkVerifying component placement and connections against schematic before final assembly.
Milling Error Correction
Milling errors resulted in copper "hairs" creating unintended shorts between traces. I used continuity mode on a multimeter to probe for shorts and an exacto knife to carefully cut away the excess copper, restoring proper isolation between traces.
Milling error with copper hairExcess copper "hair" creating short circuit between traces, requiring careful removal.Design showing where to cutPCB design highlighting areas requiring isolation cuts to remove copper shorts.
In the end, I had to isolate the common ground on the interrupt side so that RD (red LED) and IRD (infrared LED) grounding terminals are isolated from the rest of the GND plane. This isolation may be necessary for more accurate sensor readings by preventing ground loops and interference.
Network Type & Protocol
Network Type: Long-range wireless mesh network (LoRa) Protocol: Meshtastic (open-source mesh networking protocol over LoRa radio) Radio Module: Wio-SX1262 (Semtech SX1262 LoRa transceiver) Addressing Scheme: Meshtastic node ID and user ID for device identification
The Wio-SX1262 module provides long-range, low-power wireless communication using LoRa (Long Range) spread spectrum modulation. Meshtastic runs on top of LoRa, creating a decentralized mesh network where devices can relay messages to extend range. The system uses the ESP32-S3 as the main processor, running Meshtastic firmware that manages the LoRa radio and mesh networking protocols.
Meshtastic device screenMeshtastic interface showing device status, node information, and network connectivity.Device found in Meshtastic appiPhone Meshtastic app detecting the invisible string device on the mesh network.
Device connected to MeshtasticSuccessful connection established between iPhone app and invisible string device on mesh network.Meshtastic connection processVideo showing the connection sequence between Meshtastic app and device during network initialization.
Working Meshtastic screenMeshtastic device operating on mesh network, showing node information: Node Number 3665045004, User ID !da74260c, Device name: Meshtastic 260c.
Input & Output Devices
Input Devices: Heart Rate Sensor Module MAX30102 (pulse detection, blood oxygen concentration measurement) — I²C interface for data communication
Output Devices: SSD1306 I²C 128×64 OLED display for sensor data visualization and system status
The MAX30102 sensor uses optical reflection (PPG - photoplethysmography) to detect heart rate and blood oxygen levels. It integrates red (660nm) and infrared (880nm) LEDs with a photodetector to measure changes in blood volume through the skin.
MAX30102 sensor operatingPulse oximeter sensor active with red and infrared LEDs illuminating for heart rate and SpO2 measurement.MAX30102 LED operationVideo showing MAX30102 sensor LEDs pulsing during heart rate and blood oxygen measurement cycle.
Connections
The XIAO ESP32-S3 connects to the Wio-SX1262 module via SPI interface for LoRa radio control. The MAX30102 sensor connects via I²C (SDA and SCL lines) with pull-up resistors. The OLED display also connects via I²C on the same bus. Power is distributed from the ESP32-S3's 3.3V and 5V rails to all modules. The interrupt pin (INT) from MAX30102 can be connected for event-driven data acquisition, and the RD/IRD pins are isolated from the main ground plane for improved sensor accuracy.
See Network Address Tables for complete device addressing information including Node Number, User ID, and Device Name for the Meshtastic network.
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
LoRa Meshtastic Network (Invisible String)
Device
Component
Node Number
User ID
Device Name
Network Type
Input Device
Output Device
Invisible String Node
XIAO ESP32-S3 + Wio-SX1262
3665045004
!da74260c
Meshtastic 260c
LoRa (Meshtastic)
MAX30102 Heart Rate Sensor
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment. For Meshtastic networks, devices are identified by Node Number (unique identifier), User ID (hexadecimal user identifier), and Device Name (user-configurable name). The LoRa radio uses spread spectrum modulation for long-range, low-power communication without requiring IP addresses.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
Cursor AI assisted with comprehensive updates to Week 11 documentation, including ethical AI use section updates, design files documentation with pseudocode and key features, group assignment CircleSquareCamera AR section development, network address tables, and overall page organization. The AI helped refine technical content, create professional code documentation, add media captions and descriptions, and organize information for optimal readability and aesthetic presentation.
The AI was used to:
Update ethical AI use section with ChatGPT co-development transcript references
Create comprehensive design files documentation with pseudocode, key features, and "what you need to know" sections
Develop group assignment CircleSquareCamera AR subsection with detailed technical explanations
Add network address tables for CircleSquareCamera AR system with all device MAC addresses
Implement download links for all code files (zip archives and individual files)
Add professional captions and descriptions to all images and videos
Refine and organize content for professional presentation while preserving technical accuracy
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
Cursor AI assisted with updating the Week 11 recitation notes section with comprehensive system integration content. The AI helped refine and organize raw notes from the recitation session into a well-structured, professional format with proper headings, bullet points, and external links. The updated section includes usability principles from Dr. Jan Borchers's session, failure modes analysis, testing strategies, repair considerations, lifecycle planning, and assignment details.
The AI was used to:
Organize raw recitation notes into structured sections with proper headings
Format content with consistent styling matching the rest of the page
Add external links to resources (Slack message, class pages, GitLab, etc.)
Create clear bullet points and lists for readability
Integrate the refined notes into the existing recitation section
Maintain technical accuracy while improving presentation
Updated the right image (3D CAD rendering) in the "Invisible String Final PCB Layout" highlight:
1. Changed object-fit from cover to contain so the full image is visible
2. Added padding:0 8px for left and right spacing
3. Added background:white to provide white space around the image
Both images now show their full content with white space: the left image has top/bottom padding, and the right image has left/right padding.
Download Options
For the complete technical details and full conversation transcript:
![[Profile Image Placeholder]](profile.jpg)
This work is licensed under a
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License
