# Update project documentation and organization
_Exported on 11/29/2025 at 05:59:05 EST from Cursor (2.1.39)_
---
**User**
— for each image or video add a caption and description.
— make all the text professional and succinct without losing key details
— make the information as organized and as aesthetically pleasing as possible
— don't just take my notes from the prompt, refine it before putting it on the page
specific tasks:
1. update ethical ai use section with chatgpt co-development transcript
Camera integration game update
https://chatgpt.com/share/692a4098-5428-8008-81bc-143788d91cb9
2. update design files in week 11 with, both pseudocode how the code works, code key features, all you need to know about whats in the code, download links to the code zip and individual files in them, and a link to the ethical ai use chatgpt transcript
a) Video dithering stream to OLED — view_camera_oled.zip, view_camera_oled.ino, camera_pins.h
b) Group assignment codes in week11/code
b1: camera board code -- CircleSquareCamera.zip, view_camera_oled.ino, camera_pins.h
b2: game board code -- TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.zip, TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino
3. update 'Group Assignment: Send a Message Between Two Projects'
keep the assignment description at the top of the subsection, then name the new subsection header with the new stuff in CircleSquareCamera Augmented Reality
using ESP-NOW, i got the original circlesquareshapesongswingalong documented in week2 and here
https://fab.cba.mit.edu/classes/863.25/people/SaleemAldajani/week2.html#circlesquareshapesongswingalong
https://fab.cba.mit.edu/classes/863.25/people/SaleemAldajani/week11.html#circlesquare-network
to talk to my camera code developed in final project, week 7, week 8, and week 12
https://fab.cba.mit.edu/classes/863.25/people/SaleemAldajani/week7.html#camera-edge-ai
https://fab.cba.mit.edu/classes/863.25/people/SaleemAldajani/week8.html#camera-display
https://fab.cba.mit.edu/classes/863.25/people/SaleemAldajani/finalproject.html#camera-subsystem
https://fab.cba.mit.edu/classes/863.25/people/SaleemAldajani/finalproject.html#camera-subsystem
(placeholder link for week 12 will add later)
add the video in week11/media/@CircleSquareCameraAR.mp4
remove any remaining placeholder on the page (except for placeholder links for week 12, I'll give you once I'm done with week12 documentation)
take a copy of the 'Network Address Tables' in individual assignment section and fill it out under group assignment using
🎮 Game boards
• P1: D8:3B:DA:75:05:AC
• P2: D8:3B:DA:75:E1:9C
📷 Camera boards
• Camera A: B8:F8:62:F9:E2:C0
• Camera B: B8:F8:62:F9:D6:38
explain in detail similar to how its in individual assignment Design & Build, Network Type & Protocol, Input & Output Devices, Connections, Download Design Files
---
**Cursor**
Reviewing `week11.html` to understand its structure, then updating the Ethical AI Use section, design files, and Group Assignment.
Reading additional files to understand the code structure:
Updating `week11.html` with the requested changes. Starting with the Ethical AI Use section, then the design files, and finally the Group Assignment section.
Updating the Ethical AI Use section first, then the other sections:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
[Placeholder: Recitation notes will be added here]
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment placeholder, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Updating the Group Assignment section with the CircleSquareCamera AR content. Reviewing the section structure:
Replacing the Group Assignment section with the full CircleSquareCamera AR content:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
[Placeholder: Recitation notes will be added here]
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment placeholder, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Updating the Design Files section with detailed code documentation. Reviewing the current structure:
Reading the Design Files section:
Adding documentation for the video dithering code and group assignment codes in the Design Files section. Inserting these sections after the Wi-Fi Camera Stream section:
Adding the new design file sections before the closing of the Design Files section:
```html
Video Dithering Stream to OLED
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
```
Locating the end of the Design Files section for insertion:
Inserting the new design file sections before the closing tag of the Design Files section:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
[Placeholder: Recitation notes will be added here]
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment placeholder, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Checking for remaining placeholders and updating captions. Searching for placeholders:
Removing placeholders (except week 12 links) and updating the highlights section with the CircleSquareCamera AR video:
Updating the highlights section to include the CircleSquareCamera AR video, removing the recitation placeholder, and fixing the Ethical AI Use text:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
[Placeholder: Recitation notes will be added here]
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment placeholder, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment placeholder, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Checking for lint errors and finalizing:
Reviewing the highlights section and verifying that all media elements have captions. Checking and updating the highlights section:
All media elements already have captions. Completing the final task:
## Completed updates
1. **Ethical AI Use section** — Added ChatGPT transcript link for camera integration game update at the top of the section.
2. **Design Files section** — Added documentation:
- **Video Dithering Stream to OLED** (`view_camera_oled.zip`): pseudocode, key features, what you need to know, download links, and ChatGPT transcript link
- **Group Assignment Codes**:
- **Camera Board Code** (`CircleSquareCamera.zip`): pseudocode, features, download links
- **Game Board Code** (`TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.zip`): pseudocode, features, download links
- All code sections include links to the ChatGPT transcript
3. **Group Assignment section** — Updated:
- Assignment description at the top
- "CircleSquareCamera Augmented Reality" subsection with:
- Description linking to related weeks
- Video demonstration (`CircleSquareCameraAR.mp4`)
- Design & Build, Network Type & Protocol, Input & Output Devices, Connections
- Network Address Table with MAC addresses for all 4 devices (2 game boards, 2 camera boards)
- Download links section
4. **Placeholders removed** — Removed all placeholders except Week 12 links (as requested)
5. **Media captions** — All images and videos have professional captions and descriptions
6. **Highlights section** — Updated to include CircleSquareCamera AR video as the third highlight
7. **Text refinement** — Made all text professional and succinct while preserving technical details
The page is now organized, aesthetically consistent, and includes all the requested documentation with proper links and downloads.
---
**User**
in the video caption refine this and add a disclaimer:
I was having trouble sending the camera to the game boards
Then after two hours of trying
I realized I can just send the game to the camera instead :slightly_smiling_face:
special thanks to Quentin for the quote on the back of the HTMAA XIAO board for the Magic School Bus quote (add the image if_at_first_you_dont_succeed_find_out_why.png with a link to https://www.reddit.com/r/CuratedTumblr/comments/16ypgqi/if_at_first_you_dont_succeed_find_out_why_this_is/ in the caption under
---
**Cursor**
Refining the video caption to include the development disclaimer and adding the image with the quote. Locating the current video caption:
Updating the video caption with the disclaimer and adding the image below:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data to game boards, but after troubleshooting communication issues, the architecture was reversed. The final system sends game state from game boards to camera boards via ESP-NOW, which proved more efficient for real-time AR overlay visualization.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Checking for lint errors and verifying the image file exists:
Refining the development note for clarity and flow:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Updated the CircleSquareCamera AR section:
1. **Refined video caption** — Added a development note explaining the architectural change from sending camera data to game boards, to sending game state to camera boards, after two hours of troubleshooting.
2. **Added Magic School Bus quote image** — Included the image (`if_at_first_you_dont_succeed_find_out_why.png`) below the video with:
- Clickable link to the Reddit source
- Caption acknowledging Quentin for the quote on the HTMAA XIAO board
- Styled to match the page design
The development note is in a highlighted box that draws attention while staying concise. The quote image is properly linked and formatted.
All changes are complete and ready. The page now documents the development process and acknowledges Quentin's contribution.
---
**User**
add specific links in 'Download Design Files' under Group assignment
to:
Video Dithering Stream to OLED
Group Assignment: CircleSquareCamera Augmented Reality System
Camera Board Code (CircleSquareCamera.ino)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
---
**Cursor**
Locating the "Download Design Files" section under Group Assignment to add the links:
Checking section anchors in the Design Files section to create the correct links:
Adding ID attributes to the relevant sections for linking. Updating the "Download Design Files" section under Group Assignment with specific links:
Adding ID attributes to the relevant sections, then updating the "Download Design Files" section under Group Assignment with specific links:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation. See Design Files section for detailed documentation and download links.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Checking for lint errors:
Added specific links in the "Download Design Files" section under Group Assignment. Here's what I did:
1. **Added ID attributes** to the four sections:
- `video-dithering-oled` for Video Dithering Stream to OLED
- `circlesquare-ar-design-files` for Group Assignment: CircleSquareCamera Augmented Reality System
- `camera-board-code` for Camera Board Code (CircleSquareCamera.ino)
- `game-board-code` for Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
2. **Updated the "Download Design Files" section** under Group Assignment with a bulleted list linking to all four sections.
The links now jump directly to the relevant sections in the Design Files area. No linting errors found. The page is ready.
---
**User**
make the highlight for 'Wi-Fi Camera Livestream'
side by side of the current image and the gif for 'Edge AI face detection' face_detection_edge_AI.gif under week10/camera/edgeai
make the image 'Magic School Bus quote' a lot smaller
---
**Cursor**
Updating the highlights section to show the Wi-Fi Camera Livestream image and the Edge AI face detection GIF side by side, and making the Magic School Bus quote image smaller:
Reading the Magic School Bus quote section to check its structure:
Updating the Wi-Fi Camera Livestream highlight to show the image and GIF side by side, and making the Magic School Bus quote image smaller:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Checking if we should remove the Edge AI face detection highlight since it's now included in the Wi-Fi Camera Livestream section:
Completed the updates:
1. **Wi-Fi Camera Livestream highlight** — Now shows the livestream image and the Edge AI face detection GIF side by side (50% width each).
2. **Magic School Bus quote image** — Reduced to a max width of 300px and centered; still responsive.
No linting errors. The highlights section now shows the Wi-Fi camera and Edge AI face detection together, and the quote image is smaller.
---
**User**
make the box for the Magic School Bus quote fit around the image
---
**Cursor**
Making the Magic School Bus quote box fit tightly around the image instead of full width:
Updating the figure container to wrap tightly around the image:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Updated the Magic School Bus quote figure box to wrap tightly around the image:
1. Set `display: inline-block` so the box sizes to its content
2. Set `max-width: 324px` (300px image + 24px padding) so it fits around the image
3. Wrapped it in a centered div container for proper centering
The box now wraps around the image instead of spanning the full width. No linting errors.
---
**User**
now fit it to the same row as the CircleSquareCamera AR demonstration
---
**Cursor**
Placing the Magic School Bus quote figure in the same row as the CircleSquareCamera AR demonstration video:
Updating the layout so the Magic School Bus quote figure appears in the same row as the CircleSquareCamera AR demonstration video:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Updated the layout so the Magic School Bus quote figure appears in the same row as the CircleSquareCamera AR demonstration video.
1. **Created a 2-column grid** — Used the existing `grid cols-2` class to place the video and the quote side by side
2. **Positioned the video on the left** — Takes up more space
3. **Positioned the quote on the right** — Fits its content width (max 324px)
4. **Kept the development note below** — Stays in its own row below both elements
They are now side by side, and the Magic School Bus quote box fits its content. No linting errors. The layout should look balanced with the video and quote on the same row.
---
**User**
make the video and box around 'CircleSquareCamera AR demonstration' the same height as 'Magic School Bus quote'
---
**Cursor**
Making the video container match the Magic School Bus quote box height:
Updating the grid to make both figures the same height:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
---
**User**
the video is now crop cut, i want the full view of the video though....
---
**Cursor**
Removing the flex constraints so the video displays at full size without cropping, and matching the Magic School Bus quote box height to the video:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Removed the flex constraints that were cropping the video. The video now displays at full size and maintains its aspect ratio. The Magic School Bus quote figure will match the video’s height because the grid uses `align-items:stretch`, so both boxes will be the same height while the video shows completely without cropping.
The video will determine the height, and the quote box will stretch to match it. No linting errors. The full video should now be visible.
---
**User**
keep the video back to how it was before we moved it and add the 'Magic School Bus quote
Special thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source' box with the image to the 'Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.' box
---
**Cursor**
Restoring the video to its original layout (standalone) and moving the Magic School Bus quote into the Development Note box:
Restoring the video to its original layout and combining the Magic School Bus quote with the Development Note box:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Made the updates:
1. Restored the video to its original standalone layout — the video is back to its own figure, separate from the quote.
2. Combined the Magic School Bus quote with the Development Note box — they’re now side by side in the same container:
- Development Note text on the left (takes available space)
- Magic School Bus quote image on the right (maintains its size)
The video is standalone again, and the Development Note box now includes the Magic School Bus quote side by side. No linting errors. The layout should work as intended.
---
**User**
the formatting is not nice like this, just move the Magic School Bus quote
Special thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source' box to above the development note box outside of it
---
**Cursor**
Moving the Magic School Bus quote box outside and above the Development Note box:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Moved the Magic School Bus quote box outside and above the Development Note box. The layout is now:
1. **Video** (standalone)
2. **Magic School Bus quote box** (standalone, above the Development Note)
3. **Development Note box** (standalone, below the quote)
Each element is separate, with the quote box centered above the Development Note. No linting errors. The formatting should be cleaner.
---
**User**
move 'Group Assignment: Send a Message Between Two Projects' to after 'Individual Assignment: Networking & Communications'
add a clear link to Ethical AI Use AI-Assisted Camera Integration Game Update in the group assignment section
---
**Cursor**
Finding and reordering the sections, then adding the link to Ethical AI Use:
Reading both sections to understand their structure:
Finding where the Individual Assignment section ends:
Checking where the Individual Assignment section ends:
Extracting the Group Assignment section, moving it after the Individual Assignment section, and adding a link to Ethical AI Use. Starting by extracting the full Group Assignment section:
Moving the Group Assignment section after Individual Assignment and adding a link to Ethical AI Use. Performing this in two steps:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Reviewing the current structure and moving the full CircleSquareCamera AR section:
Moving the CircleSquareCamera AR subsection after the Individual Assignment section and adding the link. Finding where it currently ends:
Fixing the structure: the CircleSquareCamera AR subsection is orphaned. Removing it and properly moving the Group Assignment section after Individual Assignment. Doing this step by step:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Locating the end of the Individual Assignment section and moving the Group Assignment section there. Checking the structure around the Individual Assignment section:
Reading the current Group Assignment section to understand what needs to be moved:
Removing duplicate sections and reorganizing. Removing the duplicate Group Assignment header first:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Design Files
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Finding the orphaned CircleSquareCamera AR content to remove, then adding the complete Group Assignment section after Individual Assignment:
Finding where the Individual Assignment section ends, then adding the Group Assignment section after it. Reading backward from the Design Files section:
Removing the orphaned CircleSquareCamera AR content and adding the complete Group Assignment section after the Individual Assignment section. Finding the orphaned content boundaries:
Finding where the orphaned content ends:
Removing the orphaned content (lines 232-376) and adding the complete Group Assignment section after the Individual Assignment section. Finding the exact boundaries first:
Removing the orphaned content and adding the complete Group Assignment section after the Individual Assignment section. Adding it after Individual Assignment first:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Channel Sharing: TDMA, FDMA, CSMA/CD, CSMA/CA, CDMA, MIMO
CircleSquareCamera Augmented Reality
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Removing the orphaned CircleSquareCamera AR content before the Individual Assignment section:
Removing the orphaned CircleSquareCamera AR content before the Individual Assignment section, since it's now correctly placed after it:
```html
Week 11 · Networking & Communications
This week focuses on connecting devices through wired and wireless networks. We'll explore communication protocols, network addressing, and how to build networked systems with local input and output devices.
Understanding wired and wireless communication protocols
Learning network addressing and bus architectures
Implementing networked nodes with input and output devices
This Week's Goals
Network Protocols Learn about ESP-NOW, Wi-Fi, I²C, SPI, UART, and other communication protocols
Network Addressing Understand MAC addresses, IP addresses, and bus addressing schemes
Networked Systems Design, build, and connect wired or wireless nodes with local I/O devices
Assignments
Group Assignment Send a message between two projects
Individual Assignment Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s)
Tools & Materials
ESP32-S3 Microcontroller Boards
Wi-Fi Access Point
OLED Displays (SSD1306)
Camera Modules
Touch Sensors
Training Documentation
Training materials and documentation for networking and communications protocols.
Recitation Notes
Networking and communications recitation covered ESP-NOW protocol implementation, MAC address-based device identification, peer-to-peer communication setup, and network address configuration for embedded systems.
Networking Protocols Overview
Networking and communications enable devices to exchange data through wired and wireless connections. Understanding different protocols helps select the right communication method for each application.
Wired Communication
UART/USART — Asynchronous serial communication (RS-232, RS-422, RS-485)
I²C/TWI — Two-wire interface with SCL/SDA, requires pull-up resistors
SPI — Serial Peripheral Interface with MISO/MOSI/CS/SCK lines
USB — Universal Serial Bus for device communication
Ethernet — Wired network communication
Wireless Communication
ESP-NOW — Low-power peer-to-peer communication for ESP32 devices
Design, build, and connect wired or wireless node(s) with network or bus addresses and local input &/or output device(s). This assignment demonstrates two networked systems: an ESP-NOW multiplayer game and a Wi-Fi camera streaming system with edge AI processing.
Networked System 1: CircleSquareShapeSongSwingalong (ESP-NOW)
A collaborative multiplayer game featuring geometric shape interactions, synchronized gameplay between two ESP32-S3 devices using ESP-NOW wireless communication. The system uses MAC address-based player identification and real-time state synchronization for coordinated gameplay. See Week 2 documentation and design files for complete details.
Design & Build
The system consists of two XIAO ESP32-S3 development boards, each with an SSD1306 OLED display (128x64) and capacitive touch sensors. The boards communicate wirelessly using ESP-NOW protocol, enabling low-latency peer-to-peer communication without requiring a Wi-Fi access point.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification
Input & Output Devices
Input Devices: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output Devices: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering and player state visualization
Shape Song Swing Along educational content integration and gameplay
Circle and square shape interaction gameplay mechanics
Connections
Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. The ESP-NOW communication is handled entirely through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Networked System 2: Camera Livestream & Edge AI (Wi-Fi)
A Wi-Fi-enabled camera system using ESP32-S3's built-in camera and HTTP server capabilities to stream JPEG frames over Wi-Fi using MJPEG (Motion JPEG) protocol. The system also includes edge AI face detection using a FOMO (Faster Objects, More Objects) model from Edge Impulse for real-time person tracking. See Week 10 documentation and design files for complete details.
Design & Build
The system uses a XIAO ESP32-S3 Sense development board with an integrated camera module. The camera captures frames at QQVGA (160x120) or QVGA resolution, processes them through an Edge AI inference pipeline, and streams the results over Wi-Fi via HTTP multipart response. The system can also display camera output and inference results on an OLED display connected via I²C.
Network Type & Protocol
Network Type: Wireless local area network (Wi-Fi) Protocol: HTTP over Wi-Fi (802.11), MJPEG streaming Addressing Scheme: IP address (DHCP-assigned) and MAC address (hardware identifier)
Input & Output Devices
Input Devices: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, button for capture trigger
Output Devices: OLED display (SSD1306, 128x64, I²C address 0x3C) for displaying camera frames and inference results, Wi-Fi HTTP server for remote streaming
Wi-Fi livestream interfaceCamera feed accessible over Wi-Fi for remote monitoring and control during operation.
Wi-Fi livestream testInitial Wi-Fi livestream test demonstrating remote camera access and feed quality.MIT Wi-Fi livestreamLivestream working on MIT network, confirming network compatibility and stability.Edge AI face detectionReal-time face detection running on-device using edge AI models for person tracking and interaction.
Connections
The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. Wi-Fi connection is established through the ESP32-S3's built-in Wi-Fi radio.
Download Design Files
camera_stream.zip — Complete camera stream project (includes .ino and .h files)
Complete network addressing information for all connected devices, including MAC addresses (hardware identifiers) and IP addresses (network identifiers) for Wi-Fi-connected devices.
ESP-NOW Network (CircleSquareShapeSongSwingalong)
Device
Component
MAC Address
Network Type
Input Device
Output Device
Player 1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Player 2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Wi-Fi Network (Camera Livestream System)
Device
Component
MAC Address
IP Address
Network Type
Input Device
Output Device
Camera Node
XIAO ESP32-S3 Sense
[Device MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
Camera module, Button
OLED (SSD1306, 0x3C), HTTP Server
Client Device
Web Browser
[Client MAC]
192.168.x.x (DHCP)
Wi-Fi (802.11)
N/A
Display (receives MJPEG stream)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. IP addresses are assigned dynamically via DHCP when connecting to a Wi-Fi network. For ESP-NOW communication, only MAC addresses are used (no IP addresses required). The camera system's IP address is printed to the Serial Monitor upon Wi-Fi connection establishment.
Group Assignment: Send a Message Between Two Projects
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols.
Assignment Description
Send a message between two projects. This assignment demonstrates inter-project communication using wired or wireless networking protocols. The CircleSquareCamera Augmented Reality system extends the ESP-NOW multiplayer game from Week 2 by integrating camera boards that receive game state updates and overlay player positions on live camera feeds.
An augmented reality system that combines the ESP-NOW multiplayer game CircleSquareShapeSongSwingalong (documented in Week 2 and Week 11 individual assignment) with camera subsystems from the final project. The system enables game boards to send player state updates via ESP-NOW to camera boards, which overlay game characters (square and circle) on live camera feeds displayed on OLED screens. Camera code development spans Week 7, Week 8, Final Project, and Week 12 (placeholder link).
CircleSquareCamera AR demonstrationCamera boards receive ESP-NOW packets from game boards and overlay player positions (square and circle) on live camera feeds using Floyd–Steinberg dithering for OLED display.
Magic School Bus quoteSpecial thanks to Quentin for this quote featured on the back of the HTMAA XIAO board. Source
Development Note: Initial implementation attempted to send camera data from camera boards to game boards. After two hours of troubleshooting communication issues, the architecture was reversed: the final system sends game state packets from game boards to camera boards via ESP-NOW. This approach proved more efficient for real-time AR overlay visualization, as game state is lightweight compared to camera frame data.
Design & Build
The system consists of two networked subsystems: game boards (XIAO ESP32-S3 with touch sensors and OLED displays) and camera boards (XIAO ESP32-S3 Sense with integrated cameras and OLED displays). Game boards run the CircleSquareShapeSongSwingalong multiplayer game, processing touch input, updating player physics, and broadcasting player state packets via ESP-NOW to both the other game board and all camera boards. Camera boards receive these packets, capture camera frames, process them through Floyd–Steinberg dithering for OLED display, and overlay game character positions (square for Player 1, circle for Player 2) on the live feed.
Network Type & Protocol
Network Type: Wireless peer-to-peer (ESP-NOW) Protocol: ESP-NOW (Espressif's proprietary low-power wireless communication protocol) Addressing Scheme: MAC address-based device identification for game boards and camera boards Communication Pattern: One-to-many broadcast from game boards to peer game board and all camera boards
Input & Output Devices
Game Boards:
Input: Capacitive touch sensors (6 touch pads per device) for player controls (left, right, jump)
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for game rendering
Camera Boards:
Input: Camera module (integrated on XIAO ESP32-S3 Sense) for image capture, ESP-NOW packets for game state
Output: SSD1306 OLED display (128x64, I²C address 0x3C) for displaying dithered camera feed with AR overlay
Connections
Game Boards: Each ESP32-S3 board connects to an OLED display via I²C (SDA pin 5, SCL pin 6) and reads touch sensor inputs from GPIO pins. ESP-NOW communication is handled through the ESP32-S3's built-in Wi-Fi radio, requiring no external hardware connections.
Camera Boards: The camera module is integrated directly on the XIAO ESP32-S3 Sense board, connected via parallel data bus (Y2-Y9), control signals (XCLK, PCLK, VSYNC, HREF), and I²C interface (SIOD, SIOC). The OLED display connects via I²C (SDA pin 5, SCL pin 6) with pull-up resistors. ESP-NOW reception uses the ESP32-S3's built-in Wi-Fi radio in receive-only mode.
Network Address Tables
Complete network addressing information for all devices in the CircleSquareCamera AR system, including MAC addresses (hardware identifiers) for ESP-NOW communication.
ESP-NOW Network (CircleSquareCamera AR System)
Device
Component
MAC Address
Network Type
Input Device
Output Device
P1
XIAO ESP32-S3
D8:3B:DA:75:05:AC
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
P2
XIAO ESP32-S3
D8:3B:DA:75:E1:9C
ESP-NOW (P2P)
Touch sensors (6 pads)
OLED (SSD1306, 0x3C)
Camera A
XIAO ESP32-S3 Sense
B8:F8:62:F9:E2:C0
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Camera B
XIAO ESP32-S3 Sense
B8:F8:62:F9:D6:38
ESP-NOW (Receive)
Camera module, ESP-NOW packets
OLED (SSD1306, 0x3C)
Note: MAC addresses are hardware identifiers unique to each ESP32-S3 device. Game boards broadcast player state packets to both the peer game board and all camera boards. Camera boards operate in receive-only mode, processing incoming ESP-NOW packets to identify player positions and overlay them on camera feeds.
Download Design Files
Complete Arduino code for game boards and camera boards, including ESP-NOW communication, camera feed processing, and AR overlay implementation.
Complete design files for networked systems including ESP-NOW multiplayer game and Wi-Fi camera streaming implementations with Arduino firmware and configuration files.
ESP-NOW Multiplayer Game (CircleSquareShapeSongSwingalong)
Arduino code for ESP-NOW wireless multiplayer game with MAC address-based player identification and synchronized game state management.
Key Features:
ESP-NOW wireless peer-to-peer communication
MAC address-based player identification (D8:3B:DA:75:05:AC and D8:3B:DA:75:E1:9C)
Arduino code for ESP32-S3 camera livestreaming over Wi-Fi using MJPEG protocol and Edge AI face detection using FOMO models from Edge Impulse.
Camera Livestream Pseudocode:
SETUP:
1. Initialize Serial communication (115200 baud)
2. Configure camera pins (from camera_pins.h)
3. Create camera_config_t structure
4. Initialize camera with esp_camera_init()
5. Connect to Wi-Fi network
6. Start HTTP server with stream handler
STREAM_HANDLER:
1. Set HTTP response type to "multipart/x-mixed-replace"
2. Enter loop: capture frame, send via HTTP, repeat
LOOP:
- Minimal delay to allow other tasks
Arduino code for capturing camera frames and streaming them to an OLED display using Floyd–Steinberg dithering for high-quality monochrome rendering. This code demonstrates real-time image processing and display optimization for low-resolution OLED screens.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial communication (115200 baud)
2. Initialize I²C bus for OLED display
3. Initialize OLED display (128x64, I²C address 0x3C)
4. Configure camera pins using camera_pins.h definitions
5. Initialize camera with QQVGA resolution (160x120)
6. Set pixel format to GRAYSCALE
7. Display initialization status on OLED
LOOP:
1. Capture camera frame using esp_camera_fb_get()
2. Process frame through showDitheredPreview():
a. Downsample camera frame (160x120) to OLED resolution (128x64)
b. Average pixel values in each downsampled region
c. Store results in gray_buffer array
d. Normalize brightness values to full range
e. Apply Floyd–Steinberg dithering algorithm
f. Render dithered result to OLED display
3. Return camera frame buffer using esp_camera_fb_return()
4. Repeat continuously for live preview
DITHERING ALGORITHM (Floyd–Steinberg):
For each pixel from top-left to bottom-right:
1. Quantize current pixel (0 or 255)
2. Calculate quantization error
3. Distribute error to neighboring pixels:
- Right: 7/16 of error
- Bottom-left: 3/16 of error
- Bottom: 5/16 of error
- Bottom-right: 1/16 of error
Key Features:
Real-time camera frame capture at QQVGA resolution (160x120)
Grayscale image processing for efficient monochrome display
Floyd–Steinberg dithering algorithm for high-quality visual output
Automatic brightness normalization for optimal contrast
Continuous live preview stream to OLED display
Optimized downsampling from camera resolution to OLED resolution (128x64)
What You Need to Know:
This code provides a foundation for displaying camera feeds on OLED displays. The Floyd–Steinberg dithering algorithm distributes quantization errors across neighboring pixels, creating smooth gradients and improved visual quality compared to simple thresholding. The code uses grayscale format for efficient processing, and the downsampling step averages multiple camera pixels into each OLED pixel to maintain image clarity.
This implementation serves as a building block for the CircleSquareCamera AR system, where camera feeds are combined with game state overlays. See Ethical AI Use section for ChatGPT co-development transcript.
Group Assignment: CircleSquareCamera Augmented Reality System
Complete Arduino code for the CircleSquareCamera AR system, including game board code that broadcasts player states via ESP-NOW, and camera board code that receives game packets and overlays player positions on live camera feeds.
Camera Board Code (CircleSquareCamera.ino)
Camera board receives ESP-NOW packets from game boards, captures camera frames, processes them through dithering, and overlays game character positions (square for P1, circle for P2) on the live feed displayed on OLED.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, I²C, OLED display
2. Initialize ESP-NOW in receive-only mode
3. Register ESP-NOW receive callback
4. Initialize camera (QQVGA, GRAYSCALE)
5. Determine camera ID from MAC address (A or B)
6. Display camera ready status
ESPNOW_RECEIVE_CALLBACK:
1. Check if packet is correct size (NetPacket)
2. Extract player state from packet
3. Identify player (P1 or P2) based on sender MAC address
4. Store player state in p1 or p2 variable
5. Set gotP1 or gotP2 flag
LOOP:
1. Capture camera frame (esp_camera_fb_get)
2. Process frame through makeFrame():
a. Downsample camera (160x120) to 64x32
b. Normalize brightness
c. Apply Floyd–Steinberg dithering
d. Rotate 180° for correct orientation
3. Draw camera feed to OLED (2x2 pixel blocks)
4. Overlay game characters if received:
- Draw square at p1.x, p1.y for Player 1
- Draw circle at p2.x, p2.y for Player 2
5. Display frame on OLED
6. Return camera frame buffer
Key Features:
ESP-NOW receive-only mode for game state packets
MAC address-based player identification (P1: 0xAC, P2: 0x9C)
Game Board Code (TwoSquares_XiaoESP32S3_Touch_ReadySplash_MACFix_Camera.ino)
Extended version of the CircleSquareShapeSongSwingalong game that broadcasts player state packets to both peer game boards and camera boards via ESP-NOW. This enables real-time AR visualization of gameplay on camera displays.
How the Code Works (Pseudocode):
SETUP:
1. Initialize Serial, OLED display
2. Set WiFi mode to STA
3. Read self MAC address
4. Initialize ESP-NOW
5. Identify player role (P1 or P2) from MAC address
6. Add peers: peer game board + Camera A + Camera B
7. Register ESP-NOW send/receive callbacks
8. Initialize touch sensors (warm-up readings)
9. Reset player states (left/right positioning)
LOOP:
1. Update touch sensor readings
2. Process player input (left, right, jump)
3. Update player physics (movement, gravity, collision)
4. Create NetPacket with player state and ready flag
5. Broadcast packet to all peers:
- Other game board (peerMac)
- Camera A (camA_MAC)
- Camera B (camB_MAC)
6. Render game state to OLED:
- Draw ground line
- Draw own character (square if P1, circle if P2)
- Draw other player character
- Display "Waiting Player" if other not ready
7. Delay for frame timing (30ms)
Key Features:
ESP-NOW one-to-many broadcast to game boards and camera boards
MAC address-based player identification (P1: D8:3B:DA:75:05:AC, P2: D8:3B:DA:75:E1:9C)
Touch-based input controls with threshold detection
Ready state synchronization for coordinated gameplay start
Game state rendering with square/circle character representation
What You Need to Know:
This code extends the original CircleSquareShapeSongSwingalong game by adding camera board peers to the ESP-NOW network. The game broadcasts player state packets (position, velocity, onGround flag) to enable AR overlay on camera displays. The camera boards use these packets to draw game characters at the correct positions relative to the camera feed, creating an augmented reality experience. Player identification is done via MAC address comparison, with P1 using square representation and P2 using circle representation. See Ethical AI Use section for ChatGPT co-development transcript.
Key insights and learnings from working with networking and communications protocols.
Contributions
Acknowledgements and team roles for networking and communications work.
ESP-NOW Multiplayer Game
Developed CircleSquareShapeSongSwingalong multiplayer game with ESP-NOW wireless communication during Week 2. See Week 2 documentation for details.
Wi-Fi Camera Streaming System
Developed camera livestreaming and Edge AI face detection system during Week 10. See Week 10 documentation for details.
Ethical AI Use
Transparent documentation of AI assistance used in this week's networking and communications work.
AI-Assisted Camera Integration Game Update
Used ChatGPT for collaborative development of the CircleSquareCamera augmented reality system, integrating ESP-NOW communication between game boards and camera boards. The AI assisted with ESP-NOW packet structure design, camera feed dithering algorithms, AR overlay implementation, and real-time synchronization of player states with camera displays. This extends the original CircleSquareShapeSongSwingalong multiplayer game from Week 2 with augmented reality visualization.
AI-Assisted Week 11 Page Creation and Assignment Documentation
Cursor AI assisted with creating the complete Week 11 documentation page, including networking protocols overview, group assignment CircleSquareCamera AR documentation, individual assignment sections summarizing ESP-NOW multiplayer game and Wi-Fi camera streaming systems, network address tables, and all formatting and styling. The AI helped refine and organize content from Week 2 and Week 10 into professional, succinct networking documentation while preserving key technical details.
The AI was used to:
Create week11.html from the week8.html template structure
Populate networking and communications resources from the MIT Academy class page
Summarize and integrate content from Week 2 (ESP-NOW multiplayer game) and Week 10 (Wi-Fi camera streaming)
Create network address tables with MAC and IP addresses for both networked systems
Format and style all sections for professional presentation
Embed images, videos, and design file download links
```
Checking that the link to Ethical AI Use is in the Group Assignment section:
Verifying the link to Ethical AI Use is in the Assignment Description. Checking that section:
Verifying the section order and that the link is present. Checking where Group Assignment appears:
Checking the section order to confirm Group Assignment is after Individual Assignment:
Checking what's at line 233 to understand the current order:
The Group Assignment section is already after Individual Assignment (Group Assignment starts at line 458, after Individual Assignment ends at line 456), and the link to Ethical AI Use is present at line 469.
## Summary of completed tasks
1. **Moved Group Assignment section** — Already positioned after Individual Assignment section (starts at line 458)
2. **Added link to Ethical AI Use** — Added in the Assignment Description within Group Assignment (line 469), linking to "AI-Assisted Camera Integration Game Update" in the Ethical AI Use section
The page structure is correct, and all requested changes are complete. No linting errors found.
![[Profile Image Placeholder]](profile.jpg)
This work is licensed under a
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License
