Final Project Overview

final

Motivation

Have you ever realized you left something behind but were too far along to turn back? I once nearly missed a flight because I left my passport at home. Other times, I’ve forgotten to buy essential ingredients and wished I had a way to quickly record what I need in the moment. To prevent situations like these, I wanted to create a reminder system at the door that tells me exactly what I need based on where I’m going. Also, forgetting something at door is a real psychological effect; Doorway effect is a phenomenon of short-term memory loss when passing through a doorway or moving from one location to another.


I'm excited about the potential of this project, as I haven't seen any existing door-based reminder systems, and it closely relates to my research on memory-augmented systems for older adults! For example, the system could be used not only for reminders to buy and bring objects but also for safety alerts. By placing audio sensors in various locations around the home, the system can track human activity in places like the kitchen and provide reminders to turn off the stove. If an older adult with early signs of dementia is leaving the house at night and goes missing, caregivers can also check the destination log to provide support.

System Outline

final

  1. Kitchen ESP32: Sends recorded reminders to both Google API and the web server.
  2. Door ESP32: Retrieves the reminder data from the web server and displays it on the OLED screen when proximity is detected.

How it Works

final

  1. Sensors around the house: The ESP32 collects audio data in the kitchen, sends it to Google’s Speech-to-Text API, and receives the converted text (e.g., “Buy milk”).
  2. Audio-to-text conversion: The system listens to sound input, and sends to web server via HTTP POST request. Then the web server converts the audio to text, stores in a database, and uses language models to interpret and organize different inventory items based on their locations.
  3. Door handle module: A main board on the door handle uses an accelerometer to detect door-opening motion. When triggered, it prompts you to confirm where you’re going and sort out items to buy based on your destination.

Current Status & Project Plan

I used most of the input device & output device week to design something that is directly related to my final project. In the meantime, I also looked into different modules that can include audio input and output. Also, I found helpful youtube tutorial to use ChatGPT/LLM for understanding human language and providing outputs!

Awesome TA Marcello also helped me to order some of the componets that would be helpful for my project.

final
final