Final Project

What Shall I Do

The first thing is that I have thought some things through already during week 1, read that here: Week 1 Final Project Notes

Since then, I have been working on a bunch of projects outside of HTMAA and haven't done much on the BrainJam project. I've realized that maybe I should pull the SVAD project in and do it in HTMAA (which I did in Week 4). The SVAD makes sense because:

Final Project Description

SVAD Wearable: Self-Voice Activity Detector

Why Record Oneself?

HCI research areas of intelligence amplification, memory augmentation, mental health and mood sensing, social behavior monitoring, etc. all heavily dependent on user context to provide contextually relevant information to users. This is often achieved by running continuous audio and/or video recording and processing to allow HCI systems to understand the user's context, intentions, and activities. During in-the-wild pilot studies of these types of systems, the authors came across a number of issues which must be addressed in order to allow the widespread adoption of wearable assistants for intelligence augmentation.

Problem: Privacy

One such problem is the issue of privacy. Systems that continuously process speech signals (i.e. automated speech recognition, ASR) for command receival and context collection present privacy concerns for the individuals near the user. The user may feel comfortable with the privacy and data safety systems of their system, but others, of whom the system records or transcribes, may not agree to have their speech recorded and/or processed. This is further important as some nations and states implement a two-party-consent recording system, whereby all parties in a conversation must agree to recording, transcribing, or any interception of audio or visual sensing of that conversation.

SVAD Wearable: Self-Voice Activity Detector

To solve this problem, we present the SVAD - a voice-activity-detector (VAD) that works only on the voice of the wearer, hence “self voice activity detector”. The SVAD is designed to maximize privacy by never sensing, recording, or processing any speech signal that doesn’t originate from the wearer. The SVAD features a body-worn microphone that is turned off by default. The SVAD uses a body-worn inertial measurement unit to detect when the user is speaking, and only then does it turn on the microphone and stream speech audio data for recording, processing, etc. This system allows for the all-day, continuous capture of speech signals for commands and context whilst eliminating concerns around the privacy of the users fellow interlocutors, as no speech is sensed other than the wearers.

Plan

Great and good plan here to do stuff

New Plan: Alternative Cyborg

OK, outside of this class, I already made the SVAD, it's pretty nice I think:

So now I should do something else. I would like to make a "the alternative cyborg". That is a cyborg tech stack that is completely different than the mainstream, expected tech stack that most cookie-cutter cyborgs don - it consists of a smart ring and a uPhone/ShareableWearable.

The idea of the ShareableWearable/uPhone is that it's designed to benefit other people you're with, not you. "iPhone" is ego driven. "uPhone" is for others. It's a pendant necklace (like the SVAD) but it has a screen that others can see. The smart ring has buttons on the side (3) to control what I see on my smart glasses.

The ShareableWearable goal is to be comfortable and look cool and present text in a way that is easy for the people I'm talking to to read. The smart ring goal is to be comfortable enough, small enough, and low power enough to wear/use all day.



Final Project for Real This Time: uPhone - The Alternative Cyborg

All of our devices today are designed to be used by the owner. All of the apps are device-owner centric. I spent a lot of time wearing smart glasses that give me live AI information during conversations. This can make some people uncomfortable, as they feel a power balance emerging, as I am augmented by my wearable intelligence stack, and they can't see anything that I do. This led me to the idea that we could build wearables which help other people, creating a whole new design space for wearables. This came together as "The Alternative Cyborg Intelligence Stack", which consists of the uPhone and control glove.

The uPhone Smart Pendant

This is a smart necklace pendant which is designed to serve as a screen that others can view. It's always viewable by anyone who is talking to the user. Apps like translation, navigation, link sharing, AI assistans, etc. can be shown to others to help them better understand the device owner, or to help them achieve their goals. This is a case which holds a batter, touch screen full-color display, MCU, and wireless communications in a light and sleek necklace form factor. This is controlled by...

The Control Glove
A smart glove that allows users to make gestures to control the apps that are running on the uPhone display of The Alternative Cyborg Tech Stack. This glove uses hall effect sensors at the tips of the fingers, and magnet in the thumb, with a milled circuit board at the top of the hand, to pick up double tap events of 4 fingers of the hand, double tapping against the thumb. Each double tap sends a different command over bluetooth to the uPhone, which then changes its app to accomodate the command.
Mechanical

The main mechanical challenge was creating a necklace case for the uPhone necklace.

XIAO Display Hat Case

However, this was just an STL, which is near impossible to edit in CAD. So I pulled it into Fusion and made an exact copy of the inner dimensions. That way, my case would fit the display right away, and then I could iterate from there.

I added hooks for a necklace to fit in. I'd already hacked a lot on necklace wearables from the SVAD project (Week 4) so I knew how to do this quite well, and copied that approach.

Then I made the base deeper, which was needed for the lipo battery which was going to be stacked with the whole device. I actually 3D modelled the battery after finding a thin but high capacity battery that would fit within the cirumference of the existing case.

Here is the case:

Electrical

Control glove electrical was a winding road. I originally wanted to use a smart ring, but the parts were not coming in time. Then I tried to use an accelerometer on each finger, but 5 hours of tryig to reflow the ADXL343 accelerometer from a milled PCB didn't work out - I learned from Sam/Anthony after that they also failed to solder this down on a milled PCB. Then I thought to use a PPG module, but it also had a small pitch and needed supporting circuitry I couldn't get in time. Finally, I decided to use hall effect sensors in the fingers, with magents in the thumb, to detect tap events.

I used the ESP32S3 because that was what I was using in the uPhone, and it would easier to connect the two over BLE as there already exist example code for how to do that from Espressif for ESP-IDF.

uPhone electrical was very straightforward as it was COTS components. I plugged the XIAO ESP32S3 into the XIAO Display hat. I soldered on a 400mAH lipo directly onto the leads (the JST didn't fit). I snapped on the UL2 antenna, and the uPhone electical was complete. The ESP32S3 was chosen as I already had a lot of experience with the ESP32 from my time building the Open Source Smart Glasses

Firmware

The firmware of the uPhone consisted of a cobbling together of the Open Source Smart Glasses firmware and the ESP-IDF BlE example code for a GATT client: ESP32 BLE GATT Client.

The firmware of the control glove consisted of a cobbling together of some me + ChatGPT writing of some ESPIDF analog reading and double tap detection, coupled with the ESPIDF example code for a GATT server ESP32 BLE GATT Server.

Textiles

I embedded this all into a glove. The glove I got as cheap as I could on Amazon. All of the electronics were secured to the glove using hot glue.

All Together - The Alternative Cyborg

I love this thing. In review, the glove is too cumbersome, but I think the uPhone presents a whole world of new possiblities in terms of the applications that could be written. Here is the demo video: