I made a wand that draws :)

Basic overview of system

There are 3 main parts to the system:

• A "wand" with an Infrared led attached to the end as well as a button, an rgb led, and bluetooth communication. the button acts as a way to distinguish between pen down and pen up while drawing. The rgb led shines whatever color you set as the pen color in the UI.
• An Infrared camera attached to an esp32s3 with bluetooth communication. The IR camera tracks the LED on the end of the wand and sends a position over bluetooth to the UI
• A web UI that plots the points as the wand is moved around and creates a drawing. Acts as the center for communication with both the wand and the camera

Here is the final product in action

I'll go into depth on each part of these three parts

How I made an IR camera

some googling told me that the xiao esp32 sense cameras come with an IR filter so they block out IR light. But that googling also told me how to remove the filter. I unscrewed the lens, and took tweezer and just chipped it off. Then I put it back on and tried it with the IR led. It could see the led that I couldn't see with my eyes so that was promising.

picture taken with IR Camera, using a light black garbage bag. later iterations used a darker bag.

Then came the challenge of blocking out visible light. I was looking into IR pass filters and Anthony told me that black plastic like a garbage bag actually works for this. I gave it a go and it works quite well.

I CADed a little case for my camera so that it stands upright, is safe-ish from damage, and so I could attach the trash bag filter.

I embossed a logo into it, but I didn't think about the fact that I would be covering that side with a trash bag. silly me.

How I made a drawing application

A lot of the groundwork for this I did in applications and interfaces week. The main additions were working with 2 bluetooth devices simulataneuosly and transmitting the color changing command to the led. I also added some extra features like drawing lines instead of points, being able to change the stroke width, and only drawing when the button was pushed.

You can visit my web app here

How I made a magic wand

The first thing I had to do for the wand was the pcb design. I wanted the wand to be small yet have on device power and last a reasonably long time. I put a 6 pin screw terminal on one end so I could attach both of the leds. (4 pins for rgb 2 pins for ir)

Producing this PCB ending up taking longer than the other pcbs I had made for this class because, at first I used a bad endmill bit and got horrible traces. Then I remilled it and got somewhat better traces but realized that screw terminals have a different hole size than normal connectors. So I remilled it once more. I had to use solder paste for the battery pad and I hadn't really used solder paste before so it took me a while to get it to melt and get the xiao on. Then I soldered everything else on.

Sad board from bad endmill bit

When I started testing, I ran into more problems. It seemed like one of my pins was shorted to ground and I assumed I had messed up something with the solder paste underneath, so I desoldered and resoldered it. But I put it on backwards so I had to do that again. Aaaaaand it was still shorted so I tried again and got it working.

Next came code. The wand code communicates through bluetooth with the website. It receives messages about what color to draw in hex format (#ffffff) and then parses that into the ints and uses pwm to set the led to a custom color (I borrowed a bit of code from 6.08 to do the led pwm coloring). The wand also transmits messages to the UI. When the button is pushed it sends a message to the UI telling it the pen is on and when the button is released it transmits "off"

Designing the packaging for the wand was one of the hardest parts. I wanted it to have a magic wand shape and not be too big. I wanted it to not use screws since that wouldn't look very magical. and I wanted it to be easy to deassemble and reassemble so I could turn it on or off and replace the battery when needed.

I came up with an idea to use threaded pieces that could twist together into one device and hold together pretty well. I also needed a way for the button to be pressed through the packaging, so i made a hole and a little cap thing that would go through the side. I got my hands on wood pla filament since i thought that would look cool for a wand (thanks anthony!).

It took me a few tries to get the sizing exactly right. I did a test print and I discovered that using the prusa printer I didn't need to add any offset to my CADed threads. Unfortunately, I printed one iteration on a bambu and the pieces didn't really twist together. So, I stuck to using Prusas for the rest of my prints

Eventually when I had code I was happy with and packaging made, I started trying to use the battery. turns out that in my times desoldering and resoldering the battery pad was no longer making consistent contact (it would turn on when the xiao was pushed down). Sooooopoooo I had to resolder it again. This time I asked Anthony for some help since I wasn't sure how many more resolders my microcontroller/board could take. Then the battery was working, but my board had a short again. I asked Anthony for help and he scraped out a bit of copper and got it working. Turns out there were some copper dangley bits hanging around which sometimes shorted to ground. It was a frustrating conclusion to a long saga but I was just glad it was finally working.

Assembly process. the large pieces are all threaded so they twist together.

Materials

Item	Cost
Xiao ESP32C3	$4.99
Xiao ESP32S3 Sense	$13.99
Infrared LED	$0.75
RGB LED	$2.25
Battery	$2.50
Total	$24.48

Plus some additional small items from the inventory such as a button, resistors, and a switch

Sources/Inspiration

When my IMU idea wasn't working and I wanted to pivot, I knew that Wii remotes use an IR LED array and an IR camera to detect motion. so I though this could be a good starting point.

I looked on the HTMAA website to see if other people had experiences with IR detection but it seemed no one had done more than just check for presence of lack of presence.

I found answers on various online forums that helped me make an ir camera.

I based my bluetooth code off of example code from the arduino libraries. Specifically Arduino ESP32 BLE's example BLE_notify

Files and Links

• Camera arduino ide code
  • camera.ino
  • bluetooth helper functions
  • image helper functions
  • vector helper functions (this was more useful when I was doing imu stuff)
• Wand arduino ide code
  • camera.ino
  • bluetooth helper functions
  • led helper functions
• Web UI files
  • web ui html
  • bluetooth js helper functions
  • js class for describing bluetooth device
  • js drawing functions
  • C++ so its pretty
• wand cad
• camera case cad
• wand pcb design

Previous iterations

I want to make a magic marker that can draw on any flat surface. I'm thinking of trying to use an IMU with 9 degrees of freedom to keep track of the pens location. Then I'll have some sort of application interface where you can see what is being drawn.

There exist similar products with a drawing pad where you draw on the pad and it shows up on your computer, but the main difference is that I want it to work on any surface.

Basic overview of system

Design Considerations

• Microcontroller: The main challenge I anticipate is speed especially with regard to the imu. I think having a dual core microprocessor could be useful. Position tracking is challenging so my plan is to have one core just running IMU operations. That way it won't drift due to lags. Then the second core can be used for the bluetooth communication, and just read the current position from shared memory. One option for this is the ESP32S3.
• Communication: I need to be able to communicate between my device and my user interface. The two main options are bluetooth or WiFi. I'm leaning towards using bluetooth because I tried to do some WiFi stuff in input devices week and found it a bit cumbersome. The quality of WiFi is really variable and getting the device to connect sometimes took a bit. The variability also made it hard to debug. I am interested in exploring bluetooth in javascript so I will see how that goes. I think in Networking and Communication week, I'll start by trying to work with bluetooth and pivot to wifi if that fails
• Power: I don't want to have to use a wire to power my device, so I need some sort of in device power. When I get to the packaging phase of my project I will need to do some power calculations to figure out what sort of battery I need
• User Interface: I'm pretty familiar with HTML and javascript so I think I'll stick with that. But maybe worth considering alternatives *shrug*

Tasks

• Read IMU Data: connecting an imu to my microcontroller and integrating over acceleration to obtain a rough position
• Transmit IMU Data: using bluetooth to send the marker position to a web ui using bluetooth. This will be my communications and networking week project
• Basic Interface: A web UI to display a drawing based on device position. As they move the marker the drawing will accumulate. At this point I will assume the user is drawing on horizontal plane and choose an arbitrary orientation.
• Choose Drawing Plane: The user can lie the marker along a plane and press a button to indicate this is the plane they want to draw on. Then the user can tap four points with the marker to indicate where they want to corners of their surface to be. Will require some small changes to both how position is computed and how the UI decides canvas size
• UI Improvements: There are a lot of UI improvements I can foresee making. Some of these I might be able to do while working on the basic ui and others might never get done. Some Ideas: changing marker color or size, erasing, drawing quality, etc.
• Packaging: below is a rough idea of what I see the packaging looking like. I'll have to figure out on device power. I will also need to make decisions about how to fit it in as small a space as possible and how to include an antennae for bluetooth.
• Final Touches: Leaving myself time here to make whatever changes I didn't have time for.
• Video and Presentation: I need to make a 1 minute video explaining my project

idea for what the packaging will look like

Updates

• (12/4): the imu is not working as well as I'd hoped. I will play around with it for a bit more but if I can't get it more accurate, I may have to make some changes to my project. I think I could pivot to a camera based led tracking system for drawing, where you would have a camera module and a marker module (with an led on the end) and the same drawing website. Might look into ir leds as well.