machine week

2023-12-06

We built a machine for producing video clips of a subject simultaneously from multiple angles, and compositing these clips into a single video (set to music).

remote modular things

system diagram (2 concepts)

I wrote a bridge that enables you to centrally control modular things plugged into many different devices from a single instance of the web interface.

The bridge is a python script that you run on as many devices as you want. It talks to a message broker (NATS). When you plug modular things into a device with the bridge running, the bridge discovers the serial connection, announces it on the broker, subscribes to a write topic for the thing, and publishes any data sent by the thing to a read topic.

I adapted a version of the WebSerial connector in the modular-things frontend to connect to the message broker instead of WebSerial and automatically discover devices. You can then use the frontend as normal.

Unfortunately, we're not using this bridge in the live system because I ran out of time to debug some issues (discussed below).

why this approach

The point of this was to get rid of some wires. Currently, all of the modular things need to be plugged into the single computer running the frontend because they have to talk over webserial. This is annoying, and we have raspis on each of the systems anyway to talk to the cameras, so I wanted to just move the usb cables onto the raspis instead.

`socat`

Originally, I wanted to use socat as a point-to-point bridge over TCP. Something like:

# remote (with modular things plugged in)
$ socat -d /dev/ttyACM0,raw,echo=0 TCP-LISTEN:0.0.0.0:1234

# on host running modular things frontend
$ socat -d TCP:remote:1234 PTY,link=/dev/ttyV1,raw,crnl

I was hoping that Chrome would connect to /dev/ttyV1 on the second host, but it seems that the webserial implementation will only connect to devices that the host kernel identifies as having serial drivers, so this doesn't work.

issues

I got the bridge implementation working with a single servo board initially, only really testing the discovery mechanism by plugging and unplugging the thing, and verifying that it showed up and disappeared from the frontend when I did that, without throwing errors in js or python. Discovery requires bidirectional packet transfer between the thing and the frontend, so I figured this would be a good place to start.

This worked fine, but when I added more serial ports, I ran into problems. Specifically, the python bridge started throwing exceptions when more than one serial port was connected in quick succession (e.g. because a USB hub with with multiple things was plugged in). I was pretty sure I had written the bridge in a way that precluded race conditions (the obvious diagnosis for this symptom), but I investigated this and a few other plausible causes.

Retrying the serial connection if it fails is eventually what fixed the problem. I figured there was a deeper issue with my code (not releasing an old port properly was the thread I was chasing for a long time), but based on what I saw, I'm guessing that:

The Linux usb serial driver creates the device node for the port immediately when it sees a cdc-acm device on the bus
The dev node is created in an 'unready' state, causing the exceptions I was seeing in the Python code
There's another configuration step in the driver that occurs after bus discovery, which makes the dev node ready to act as a serial port. This configuration step has some latency and happens serially, port-by-port.

This would explain the behavior and why retries resolve the issue -- a single device won't cause the problem because there's very little latency between the appearance of the device on the bus and the second configuration step, where the dev node is "ready". A bunch of devices at once lead to several of them being unready while devices ahead of them in the queue are configured.

All inference made to fit this set of symptoms, so I don't know for sure if it's right, but the fact that connection retry works makes me pretty confident this actually isn't a bug in my code.

24v interface boards

We wanted the linear axes to move faster than the modular things boards could do with their onboard power supplies. There's an existing design for a 24v power board that would give us this extra speed -- I soldered connectors onto these boards.

Spacers for interposing the 24v boards into the stackup:

Alan 3d printed these spacers and I cleaned them up. This is approximately how they looked stacked up with the modular things boards, though this image has a different spacer part in it:

raspi setup

I helped configure the final raspis. Lancelot reflashed them near the end, and I:

helped him set them up with static IPs
set up my laptop as an optional router with NAT to its wifi network
- this internet access wasn't required for the final working configuration, but was used for installing dependencies
wrote a simple script to install the basic dependencies

static ips

Surprisingly, the static IP configuration had unanticipated problems. I normally only need a static IP on a temporary basis, so I typically go with something like:

$ ip link set dev ethz up
$ ip a add xxx.xxx.xxx.xxx/yy dev ethz

We did this manually to be able to reach them over ssh, then Lancelot made this a systemd unit so they'd have IPs on reboot, but it unfortunately didn't work properly -- two of the three raspis would lose their configuration spontaneously. Most likely the problem was some combination of not having disabled NetworkManager and that if the link ever went down (cable momentarily unplugged), the settings would be lost.

So instead, since NetworkManager was running anyway, we used it:

$ nmcli con mod ethz \
    ipv4.addresses xxx.xxx.xxx.xxx/yy \
    ipv4.gateway www.www.www.www \
    ipv4.dns 1.1.1.1 \
    ipv4.method manual