Week 8: Input Devices
Assignment
This week I was thinking about two things:
1. What if Foodcam was a hat?and probably more interestingly
2. What if a plant and a satellite could dance?I attended a lecture by Benjamin Bratton where he spoke about how the satellites that orbit earth are now creating a kind of figurative skin around the planet. I had a number of what I think are interesting (weird) ideas during the lecture, but I found the thought about satellites and extensions towards planetary computing very evocative. It got me thinking immediately about this week’s assignment.
I haven’t spent much time thinking about satellites previously, but it suddenly felt incredible to me that there are an increasingly large number of advanced technologies hurtling around space orbiting the planet, powering many aspects of our lives, yet we don’t have much of a relationship with them.
It immediately triggered a thought about the contrast of these satellites; advanced human created technological devices, blasted into space and moving at 17,000 mph, and for example a tree or a plant, spawned of nature, utterly grounded, and for all intents and purposes stationary.
Satellites and Plants: an interspecies speculation
One of my research interests is the interactions between species, particularly outside of the human context. As a lot of my work in the past has been about plants, I naturally started thinking about the ways in which a plant and satellite might exchange. What might a plant be able to sense from a satellite?
What kinds of connection, relationship, or exchange might occur between a plant and satellite? Of course, satellites are not a traditional species, but as more and more objects orbit earth, they start to form a new ecosystem there and as technology advances particularly via AI, these objects may in the future be seen in a new light. They may even form relationships with each other or other species.
How could a plant sense a satellite?
Coming back to earth for a moment, I began thinking about the types of exchange that might be possible (broadly speaking): light, electromagnetic, gases? Looking around the CBA shop I didn’t see much that could help me on those fronts, but I did find some GPS modules.
It occurred to me (and confirmed in conversation with Erik) that there might be some way to use a GPS module to establish a connection with satellites. If I could obtain the position of a satellite, in theory it would be possible to move a plant in sync with the satellite, creating a sort of connection, almost a dance between these two greatly estranged and vastly separated beings.
Dancing with Satellites
I liked the metaphor of a dance, especially as immediately I thought about working with a robot arm to mediate and assist an unmoving plant in its potential waltz with a satellite.
After some research I found some hints that there were pathways towards this idea. The general high-level idea was something like this:
- Connect a GPS module to a microcontroller (probably XIAO ESP32) as I’m keen to work with these powerful low-cost boards
- Use a library to obtain information about a satellite (or many satellites) depending on what’s available
- Translate this information into usable coordinates (probably cartesian)
- Use these coordinates to power a robot arm that could support an unmoving plant in its dance with the evermoving satellite
Prototyping:
- I soldered a GT-U7 GPS Module and studied its datasheet; RX, TX, Power, and GND were essentially the core pins. I connected these to a XIAO ESP32S3 Sense, with RX routed to Pin 6 and TX routed to Pin 7.
- I found a great library called GPSPlus and started by running some example code.
- During my first attempts, the GPS didn’t connect at all. When I tried again near a window, it eventually got a signal. The GPS ideally needs to be outside for best performance.
- Even after getting a signal, the example code didn’t run as expected. I had to modify the code significantly to work with an ESP32S3, mainly due to differences between SoftwareSerial and HardwareSerial.
- Initially, despite proper connections, no data appeared in the serial monitor. I found a solution in a forum post, which suggested setting RX and TX pins to -1 (the default setting).
- I also had to set Hardware Serial to 0, instead of 1, as the example code provided.
- Once these changes were made, I could see some data. I started by receiving raw data to ensure everything was functioning:
char c = SerialGPS.read();
Serial.write(c);
This output something like this:
11:52:07.113 -> $GPVTG,,T,,M,0.031,N,0.057,K,A*23
11:52:07.145 -> $GPGGA,155209.00,4221.62747,N,07105.20672,W,1,07,1.04,62.3,M,-33.2,M,,59
11:52:07.241 -> $GPGSA,A,3,23,12,25,28,24,10,21,,,,,,2.23,1.04,1.9700
11:52:07.304 -> $GPGSV,3,1,09,10,84,256,29,12,32,091,41,21,19,309,30,23,47,145,43*7D
- I noticed a variable
gps.satellites.value()in the code. I modified the loop function to print the number of satellites for a simple test. - Exploring the library further, I discovered more data about satellites:
- Satellite ID (PRN number): Identifies the satellite.
- Elevation: Angle above the horizon.
- Azimuth: Direction relative to true north.
- SNR: Signal-to-noise ratio, indicating signal strength.
Azimuth was new to me, so I looked it up:
The azimuth is the angle between the north vector and the star's vector on the horizontal plane, typically measured for celestial coordinates.
#include <TinyGPSPlus.h>
void loop()
{
// Dispatch incoming characters from GPS module
while (SerialGPS.available() > 0)
{
char c = SerialGPS.read();
gps.encode(c); // Pass the GPS data to TinyGPSPlus for parsing
if (totalGPGSVMessages.isUpdated())
{
int connectedSatellites = gps.satellites.value();
Serial.print(F("Connected Satellites: "));
Serial.println(connectedSatellites);
Serial.println(F("Satellite Details:"));
for (int i = 0; i < 4; ++i)
{
int satID = atoi(satNumber[i].value());
if (satID >= 1 && satID <= MAX_SATELLITES)
{
sats[satID - 1].elevation = atoi(elevation[i].value());
sats[satID - 1].azimuth = atoi(azimuth[i].value());
sats[satID - 1].snr = atoi(snr[i].value());
sats[satID - 1].active = true;
}
}
// Print out formatted details for each active satellite
for (int i = 0; i < MAX_SATELLITES; ++i)
{
if (sats[i].active)
{
Serial.print(F("Satellite ")); Serial.print(i + 1);
Serial.print(F(": Elevation=")); Serial.print(sats[i].elevation);
Serial.print(F(" Azimuth=")); Serial.print(sats[i].azimuth);
Serial.print(F(" SNR=")); Serial.println(sats[i].snr);
// Reset the satellite status after printing
sats[i].active = false;
}
}
Serial.println();
}
}
}
The output looked like this:
Connected Satellites: 7
Satellite Details:
Satellite 1: Elevation=84 Azimuth=256 SNR=29
Satellite 2: Elevation=32 Azimuth=91 SNR=41
Satellite 3: Elevation=19 Azimuth=309 SNR=30
Satellite 4: Elevation=47 Azimuth=145 SNR=43
Converting the Output into Something Useful
I started out by trying to:
- Convert positional data into x, y, z coordinates
- Filter the satellites so that I would only get positional data from the nearest satellite each query
- To convert positional data (azimuth and elevation) into x, y, z coordinates, I assumed the GPS receiver as the origin and translated the polar coordinates (azimuth and elevation) into Cartesian coordinates.
To translate the Elevation and Azimuth into a very precise set of coordinates, an important step was defining the distance. However, since it was already Monday evening, I decided to choose an arbitrary unit distance (e.g., 1 unit or 1,000 units). This distance can be scaled later if more precise measurements are needed. It would provide a symbolic way of relating to the satellite's position.
- Convert Angles to Radians:
Since trigonometric functions expect angles in radians, we need to convert azimuth and elevation from degrees to radians.
- Use the Spherical to Cartesian Conversion Formula:
With the distance set to r, azimuth θ (horizontal angle from north), and elevation φ (vertical angle from the horizon), we calculate:
• x = r * cos(φ) * sin(θ)
• y = r * cos(φ) * cos(θ)
• z = r * sin(φ)
With the help of AI, this allowed me to quickly apply the necessary mathematics and rewrite the code to include a new function to get the Cartesian coordinates (x, y, z) while filtering the satellites from a maximum of 40, so I could establish a connection with the closest one, as defined by the strength of the Signal-to-Noise Ratio (SNR).
#include
#include
#include
// Define the RX and TX pins for the GPS module
#define GPS_RX 6 // Connect to TX on GPS
#define GPS_TX 7 // Connect to RX on GPS
// The TinyGPSPlus object
TinyGPSPlus gps;
// The serial connection to the GPS device
HardwareSerial SerialGPS(0); // Use hardware serial channel 1
// The GPGSV fields
static const int MAX_SATELLITES = 40;
//timer fields
unsigned long lastUpdate = 0; // Stores the last update time
const unsigned long updateInterval = 5000; // 5 seconds
// Convert degrees to radians
float degreesToRadians(float degrees) {
return degrees * PI / 180;
}
TinyGPSCustom totalGPGSVMessages(gps, "GPGSV", 1); // $GPGSV sentence, first element
TinyGPSCustom messageNumber(gps, "GPGSV", 2); // $GPGSV sentence, second element
TinyGPSCustom satsInView(gps, "GPGSV", 3); // $GPGSV sentence, third element
TinyGPSCustom satNumber[4]; // to be initialized later
TinyGPSCustom elevation[4];
TinyGPSCustom azimuth[4];
TinyGPSCustom snr[4];
struct
{
bool active;
int elevation;
int azimuth;
int snr;
} sats[MAX_SATELLITES];
void setup()
{
// Start the Serial Monitor for debugging
Serial.begin(9600);
while (!Serial);
// Initialize the GPS hardware serial communication
SerialGPS.begin(9600, SERIAL_8N1, -1, -1);
Serial.println(F("SatelliteTracker.ino"));
Serial.println(F("Monitoring satellite location and signal strength using TinyGPSCustom"));
Serial.print(F("Testing TinyGPSPlus library v. ")); Serial.println(TinyGPSPlus::libraryVersion());
Serial.println(F("by Mikal Hart"));
Serial.println();
// Initialize all the TinyGPSCustom objects for GPGSV data
for (int i = 0; i < 4; ++i)
{
satNumber[i].begin(gps, "GPGSV", 4 + 4 * i); // offsets 4, 8, 12, 16
elevation[i].begin(gps, "GPGSV", 5 + 4 * i); // offsets 5, 9, 13, 17
azimuth[i].begin(gps, "GPGSV", 6 + 4 * i); // offsets 6, 10, 14, 18
snr[i].begin(gps, "GPGSV", 7 + 4 * i); // offsets 7, 11, 15, 19
}
}
void loop()
{
// Dispatch incoming characters from GPS module
while (SerialGPS.available() > 0)
{
char c = SerialGPS.read();
gps.encode(c); // Pass the GPS data to TinyGPSPlus for parsing
// Check if 5 seconds have passed
if (millis() - lastUpdate >= updateInterval)
{
lastUpdate = millis(); // Update the last update time
if (totalGPGSVMessages.isUpdated())
{
int connectedSatellites = gps.satellites.value();
Serial.print(F("Connected Satellites: "));
Serial.println(connectedSatellites);
Serial.println(F("Collecting Nearest Satellites by SNR..."));
Serial.println(F("Nearest Satellite is:"));
// Collect satellite data
for (int i = 0; i < 4; ++i)
{
int satID = atoi(satNumber[i].value());
if (satID >= 1 && satID <= MAX_SATELLITES)
{
sats[satID - 1].elevation = atoi(elevation[i].value());
sats[satID - 1].azimuth = atoi(azimuth[i].value());
sats[satID - 1].snr = atoi(snr[i].value());
sats[satID - 1].active = true;
}
}
// Create an array to store only active satellites
struct SatelliteData
{
int id;
int elevation;
int azimuth;
int snr;
};
SatelliteData activeSats[MAX_SATELLITES];
int activeCount = 0;
// Filter active satellites
for (int i = 0; i < MAX_SATELLITES; ++i)
{
if (sats[i].active)
{
activeSats[activeCount] = {i + 1, sats[i].elevation, sats[i].azimuth, sats[i].snr};
activeCount++;
sats[i].active = false; // Reset for next round
}
}
// Sort satellites by SNR in descending order
for (int i = 0; i < activeCount - 1; i++)
{
for (int j = i + 1; j < activeCount; j++)
{
if (activeSats[i].snr < activeSats[j].snr)
{
SatelliteData temp = activeSats[i];
activeSats[i] = activeSats[j];
activeSats[j] = temp;
}
}
}
// Print details for the 5 strongest satellites by SNR
int maxSatsToDisplay = min(1, activeCount);
for (int i = 0; i < maxSatsToDisplay; ++i){
int satNum = activeSats[i].id;
int azimuth = sats[satNum - 1].azimuth;
int elevation = sats[satNum - 1].elevation;
int snr = sats[satNum - 1].snr;
// Print Cartesian coordinates for this satellite
printCartesianCoordinates(satNum, azimuth, elevation, snr);
// Serial.print(F("Satellite ")); Serial.print(activeSats[i].id);
// Serial.print(F(": Elevation=")); Serial.print(activeSats[i].elevation);
// Serial.print(F(" Azimuth=")); Serial.print(activeSats[i].azimuth);
// Serial.print(F(" SNR=")); Serial.println(activeSats[i].snr);
}
Serial.println();
}
}
}
}
// Convert azimuth and elevation to Cartesian coordinates
void printCartesianCoordinates(int satNumber, int azimuth, int elevation, int snr) {
float r = 1.0; // Assume an arbitrary distance (unit distance)
// Convert azimuth and elevation to radians
float azimuthRad = degreesToRadians(azimuth);
float elevationRad = degreesToRadians(elevation);
// Calculate Cartesian coordinates
float x = r * cos(elevationRad) * sin(azimuthRad);
float y = r * cos(elevationRad) * cos(azimuthRad);
float z = r * sin(elevationRad);
// Print the satellite data in Cartesian coordinates
Serial.print(F("Satellite ")); Serial.print(satNumber);
Serial.print(F(": X=")); Serial.print(x, 3);
Serial.print(F(" Y=")); Serial.print(y, 3);
Serial.print(F(" Z=")); Serial.print(z, 3);
Serial.print(F(" SNR=")); Serial.println(snr);
}
Which gives an output like:
12:57:01.043 -> Connected Satellites: 6
12:57:01.043 -> Collecting Nearest Satellites by SNR...
12:57:01.043 -> Nearest Satellite is:
12:57:01.043 -> Satellite 31: X=-0.760 Y=-0.494 Z=0.423 SNR=25
Creating the dance
Next I wanted to try and see how this data could actually control a Robot Arm. My initial data was quite static, with the same satellite in position for quite a few minutes before changing over (which is nice for an art exhibit but not so much for a demo).
So for the sake of the project demo being more dynamic I decided to change the way the logic worked such that the GPS module would detect X satellites in proximity, and then in the code, we would select at random one of these nearest satellites and display its coordinates.
Knowing that the GPS module did not work currently in the same room as the Robot Arm, I planned to store coordinate data in an array to then send to the Robot Arm and test how it responds and moves.
This was tricker than expected to do correctly. It was also a colder day than expected and I spent quite some time outside trying to get some good coordinate data. The main issues I faced were with writing code that would store positional data in an array for easy use later and when I did manage to do that, the same 4 satellites seemed to always be selected. In the end it turned out it was much more convenient given the time constraints to simply run the code below, without trying to automatically store serial data, and manually scrape the positional information.
#include
#include
#include
#define GPS_RX 6 // Connect to TX on GPS
#define GPS_TX 7 // Connect to RX on GPS
TinyGPSPlus gps;
HardwareSerial SerialGPS(0);
static const int MAX_SATELLITES = 50; // Maximum number of satellites to store
const unsigned long updateInterval = 5000; // 5 seconds
unsigned long lastUpdate = 0;
// Define a structure to hold satellite data
struct SatelliteData {
int satelliteNumber;
float x;
float y;
float z;
};
// Array to hold satellite data
SatelliteData satelliteTable[MAX_SATELLITES];
int entryCount = 0; // Count of stored entries
// Convert degrees to radians
float degreesToRadians(float degrees) {
return degrees * PI / 180;
}
// Other TinyGPSCustom objects
TinyGPSCustom totalGPGSVMessages(gps, "GPGSV", 1);
TinyGPSCustom satNumber[MAX_SATELLITES];
TinyGPSCustom elevation[MAX_SATELLITES];
TinyGPSCustom azimuth[MAX_SATELLITES];
TinyGPSCustom snr[MAX_SATELLITES];
struct {
bool active;
int elevation;
int azimuth;
int snr;
} sats[MAX_SATELLITES];
void setup() {
Serial.begin(9600);
while (!Serial);
SerialGPS.begin(9600, SERIAL_8N1, -1, -1);
// Initialize TinyGPSCustom objects
for (int i = 0; i < MAX_SATELLITES; ++i) {
satNumber[i].begin(gps, "GPGSV", 4 + 4 * i);
elevation[i].begin(gps, "GPGSV", 5 + 4 * i);
azimuth[i].begin(gps, "GPGSV", 6 + 4 * i);
snr[i].begin(gps, "GPGSV", 7 + 4 * i);
}
randomSeed(analogRead(0));
}
void loop() {
while (SerialGPS.available() > 0) {
char c = SerialGPS.read();
gps.encode(c);
if (millis() - lastUpdate >= updateInterval) {
lastUpdate = millis();
// Reset the satellite data when starting new collection
if (entryCount == 0) {
Serial.println(F("Resetting satellite data entries."));
for (int i = 0; i < MAX_SATELLITES; i++) {
sats[i].active = false; // Clear previous states
}
}
if (totalGPGSVMessages.isUpdated()) {
int connectedSatellites = gps.satellites.value();
Serial.print(F("Connected Satellites: "));
Serial.println(connectedSatellites);
Serial.println(F("Currently connected to:"));
// Collect active satellites' data
for (int i = 0; i < connectedSatellites && i < MAX_SATELLITES; ++i) {
int satID = atoi(satNumber[i].value());
if (satID >= 1 && satID <= MAX_SATELLITES) {
sats[satID - 1].elevation = atoi(elevation[i].value());
sats[satID - 1].azimuth = atoi(azimuth[i].value());
sats[satID - 1].snr = atoi(snr[i].value());
sats[satID - 1].active = true; // Mark satellite as active
}
}
// Store only active satellites' data
for (int i = 0; i < MAX_SATELLITES; ++i) {
if (sats[i].active) {
storeSatelliteData(i + 1, sats[i].azimuth, sats[i].elevation, sats[i].snr);
}
}
// Print the stored satellite data only if there's new data
if (entryCount > 0) {
printSatelliteTable();
Serial.println();
}
}
// Clear the satellite data for the next cycle
// Consider changing this to reset only if entryCount is 0
for (int i = 0; i < MAX_SATELLITES; i++) {
sats[i].active = false; // Reset states for the next update cycle
}
}
}
}
// Function to calculate and store satellite data
void storeSatelliteData(int satelliteNumber, int azimuth, int elevation, int snr) {
if (entryCount < MAX_SATELLITES) { // Ensure we do not exceed the array bounds
float r = 1.0; // Assume an arbitrary distance (unit distance)
float azimuthRad = degreesToRadians(azimuth);
float elevationRad = degreesToRadians(elevation);
// Calculate Cartesian coordinates
float x = r * cos(elevationRad) * sin(azimuthRad);
float y = r * cos(elevationRad) * cos(azimuthRad);
float z = r * sin(elevationRad);
// Store satellite data
satelliteTable[entryCount++] = {satelliteNumber, x, y, z};
// Print the satellite data
Serial.print(F("Satellite ")); Serial.print(satelliteNumber);
Serial.print(F(": Position: X=")); Serial.print(x, 3);
Serial.print(F(", Y=")); Serial.print(y, 3);
Serial.print(F(", Z=")); Serial.println(z, 3);
}
}
// Function to print the satellite table
void printSatelliteTable() {
Serial.println(F("Stored Satellite Data:"));
for (int i = 0; i < entryCount; ++i) {
Serial.print(F("Satellite ")); Serial.print(satelliteTable[i].satelliteNumber);
Serial.print(F(": Position: X=")); Serial.print(satelliteTable[i].x, 3);
Serial.print(F(", Y=")); Serial.print(satelliteTable[i].y, 3);
Serial.print(F(", Z=")); Serial.println(satelliteTable[i].z, 3);
}
}
This resulted in the creation of a nice table which gave me a timestamp, satellite number, coordinates (x, y, z) and the SNR. Of course all I really needed for now was a simple array of positional data so I formatted it as follows:
coordinates = [
[0.895, 0.078, 0.438],
[0.731, 0.571, 0.375],
[-0.321, -0.476, 0.819],
[0.087, -0.989, 0.122],
[0.752, 0.146, 0.643],
[0.087, -0.989, 0.122],
[0.895, 0.078, 0.438],
[0.109, 0.884, 0.454],
[0.086, 0.017, 0.996],
[0.109, 0.884, 0.454],
[0.085, 0.021, 0.996],
[0.544, 0.838, 0.052],
[-0.313, -0.464, 0.829],
[0.109, 0.884, 0.454],
[-0.689, 0.714, 0.122],
[0.085, 0.021, 0.996],
[0.752, 0.146, 0.643],
[0.087, -0.989, 0.122],
[0.099, 0.034, 0.995],
[0.099, 0.034, 0.995],
[0.120, 0.980, 0.156],
[-0.689, 0.714, 0.122],
[0.120, 0.980, 0.156],
[-0.688, 0.712, 0.139],
[0.600, 0.797, 0.070],
[0.137, 0.978, 0.156],
[0.894, 0.094, 0.438],
[-0.603, 0.524, 0.602],
[-0.595, 0.517, 0.616],
[0.765, 0.135, 0.629],
[0.086, -0.986, 0.139],
[0.097, 0.039, 0.995],
[0.097, 0.039, 0.995],
[0.768, 0.122, 0.629],
[0.768, 0.122, 0.629],
[0.726, 0.588, 0.358],
[0.731, 0.571, 0.375],
[-0.689, 0.714, 0.122],
[-0.313, -0.464, 0.829],
[0.765, 0.135, 0.629],
[0.765, 0.135, 0.629],
[0.731, 0.571, 0.375],
[0.099, 0.034, 0.995],
[0.109, 0.884, 0.454],
[0.085, 0.021, 0.996],
[0.544, 0.838, 0.052],
[-0.313, -0.464, 0.829],
[0.109, 0.884, 0.454],
[-0.689, 0.714, 0.122]
Dancing in the arms of a robot
In normal circumstances the next step of working with a robot arm would be pretty tricky both in terms of how do you get access to one and also figuring out how to use it.
Fortunately for me, Kye had been experimenting a bit with a robot arm on the 4th floor, so after some coaxing he agreed to help create a first demo of this plant-satellite dance. It was now late Tuesday evening, so in lieu of actually fabricating a holder for a plant (which is still the dream), I decided to feature a video of a plant on the existing screen affixed to the robot arm.
We inputted the array into a program that Kye has been working with to control the arm partly written in python, partly controlled by something called RoboDK.
The First Dance
I found the the first dance between the satellites and the plant quite beautiful. I like how the concept naturally progressed with the constraints, ultimately displaying a sunflower (very on the nose) onscreen dancing with the satellites. The dance itself turned into a dance with multiple partners almost like a flower and pollinators.
Future Plans
I’ve already been in touch with Miana to figure out the robot arm more by myself and will likely spend the upcoming week on that. I think with the right staging this could be a very beautiful piece of art.
Conceptually I find this idea very rich, the project working name for now is Ground Control; stemming from the classic Bowie song Space Oddity which I think would be a very fitting soundtrack to a dance between a satellite and a plant.
I’d love to get more accurate satellite data, and am already looking into types of orbit that could also yield alternative ideas for motion. I’m particularly interested in the ‘flower constellation’ orbit which as the name suggests is in the shape of a flower