An operator console for robots on a space station
Cargo Ops is a browser simulation of the ground side of station robotics: one person on the ground supervising a fleet of free-flying robots over a slow, unreliable link, stepping in by hand when autonomy fails, and turning each fix into training data. These notes explain how it works and why.
A scripted 94-second run: the first arm drills, the map growing, a robot asking for help and the teleop fix, then the build outside. Sped up about 2×.
The operator's job
Icarus's public plan is teleoperation first, then partial autonomy where one person supervises several robots by choosing higher-level commands ("open the bag", "unstow the items"), then full autonomy, with the data from each stage training the next. Cargo Ops plays the middle stage. The robots fly, route, grab and charge on their own. What they can't do yet is know what the crew needs first, or get unstuck. So the person on the ground:
- sets the order of work: a checklist the robots take from, top first. There is no dragging cargo around: every manifest order already says where its bag goes and lands in the checklist with its destination;
- answers assist calls: retry, skip, or take the arms (teleop) when a grasp fails;
- owns safety: an e-stop that has to cross the link like everything else;
- shares the fleet: hands a robot to another console and takes it back, explicitly.
Three clocks and the link
Everything on the ground is old. A command leaves now, the robot acts on it one link delay later, and the operator sees the result one more delay after that. The console shows both clocks on every event-log line.
| Delay | One-way latency L (0.3 to 3 s, default 1.2) with ±10% jitter on every packet. Telemetry frames go down at 20 Hz. |
|---|---|
| Loss of signal | Scheduled windows when no relay satellite is in view (the zone of exclusion, shown on the orbit view). Uplink packets sent into a window are lost. Downlink is buffered onboard and arrives in a burst at the next acquisition of signal, so the map freezes, then jumps. |
| Command life | SENT → ACKED → EXECUTING → DONE, or EXPIRED (no acknowledgement in time) or FAILED (refused onboard). Each command carries an expiry, and the station never acts on a stale one. The ground timeout allows for known LOS windows, so an ACK sitting in the station's buffer is not counted as lost. A late ACK brings an expired command back. |
| Sky | The Earth below is kept plain so it never competes with the station: its tone and the light shift through day, dusk and night once per compressed orbit, following the orbit view. The integrated truss carries the eight solar array wings and the radiators, drawn see-through at a reduced scale. |
| Relay | TDRS by default (what the ISS uses today: near-continuous coverage with gaps). A what-if option models a low-orbit relay mesh: only brief handover gaps and much lower latency. |
Messages
Uplink (ground → station)
| PLAN | A whole checklist in one command, with its LOS policy |
| MOVE | A single transfer, sent now |
| RETRY / SKIP | Answers to an assist |
| TELEOP | Teleop finished, the robot resumes its item |
| TAKE / DRIVE / GRAB / RELEASE | First-person direct control |
| ESTOP / RESUME | Fleet-wide halt and release |
| HANDOFF / TAKEBACK | Lend a robot to another console, take it back |
Downlink (station → ground)
| ACK / NACK | Received; or refused, with a reason |
| EXEC / STEP / DONE | Item assigned to a robot, progress, finished |
| ASSIST / RESOLVED | A robot is stuck and waiting; the answer took effect |
| HOLD / RESUME | What the fleet did when the signal dropped and came back |
| ESTOPPED / RESUMED | Confirmation onboard |
| HANDBACK | A lent robot is back under this console |
| telemetry | Robot pose, state, battery, task, route; cargo; segments |
Learning the controls first
A first visit opens with a short guided tour of the console while four robots run a real checklist; it ends by starting level 1.0. From there every player starts the same way: one robot, JOY-1, in one corner of the station (Harmony, the PMA-2 hatch and a docked Dragon). It gets no work until you can work its arms. Two drills come first:
- Arms, short delay. Free a snagged strap on the teleop bench with the link cut to 0.25 s, to learn the grippers.
- Arms, real delay. Open a zip over the full 1 s link. Move, then wait for the arm.
Then comes the first task, and with each level the map grows: the camera eases out as Unity, Destiny, the Russian segment and Kibo fade in. When nothing has changed for a while and something is waiting on you (a draft not uplinked, a robot asking for help, a refused line, the E-stop left on, a drill step that drags on), the guide points at it. In the first levels the button to press next carries a NEXT tag, and every teleop fix ends on a card that says what was logged.
Seeing what the robot sees
Hover any robot to get a small inset from its forward camera. Double-click it to take direct control: the whole map becomes that camera, with racks, lockers, handrails and hatch frames inside each module, and the image is one link delay old. With hand tracking on, your hands work the map: pinch and move to orbit, pinch with both hands to zoom and turn.
Onboard autonomy
- Allocation. Each checklist item goes to the nearest free robot, in priority order. Robots take the next item as soon as they are free.
- Traffic. Routes are shortest paths over hatches (each module treated as convex). One robot at a time through a hatch, and through a short connector as a whole, with a timeout so nothing deadlocks.
- Waiting, not failing. A move into a full module waits for a slot. An unpack or install waits for its bag. A spacecraft part waits for the one it bolts to.
- LOS policy. A checklist can say continue (robots keep working with no signal) or hold. Single moves always hold.
- Power. Battery drains by activity. Each robot has a charging window, staggered so only one is off the line, and docks early below 18%.
- Teams and EVA. Big parts take two robots who meet, lift and bolt together. Robots leave through the Bishop airlock one cycle at a time and fly slowly outside.
Assists, teleop and the data loop
When a robot gets stuck (a snagged strap, a stiff zip, a fastener that won't seat) it stops and waits for a person. The operator can retry, skip, or open the teleop bench: a close-up of the robot's two arms, driven over the same link. With the camera on, each of your hands drives one gripper (MediaPipe hand tracking, pinch to close), and your mirrored camera image sits faintly behind the arms so your fingers line up with the grippers. Mouse and touch work too, one arm at a time.
- Where you told it to go, and where it is. A cyan ghost gripper follows your hand immediately. The real arm follows a round trip later, because the robot acts on commands one delay old and you see it one delay after that. The tasks are built to teach this: pull a zip without holding the bag and the bag just slides away.
- Grasp assist. Close the gripper near a handle and the robot finishes the reach and grasp itself, the way shared autonomy splits the work: the person aims, the robot does the last few centimetres. Shadows and a pulsing cue ring give depth; a practice mode shortens the delay to 0.25 s.
- Contact is decided on the robot's clock. Grasps, slips and threading are simulated where the robot is. The guidance only reads what the operator can see.
- Each fix is a labelled episode. Robot, failure reason, latency, the command stream and the outcome are recorded and exported with the shift.
- The fleet learns between shifts. At the end of a shift the ground "retrains" and uplinks a new model. Each teleop demonstration of a failure type makes it 40% less likely, a retry 10%, and a skip teaches nothing. The Autonomy panel shows the rates falling across shifts.
The learning rule is deliberately simple. The point is the loop, not the model: operator work becomes structured data, and that data visibly reduces operator work.
Sharing the fleet
Mid-shift, a second console (payload ops) asks to borrow a robot. Only one console commands a robot at a time: approving hands it over through the link, the allocator stops giving it work, and it comes back when the experiment ends. You can take it back early, explicitly, with a confirmation. This comes straight from a system I built and ran in classrooms, where teachers keep control of their headsets until another teacher explicitly takes over.
Modelled, approximate, invented
| Item | Status |
|---|---|
| Which module sits on which port | realISS configuration; zenith and nadir modules drawn beside their port, lengths stretched |
| Module shapes | approximateZvezda's stepped hull and sphere, Prichal's ball, the Cupola's six-sided dome, PMA cones, vehicle silhouettes |
| TDRS relay, zones with no coverage | real conceptTiming compressed so a shift sees four |
| Cargo transfer bags, rack stowage | real conceptCounts and categories invented |
| JOY's two arms and fans | approximateFrom public descriptions; not its real design |
| Speeds, batteries, failure reasons and rates | inventedTuned for play |
| Outpost-1 and the spacecraft | inventedLevels 3.1 and 3.2 |
What I'd build next with a real robot
- A ROS 2 bridge. The console speaks the same messages over rosbridge or Foxglove to a simulated free-flyer (NASA's Astrobee simulator is open source), then to hardware.
- Video in the bench. WebRTC with an explicit latency and bandwidth budget per robot, and the ghost-gripper overlay drawn on the video.
- Episodes in a standard format. MCAP or LeRobot, carrying video, state, operator commands and the assist labels, straight into training.
- Crew and flight rules. Astronauts in the modules, keep-out zones and speed limits near crew, and go/no-go before EVA.
Built with
Plain JavaScript and canvas, three.js r128 for the 3D station and the teleop bench, MediaPipe Hands for hand tracking, and NASA's ISS model behind the briefing. No build step, about 3,200 lines. Six levels in three stages, a guided tour, a quick reference, replay and JSONL export.
Sources for Icarus's public plan: TechCrunch, Silicon Republic, Payload. Station model: NASA. Nick Bisesi, New York · nick@bisesi.dev