Case study · Playbook
Two people or two thousand
Playbook worked for a big room and was useless for two people. Rebuilding it taught me where the real design lives in a live product: not in the network, but in what each person is allowed to see, and when.
What Playbook does
Playbook turns training content into activities a facilitator runs with a room. Each one asks the group a question, and everyone answers on their own phone. Once you’ve answered, you see everyone’s answers live, and you talk about them. It works for a few friends around a table or a room at an event, live or over a week. One link, no app, no account.
The test that broke it
The first version was built for workshops: a trainer, a projector, a room full of people. In October I tried it with one friend and it was useless. I wrote down why the same day:
- The floor of five. Every chart waited until five people had answered, so two people never saw anything but “2 answered.”
- The host couldn’t play. The host’s screen was a remote control. With two people, one answered and the other pressed buttons.
- Ceremony. Every question needed Open, Advance and Reveal. One activity was six steps.
- Setup. An account, a console, a second screen and a code, before anyone answered anything.
The parts underneath were fine: the relay, the live aggregation, the charts. What was wrong was the shape of the session around them. So the rebuild starts with one question, asked when you hit Run: who’s here?
| A few of us | A big room | |
|---|---|---|
| For | friends, a team, a meeting | a class, a workshop, an event |
| Names | shown on every answer | never shown |
| Results | live from the first answer | live once five have answered |
| Host | plays from their own phone | plays, and can put it on a screen |
The rebuild was 55 commits over a day and a half, checked by two scripted walks: the host and one friend in a named room, and the host and four others in an anonymous one. Both pass every step, and the first live result shows up about eleven seconds after Run.
Your answer stops at the server
Everything runs through one Node process with a WebSocket per phone. The rule I built it around: a person’s answer goes to the server and stops there. What leaves is a count, a shape of everyone’s answers, and your own value back to you.
Concretely: your phone sends one value. The server checks it against the question that’s open, stores one value per person (a second answer replaces the first), and sends you an acknowledgement with no value in it. Everyone gets a count. Then the server builds the shape, a histogram or a set of bars or a 2×2, encodes it once, and sends it with your own value attached only to your socket. The session record is written from the same public shape, never from the rows. Free text is kept as a count of words, not the words.
And you only see the room after you’ve answered. That isn’t a client setting: the server doesn’t send the shape to a phone that hasn’t answered, so your answer can’t be steered by everyone else’s.
The floor, and the two ways it leaked
In a big room nothing is shown until five people have answered. A named room has no floor at all, because everyone already knows who’s there. That’s one switch in the aggregation code, set by the question at Run.
The floor sounds simple. It leaked twice before it held:
- By elimination. Under the floor, the server still sent the shape’s empty bins. In a room of four, the bins plus a little arithmetic gave away each person’s answer. Now nothing but the count leaves under the floor.
- By skipping. The floor counted rows, and a skip was a row. One answer and four skips cleared it, and published that one person’s slider to the room. Now it counts answers.
Neither was a networking bug. Both were about what a number lets you infer, which is where I’d look first in any product that shows a group its own data.
Measure the frames, not the latency
The first load test, in September, put 1,500 simulated phones in one room and had them send 1,000 words. Latency looked fine. The problem was elsewhere: every join announced itself to every other phone, which is quadratic. The phones were sent 2.6 million frames they never drew.
| 1,500 phones, 1,000 words | Before | After batching |
|---|---|---|
| Presence frames during the join | 1,125,750 | 2,562 |
| Total frames to phones | 2,628,750 | 246,561 |
| Delivery time, 95th percentile | 183 ms | 121 ms |
| Worst event-loop stall | 216 ms | 52 ms |
The fix was to batch the fan-out, coalesce presence, and stop telling phones about each other at all. The live result now updates at most every 3.2 seconds, and that number comes from the chart’s own animation: it takes 2.8 seconds for a new answer to arrive on screen, and pushing faster than that made phones stutter.
2,500 on one core, and what broke
The next test ran the new protocol with 2,500 simulated people on a slider, the relay pinned to one CPU core.
Three things broke, and they’re the more useful part:
- The card wall. One activity shows everyone’s card. Its shape grows with the room, and the encoder re-encoded it per phone: 75 KB per phone per update and a one-second stall at 2,500. Constant-size shapes are fine; this one is quadratic.
- The reconnect storm. Dropping everyone and letting them back in took 4.7 seconds on a slider and 8.6 on the card wall. Reconnects need jitter, and they don’t have it yet.
- The room cap. A room holds 2,000 connections by default, raised from 300 after the first test. At that default, 501 of the 2,500 were turned away, so a room that size needs the cap raised on purpose.
To be plain about these numbers: they’re from one machine on loopback with simulated clients, not real phones on real networks. The fixes for the card wall and the reconnects aren’t shipped. The largest real session so far is five people. The rebuilt product runs on staging, not production.
What I’d tell someone building the same thing
- Test with two people before you test with two thousand. The big-room test passed for weeks while the product was useless for the group most people actually have.
- Decide what each person may see, and enforce it on the server. Hiding something in the client is not hiding it.
- Count frames. A latency chart can look healthy while you send a million messages nobody uses.
- Write down what breaks. The broken list is the plan for the next round, and it keeps the numbers honest.
Part of Playbook. Related: Testing an AI product with its own prompts. Questions: nick@bisesi.dev.