Case study · Playbook
Testing an AI product with its own prompts
The evaluation harness I built for Playbook calls the product’s own prompts instead of re-typing them, grades with code before it grades with a model, and caught a 71% error that every demo had missed.
What Playbook does, in one paragraph
A trainer uploads a slide deck. Two model calls turn it into a live workshop: a planning call decides which slides deserve an activity and what kind (a poll, a sorting exercise, a 2×2 grid, a pair discussion), and an authoring call writes each one. In the room, the activities run on everyone’s phones. Every wrong choice shows up in front of a hundred people, so the pipeline needed a way to be measured before a room ever saw it.
The one idea
The obvious way to build an eval is to copy the system prompt, the tool schema and the seeding rules into a Python script and score what comes back. That copy is a mirror, and in one week the mirrors in this codebase went stale five times: a rate limit, a reserve constant, four copies of the session arc, a slot-shape map, and a comment claiming a wait was free. Each one looked right. Each one was reporting confidently on a product that didn’t exist.
So the harness runs the product. A small Node bridge imports the relay’s own prompt builders, tool schemas and slide parser and hands Python the exact requests production would send. Python owns the eval logic, the API call, the scoring and the reports, and it never restates a product fact.
Two grading layers, and a hard line between them
Thirteen deterministic checks decide everything a computer can decide: is the output schema-valid, does every activity trace to a real slide, are answer keys well-formed, is a drawn 2×2 read into the right corners, are two activities near-duplicates, is one kind over-used, do two of the same kind sit next to each other, does each activity land in the stage its own plan assigned, how many passes came back empty.
A judge model gets only the four things that need reading: faithfulness to the deck, coverage of its ideas, whether the difficulty matches what the room is meant to do, and clarity. The judge is never asked to count, and it never sees the deterministic results. A judge asked to count returns a confident number, and a confident wrong number is the most expensive output this project has produced.
Three rules the reports enforce
- Absent is not zero. A check that could not run reports skipped, never a 0% failure rate. A corner-placement check on a deck with no grid did not pass; there was nothing to get wrong. A report that prints those the same turns “never tested” into “works”.
- No overall score. A mean would let a 5 on clarity hide a 1 on faithfulness, and those are not tradeable. A beautifully phrased question about something the deck never said is worse than a clumsy one that is true.
- Every result says what produced it. Model, provider, rubric version and a fingerprint of the live prompt and tool schema. The comparison script refuses to call a difference attributable when any of those moved. Without that, “the model got worse” and “somebody edited the prompt on Tuesday” are the same number.
One more: the offline stub provider returns a deliberately flawed set, with a duplicated cue, a transposed grid and an empty pass. A smoke test whose every number is zero cannot tell “nothing is wrong” from “nothing is wired”.
The experiment that paid for the harness
Two test decks carry the same four labels, the same two axes and byte-identical answer keys. One draws its 2×2 as four positioned boxes; the other states it in a sentence. Any difference in score between them isolates one component: the slide parser. Same model, same prompt, same day.
Money pits Quick wins Fill-ins Big bets Low effort → High effort Low impact → High impact71% of corners wrong
Money pits Quick wins Fill-ins Big bets Low effort → High effort Low impact → High impact [Diagram] This slide draws a 2x2 grid; reading its shape positions, the label in each corner is — top-left: “Quick wins”; top-right: “Big bets”; bottom-left: “Fill-ins”; bottom-right: “Money pits”.0% wrong, prompt unchanged
The output was always well-formed. The model named all four corners in a valid shape every time; it just put the labels in the wrong boxes. The board rendered perfectly, every word on screen was correct, and only the positions were wrong, which is exactly why it had survived a deploy and the demos I ran for the training firm’s product team and its CPO. It was confidently wrong, not visibly broken.
The cause was in the parser: it read text runs in document order and discarded the shapes’ positions. The model received Money pits, Quick wins, Fill-ins, Big bets with no geometry at all. It was being asked to read a diagram it was never shown, and a 71% error on a four-way placement is roughly what guessing looks like.
The fix reads shape positions, detects a drawn 2×2, and appends one [Diagram] line to the slide naming the label in each corner. Re-run on the same four-run protocol with the prompt fingerprint unchanged: the drawn deck went to 0 of 8 wrong and the control stayed at 0. The fix added signal to the deck that needed it without disturbing the one that didn’t.
Scope, honestly. That closes the defect for a grid drawn as four top-level shapes. A grid built from grouped shapes is still invisible to the parser, and on a real 94-slide deck I measured this week, 66 slides carry text inside grouped shapes. That is the next experiment, and it is why one fixture solved is not diagrams solved.
Baseline numbers, so later runs have something to beat
Three synthetic decks of three to seven slides, Claude Sonnet as the system under test, Claude Opus as the judge, August 2026.
| Check | Failed | Note |
|---|---|---|
| Empty passes | 0 / 13 | No empty passes at this scale; not evidence against the ~20% seen on a 94-slide deck under 40 concurrent passes |
| Schema valid | 2 / 13 | Both a required field missing because the tool was not declared strict (the relay’s other tools now ship strict twins; this one cannot, the API rejects its schema) |
| Slide traceable | 0 / 13 | |
| Near-duplicate cues | 0 / 13 | |
| Stage agreement with the plan | 2 / 13 | Historically about 40% on the real deck before activities took their section’s stage |
| Adjacent same kind | 1 / 10 |
Judge scores on that run, out of 5: faithfulness 3.3, coverage 4.0, difficulty 4.0, clarity 4.0. Treat them as uncalibrated. The harness ships a blind manual-grading script and a calibration report that gives the judge’s signed bias and spread against a human, and that report has not been run yet. Until it has, the judge’s numbers are the judge’s opinion.
What the harness found about itself
A review of the first merged version found nine real defects. Six were the harness’s own:
- The answer-key check validated the key as choice text; the live schema says a 1-based index. Every correct key was being failed, and the unit tests fed it fixtures the schema never produces.
- The pipeline scored the model’s raw output. Production normalizes, rebalances and focuses the set first, deleting neighbour-slide cards and keeping one per pass. The harness now calls those same functions.
- The kind-share check failed a strict 20% share; the product caps a rounded-up count. Two of seven is within budget, so one finding was retracted.
- The stub keyed its answer to the first slide in the prompt, which is the focus window’s leading neighbour, not the slide under test.
The pattern across all six: the harness trusted its own reading of the product instead of the product’s code, the exact drift it was built to prevent. The fix in every case was to call or copy the product’s arithmetic.
What a real deck taught, before any model ran
The first real case is a 94-slide leadership-training deck paired with a plan I wrote by hand, slide by slide, of what I would generate: 55 activities over 94 slides, 59% coverage. The planner, measured on staging, chose 37, or 39%. Three things fell out of just reading the two side by side:
- Pacing follows content, not intervals. The human plan has six-activity runs and five quiet stretches. Even spacing is not the goal.
- The spacing rule fights deliberate design. The human plan puts the same activity on two consecutive slides on purpose, a two-part exercise. The rebalance pass would swap one. It now breaks only runs of three or more.
- Zero speaker notes. The notes channel the prompts leaned on was empty on a real deck.
What’s next
- One fixture per deck shape the harness lists but does not have: grouped 2×2, table, SmartArt, a long prose deck with notes. Goldens written by a person, with the
_needs_human_reviewflag deleted only once they are. - Group-aware parsing, measured the same way as the corners fix: four runs before, four after, the prompt byte-identical, the control holding still.
- Coverage: get the planner from 39% toward the 59% a human chooses without padding.
- Calibrate the judge. Grade a run blind, then print the bias.
Each round goes in the findings file with the run that produced it. The numbers above will be replaced by better ones, and the page will say when.
Part of Playbook. Questions or a deck that breaks it: nick@bisesi.dev.