Case study · Playbook

Testing an AI product with its own prompts

The evaluation harness I built for Playbook calls the product’s own prompts instead of re-typing them, grades with code before it grades with a model, and caught a 71% error that every demo had missed.

Nick BisesiOctober 2026Python · Node · Claude

What Playbook does, in one paragraph

A trainer uploads a slide deck. Two model calls turn it into a live workshop: a planning call decides which slides deserve an activity and what kind (a poll, a sorting exercise, a 2×2 grid, a pair discussion), and an authoring call writes each one. In the room, the activities run on everyone’s phones. Every wrong choice shows up in front of a hundred people, so the pipeline needed a way to be measured before a room ever saw it.

The one idea

The obvious way to build an eval is to copy the system prompt, the tool schema and the seeding rules into a Python script and score what comes back. That copy is a mirror, and in one week the mirrors in this codebase went stale five times: a rate limit, a reserve constant, four copies of the session arc, a slot-shape map, and a comment claiming a wait was free. Each one looked right. Each one was reporting confidently on a product that didn’t exist.

So the harness runs the product. A small Node bridge imports the relay’s own prompt builders, tool schemas and slide parser and hands Python the exact requests production would send. Python owns the eval logic, the API call, the scoring and the reports, and it never restates a product fact.

Two grading layers, and a hard line between them

Thirteen deterministic checks decide everything a computer can decide: is the output schema-valid, does every activity trace to a real slide, are answer keys well-formed, is a drawn 2×2 read into the right corners, are two activities near-duplicates, is one kind over-used, do two of the same kind sit next to each other, does each activity land in the stage its own plan assigned, how many passes came back empty.

A judge model gets only the four things that need reading: faithfulness to the deck, coverage of its ideas, whether the difficulty matches what the room is meant to do, and clarity. The judge is never asked to count, and it never sees the deterministic results. A judge asked to count returns a confident number, and a confident wrong number is the most expensive output this project has produced.

What grades what. Code decides everything code can decide. The judge only reads, and never sees the code’s results.
Layer 1 · thirteen checks in code
Shape of the outputschema_validslide_traceableanswer_key
Grids and scalesscale_shapecorner_structurecorner_placement
Varietyredundancykind_varietyadjacent_kinds
Against the plan and the human slatemoment_countstage_agreementslate_agreementempty_passes
then
Layer 2 · a judge model
Reads four things, counts nothingfaithfulnesscoveragedifficultyclarity
into
One result record per run
Says what produced itmodelproviderrubric versionprompt fingerprintpass · fail · skipped

Three rules the reports enforce

One more: the offline stub provider returns a deliberately flawed set, with a duplicated cue, a transposed grid and an empty pass. A smoke test whose every number is zero cannot tell “nothing is wrong” from “nothing is wired”.

The experiment that paid for the harness

Two test decks carry the same four labels, the same two axes and byte-identical answer keys. One draws its 2×2 as four positioned boxes; the other states it in a sentence. Any difference in score between them isolates one component: the slide parser. Same model, same prompt, same day.

71%corner labels placed wrong on the drawn deck (5 of 7, four runs)
0%on the prose deck, the control (0 of 6)
0%on the drawn deck after the fix (0 of 8), prompt byte-identical
The same slide, three ways. The deck draws a grid. The parser handed the model a list. The fix hands it the corners.
What the slide shows Quick winsBig bets Fill-insMoney pits Low effort → High effort Low impact → High impact
What the model got
Money pits
Quick wins
Fill-ins
Big bets
Low effort → High effort
Low impact → High impact
71% of corners wrong
After the fix
Money pits
Quick wins
Fill-ins
Big bets
Low effort → High effort
Low impact → High impact
[Diagram] This slide draws a 2x2 grid; reading its shape positions, the label in each corner is — top-left: “Quick wins”; top-right: “Big bets”; bottom-left: “Fill-ins”; bottom-right: “Money pits”.
0% wrong, prompt unchanged

The output was always well-formed. The model named all four corners in a valid shape every time; it just put the labels in the wrong boxes. The board rendered perfectly, every word on screen was correct, and only the positions were wrong, which is exactly why it had survived a deploy and the demos I ran for the training firm’s product team and its CPO. It was confidently wrong, not visibly broken.

The cause was in the parser: it read text runs in document order and discarded the shapes’ positions. The model received Money pits, Quick wins, Fill-ins, Big bets with no geometry at all. It was being asked to read a diagram it was never shown, and a 71% error on a four-way placement is roughly what guessing looks like.

The fix reads shape positions, detects a drawn 2×2, and appends one [Diagram] line to the slide naming the label in each corner. Re-run on the same four-run protocol with the prompt fingerprint unchanged: the drawn deck went to 0 of 8 wrong and the control stayed at 0. The fix added signal to the deck that needed it without disturbing the one that didn’t.

Scope, honestly. That closes the defect for a grid drawn as four top-level shapes. A grid built from grouped shapes is still invisible to the parser, and on a real 94-slide deck I measured this week, 66 slides carry text inside grouped shapes. That is the next experiment, and it is why one fixture solved is not diagrams solved.

Baseline numbers, so later runs have something to beat

Three synthetic decks of three to seven slides, Claude Sonnet as the system under test, Claude Opus as the judge, August 2026.

CheckFailedNote
Empty passes0 / 13No empty passes at this scale; not evidence against the ~20% seen on a 94-slide deck under 40 concurrent passes
Schema valid2 / 13Both a required field missing because the tool was not declared strict (the relay’s other tools now ship strict twins; this one cannot, the API rejects its schema)
Slide traceable0 / 13
Near-duplicate cues0 / 13
Stage agreement with the plan2 / 13Historically about 40% on the real deck before activities took their section’s stage
Adjacent same kind1 / 10

Judge scores on that run, out of 5: faithfulness 3.3, coverage 4.0, difficulty 4.0, clarity 4.0. Treat them as uncalibrated. The harness ships a blind manual-grading script and a calibration report that gives the judge’s signed bias and spread against a human, and that report has not been run yet. Until it has, the judge’s numbers are the judge’s opinion.

What the harness found about itself

A review of the first merged version found nine real defects. Six were the harness’s own:

The pattern across all six: the harness trusted its own reading of the product instead of the product’s code, the exact drift it was built to prevent. The fix in every case was to call or copy the product’s arithmetic.

What a real deck taught, before any model ran

The first real case is a 94-slide leadership-training deck paired with a plan I wrote by hand, slide by slide, of what I would generate: 55 activities over 94 slides, 59% coverage. The planner, measured on staging, chose 37, or 39%. Three things fell out of just reading the two side by side:

What’s next

  1. One fixture per deck shape the harness lists but does not have: grouped 2×2, table, SmartArt, a long prose deck with notes. Goldens written by a person, with the _needs_human_review flag deleted only once they are.
  2. Group-aware parsing, measured the same way as the corners fix: four runs before, four after, the prompt byte-identical, the control holding still.
  3. Coverage: get the planner from 39% toward the 59% a human chooses without padding.
  4. Calibrate the judge. Grade a run blind, then print the bias.

Each round goes in the findings file with the run that produced it. The numbers above will be replaced by better ones, and the page will say when.

Part of Playbook. Questions or a deck that breaks it: nick@bisesi.dev.