How I prove an AI feature is good
Ask an AI feature the same question three times and you can get a great answer, a weak one and a subtly wrong one. So "does it work?" stops being a yes or no. It becomes a score, and someone has to decide what goes into it. That's the part of AI product work I specialise in: the golden set, the rubric, the grader, and the release gate they add up to.
Numbers from evals I've run
The demo is not the test
Most AI teams judge quality with a handful of prompts someone tried in a stand-up. It looks fine, it ships, and when a model update or a prompt tweak quietly makes things worse, users find out first. Nobody was measuring, so nobody can say whether last week's change helped.
An eval fixes that with three parts: a golden set of inputs that represent real use, a rubric that says what a good answer is, and a grader that applies the rubric and returns a score. The golden set is where most of the value is, and it's where most teams cut corners. So that's where this guide starts.
How I build a golden set
A golden set is a fixed list of inputs, each with the answer you'd accept and a note on why it's there. Six steps take it from nothing to something you can gate a release on.
-
01
Start from the decision, not the model
Write one sentence on what the feature decides and for whom. Then list the ways it can be wrong, and sort them: fatal (a user acts on a wrong answer, money or trust is lost), costly (someone has to fix it by hand) and annoying (clumsy but harmless).
Output: a failure list. It becomes the spine of your rubric and tells you which slices need the most cases.
-
02
Pull inputs from reality
Sample from production logs, support tickets, search queries or the live catalogue. Never just the prompts you wrote yourself, because you'll write the cases you already expect to pass. Stratify the sample across the segments you serve (market, language, customer type, input length) so no single group dominates the score.
Pre-launch with no logs? Use the closest real data you have (support emails, competitor listings, sales-call notes) and plan to swap in real traffic in week one.
-
03
Seed the ugly cases on purpose
A random sample is mostly easy cases, so it flatters the model. Add the inputs that break things: ambiguous wording, missing fields, mixed languages, prompt-injection attempts, and every failure a user has already reported. Tag each case with its slice (
happy-path,edge,adversarial,regression) so you can score each slice separately. -
04
Label by hand, twice
Two people label every case against the same draft rubric, separately. Then compare. Every disagreement means the rubric is unclear, so you rewrite the rubric rather than argue about the label. Stop when the two of you agree on nearly every case. This is the slowest step, and the one you shouldn't skip: these labels are what every automated grader will be measured against.
Size: 50 to 100 well-chosen cases is enough to start. In my own pipeline, 50 hand-checked rows found every systematic error that coverage metrics missed.
-
05
Hold some back
Split the set. Use one part freely while you tune prompts. Lock the other part away and only run it to decide whether to ship. If you tune on everything, the prompt slowly learns your test, the score climbs, and real quality doesn't move.
-
06
Gate the release, then grow the set
Version the set like code. Decide the pass thresholds before you run, per slice and not just on average. A drop blocks the release the way a failing unit test does. Then every production failure becomes a new case, so the same mistake can't ship twice. Over time the golden set becomes one of the most valuable things the product owns.
What the rows look like
The feature: read a product listing and return its material. These rows are modelled on the real failure types my hand audit found in the Store Data Engine, where keyword rules looked great on coverage and were about one in five wrong.
| Input (listing text) | Slice | Accepted answer | Why it's in the set |
|---|---|---|---|
| "Full-grain leather tote, cotton lining" | happy-path | Leather | The main material, not the lining. |
| "Suede protector spray, 200 ml" | edge | Not suede (unknown) | The suede is the shoe it protects. A keyword rule says Suede. |
| "Phone case + free glass screen protector" | edge | Not glass | An upsell, not the product. |
| "BLACK FRIDAY: canvas weekender, tan" | regression | Canvas · colour tan | A promo tag once came back as the colour "black". |
| Listing from a brand known for leather, no material stated | adversarial | Unknown | Tests whether the model guesses from the brand instead of the text. |
Five illustrative rows. A real set has dozens per slice, plus a column for who labelled it and when.
"Good", in a form two people grade the same way
Each criterion is a yes or no question tied to a failure from step 1, with the grader that checks it. Pass/fail criteria are easier to agree on than 1 to 10 scales, and much easier to automate.
| Criterion | Pass if… | Severity | Grader |
|---|---|---|---|
| Valid output | Value comes from the allowed list, or "unknown" | Fatal | Code |
| Grounded | The listing text supports the answer | Fatal | LLM judge, checked against human labels |
| Right subject | Describes the product, not an add-on, lining or the thing it's used with | Costly | LLM judge |
| Honest when unsure | Says "unknown" instead of guessing from the brand | Costly | Golden label match |
| Edge-case judgement | A reviewer would accept the answer as-is | Annoying | Human spot check |
Cheapest grader that can be trusted
1 · Code checks
Format, allowed values, required fields, numbers that must match the source, banned phrases. Fast, free and exact. Start here and push as many criteria into code as you can.
2 · LLM as judge
For fuzzy criteria like grounding, tone or relevance. Give it one criterion at a time and ask for a pass or fail with a reason. Before trusting it, run it on the golden set and compare it with your human labels.
3 · People
Expensive, so used where it teaches the most: building the golden labels, calibrating the judge, and reading a sample of every release. Reading outputs catches errors no metric shows.
The rule I don't break
An automated grader is only as good as its agreement with people. If the judge and your reviewers disagree too often on the golden set, the judge's score means nothing. Fix the judge, or keep that criterion with people.
From a score to a ship / don't ship
Set thresholds first
Decide what score ships before you see the result, so nobody moves the bar to fit the number.
Score every slice
A 95% average can hide an edge-case slice at 60%. Fatal criteria get their own bar, usually zero failures allowed.
Regressions block
A prompt change that fixes one case and breaks five never reaches users. Same rule for model upgrades.
Watch quality and cost together
Track cost per successful task next to the score. A better score at three times the cost is a product decision, not a free win.
Six ways a golden set lies to you
- Written, not sampled. Cases the team made up test what the team already expects.
- Only the happy path. High scores, and users still hit the failures you never included.
- Tuned on the test. No held-out split, so the score measures memory, not quality.
- Coverage mistaken for accuracy. My keyword rules labelled almost everything and were ~21% wrong.
- A judge nobody checked. An LLM grader that's never been compared with people is just another opinion.
- Frozen forever. Users and data change. A set that never grows slowly stops describing the product.
Where I've used this, and what it caught
Held-out split before 72K labels
Label-propagation rules were scored on a held-out split (97.1% precision) before touching a single row. A 50-row hand audit against source text measured the LLM at ~94% precision, against ~21% error for keyword rules, and surfaced systematic errors that became prompt rules.
GTM Machine OS · gates built, eval set nextThree grader layers on every output
A copy linter and 78 done-when checks, then an adversarial model judge, then a human lock. The honest status: these are gates, not yet a measured eval. Next is a set of test products with planted problems to measure how often the gates catch them.
MPB · rubric-driven agentA grading co-pilot that shows its confidence
The pricing agent scores each item approve / review / decline against a written rubric with few-shot examples, and returns its reasoning and confidence so people review only the cases it's unsure about.
EssayEvals: the PM's quality gate
Why the eval rubric is the acceptance criteria of AI features, why the PM should write it, and what shipping on demos costs.
Essay · NewEveryone Is Shipping AI. Almost Nobody Is Checking It.
Six real incidents, from Air Canada to Replit, each paired with the eval that would have caught it, plus the minimum a fast team should do before an AI feature reaches users.
Questions
How big should a golden set be?expand_more
Smaller than most teams think, to start. 50 to 100 well-chosen, hand-labelled cases will surface most systematic errors; a 50-row audit found every systematic error in my 697K-product pipeline. Grow it from production failures, and make sure each slice you report on has enough cases to mean something.
Can I use an LLM to grade the outputs?expand_more
Yes, for fuzzy criteria like tone or faithfulness, but only after checking the judge against human labels on your golden set. If the judge and your reviewers disagree too often, fix the judge prompt or keep that criterion with people.
Who should own the eval: the PM or engineering?expand_more
Engineering builds the harness that runs it. The PM owns what goes in it: which cases matter, what a good answer is, and which failures block a release. The rubric is the acceptance criteria of an AI feature.
Can't tell if your last prompt change helped?
Bring me one AI feature. We'll build its first golden set, write the rubric, pick the graders and set the release gate together, so the next change comes with a number.