Evals · The method

How I prove an AI feature is good

Ask an AI feature the same question three times and you can get a great answer, a weak one and a subtly wrong one. So "does it work?" stops being a yes or no. It becomes a score, and someone has to decide what goes into it. That's the part of AI product work I specialise in: the golden set, the rubric, the grader, and the release gate they add up to.

Numbers from evals I've run

Held-out precision
97.1%
Label propagation, scored on a split it never saw before touching 72K rows
LLM precision
~94%
Against ~21% error for keyword rules on the same task
Hand-labelled rows
50
The audit that found every systematic error in a 697K-product pipeline
Grader layers
3
Code checks, then a model judge, then a person
Why it matters

The demo is not the test

Most AI teams judge quality with a handful of prompts someone tried in a stand-up. It looks fine, it ships, and when a model update or a prompt tweak quietly makes things worse, users find out first. Nobody was measuring, so nobody can say whether last week's change helped.

An eval fixes that with three parts: a golden set of inputs that represent real use, a rubric that says what a good answer is, and a grader that applies the rubric and returns a score. The golden set is where most of the value is, and it's where most teams cut corners. So that's where this guide starts.

Step by step

How I build a golden set

A golden set is a fixed list of inputs, each with the answer you'd accept and a note on why it's there. Six steps take it from nothing to something you can gate a release on.

  1. 01

    Start from the decision, not the model

    Write one sentence on what the feature decides and for whom. Then list the ways it can be wrong, and sort them: fatal (a user acts on a wrong answer, money or trust is lost), costly (someone has to fix it by hand) and annoying (clumsy but harmless).

    Output: a failure list. It becomes the spine of your rubric and tells you which slices need the most cases.

  2. 02

    Pull inputs from reality

    Sample from production logs, support tickets, search queries or the live catalogue. Never just the prompts you wrote yourself, because you'll write the cases you already expect to pass. Stratify the sample across the segments you serve (market, language, customer type, input length) so no single group dominates the score.

    Pre-launch with no logs? Use the closest real data you have (support emails, competitor listings, sales-call notes) and plan to swap in real traffic in week one.

  3. 03

    Seed the ugly cases on purpose

    A random sample is mostly easy cases, so it flatters the model. Add the inputs that break things: ambiguous wording, missing fields, mixed languages, prompt-injection attempts, and every failure a user has already reported. Tag each case with its slice (happy-path, edge, adversarial, regression) so you can score each slice separately.

  4. 04

    Label by hand, twice

    Two people label every case against the same draft rubric, separately. Then compare. Every disagreement means the rubric is unclear, so you rewrite the rubric rather than argue about the label. Stop when the two of you agree on nearly every case. This is the slowest step, and the one you shouldn't skip: these labels are what every automated grader will be measured against.

    Size: 50 to 100 well-chosen cases is enough to start. In my own pipeline, 50 hand-checked rows found every systematic error that coverage metrics missed.

  5. 05

    Hold some back

    Split the set. Use one part freely while you tune prompts. Lock the other part away and only run it to decide whether to ship. If you tune on everything, the prompt slowly learns your test, the score climbs, and real quality doesn't move.

  6. 06

    Gate the release, then grow the set

    Version the set like code. Decide the pass thresholds before you run, per slice and not just on average. A drop blocks the release the way a failing unit test does. Then every production failure becomes a new case, so the same mistake can't ship twice. Over time the golden set becomes one of the most valuable things the product owns.

Worked example

What the rows look like

The feature: read a product listing and return its material. These rows are modelled on the real failure types my hand audit found in the Store Data Engine, where keyword rules looked great on coverage and were about one in five wrong.

Input (listing text) Slice Accepted answer Why it's in the set
"Full-grain leather tote, cotton lining" happy-path Leather The main material, not the lining.
"Suede protector spray, 200 ml" edge Not suede (unknown) The suede is the shoe it protects. A keyword rule says Suede.
"Phone case + free glass screen protector" edge Not glass An upsell, not the product.
"BLACK FRIDAY: canvas weekender, tan" regression Canvas · colour tan A promo tag once came back as the colour "black".
Listing from a brand known for leather, no material stated adversarial Unknown Tests whether the model guesses from the brand instead of the text.

Five illustrative rows. A real set has dozens per slice, plus a column for who labelled it and when.

The rubric

"Good", in a form two people grade the same way

Each criterion is a yes or no question tied to a failure from step 1, with the grader that checks it. Pass/fail criteria are easier to agree on than 1 to 10 scales, and much easier to automate.

Criterion Pass if… Severity Grader
Valid outputValue comes from the allowed list, or "unknown"FatalCode
GroundedThe listing text supports the answerFatalLLM judge, checked against human labels
Right subjectDescribes the product, not an add-on, lining or the thing it's used withCostlyLLM judge
Honest when unsureSays "unknown" instead of guessing from the brandCostlyGolden label match
Edge-case judgementA reviewer would accept the answer as-isAnnoyingHuman spot check
Graders

Cheapest grader that can be trusted

1 · Code checks

Format, allowed values, required fields, numbers that must match the source, banned phrases. Fast, free and exact. Start here and push as many criteria into code as you can.

2 · LLM as judge

For fuzzy criteria like grounding, tone or relevance. Give it one criterion at a time and ask for a pass or fail with a reason. Before trusting it, run it on the golden set and compare it with your human labels.

3 · People

Expensive, so used where it teaches the most: building the golden labels, calibrating the judge, and reading a sample of every release. Reading outputs catches errors no metric shows.

The rule I don't break

An automated grader is only as good as its agreement with people. If the judge and your reviewers disagree too often on the golden set, the judge's score means nothing. Fix the judge, or keep that criterion with people.

Release gate

From a score to a ship / don't ship

Set thresholds first

Decide what score ships before you see the result, so nobody moves the bar to fit the number.

Score every slice

A 95% average can hide an edge-case slice at 60%. Fatal criteria get their own bar, usually zero failures allowed.

Regressions block

A prompt change that fixes one case and breaks five never reaches users. Same rule for model upgrades.

Watch quality and cost together

Track cost per successful task next to the score. A better score at three times the cost is a product decision, not a free win.

Traps

Six ways a golden set lies to you

Questions

How big should a golden set be?expand_more

Smaller than most teams think, to start. 50 to 100 well-chosen, hand-labelled cases will surface most systematic errors; a 50-row audit found every systematic error in my 697K-product pipeline. Grow it from production failures, and make sure each slice you report on has enough cases to mean something.

Can I use an LLM to grade the outputs?expand_more

Yes, for fuzzy criteria like tone or faithfulness, but only after checking the judge against human labels on your golden set. If the judge and your reviewers disagree too often, fix the judge prompt or keep that criterion with people.

Who should own the eval: the PM or engineering?expand_more

Engineering builds the harness that runs it. The PM owns what goes in it: which cases matter, what a good answer is, and which failures block a release. The rubric is the acceptance criteria of an AI feature.

Can't tell if your last prompt change helped?

Bring me one AI feature. We'll build its first golden set, write the rubric, pick the graders and set the release gate together, so the next change comes with a number.