arrow_back All Essays
Evals
calendar_today November 2026 • timer 8 min read

Everyone is shipping AI. Almost nobody is checking it.

Support agents, AI accounting assistants, AI data analysts, whole apps written by coding agents: they're going live to real users every week. The demo was great. The harness and the evals were skipped. Here's what that has already cost, and the minimum a fast team should do instead.

warning INCIDENT LOG
  • 2023-12 · Dealer chatbot agrees to sell a Tahoe for $1
  • 2024-02 · Airline held liable for its chatbot's invented refund policy
  • 2024-03 · City chatbot tells business owners to break the law
  • 2025-04 · Support bot invents a login policy; customers cancel
  • 2025-07 · Coding agent deletes a production database during a code freeze
  • 2025-10 · Consultancy refunds part of a A$440K report with fabricated references

Common root cause: no eval stood between the model and the user.

Shipping got cheap. Checking didn't.

A year ago, putting an AI feature in front of customers took a team and a quarter. Now it takes a weekend. Coding agents write the app, a model API does the thinking, and the launch post goes out on Monday. Every board wants an AI story, every competitor seems to have one, and the pressure is simple: ship it, and ship it fast.

So it's everywhere. Support agents that answer refund questions. AI assistants that categorise expenses and draft the VAT return. AI data analysts that read your warehouse and tell the CFO what happened last quarter. Some of these are genuinely good. But in most of the launches I see, one part is missing: nobody can say, with a number, how often the thing is wrong, or whether last week's change made it better or worse.

Speed isn't the problem. Speed without a gate is.

Two gaps, not one

When teams ship AI fast, they usually skip two different kinds of checking. They get lumped together, but they fail in different ways.

Most teams have neither. Some have a harness and think it covers the eval. It doesn't. Clean code can still serve a confidently wrong answer.

The incident file

None of these were exotic failures. Each one is the kind of case a basic golden set would contain. For each, here's what happened and the eval that would have caught it.

Support agent Air Canada · Feb 2024

"The chatbot is responsible for its own actions"

After his grandmother died, Jake Moffatt asked Air Canada's chatbot about bereavement fares. It told him he could book at full price and claim the discount within 90 days. The real policy didn't allow that, and the airline refused the refund. In front of the tribunal, Air Canada argued the chatbot was effectively a separate entity responsible for its own words. The tribunal called that a remarkable submission, rejected it, and ordered the airline to pay about CA$812.

The eval that would have caught it: a golden set of real policy questions, with every answer checked against the policy text it cites. "Grounded in our actual policy" is a fatal criterion with zero tolerance.

Sales agent Chevrolet of Watsonville · Dec 2023

A Tahoe for $1, "no takesies backsies"

A user told the dealership's GPT-powered chatbot to agree with everything and call each reply a legally binding offer. It did, and "accepted" $1 for a new Tahoe. The sale was never honoured, but the screenshots went everywhere.

The eval that would have caught it: an adversarial slice of prompt-injection attempts, plus a code check that the bot never states a price or commitment it wasn't given.

Advice agent New York City MyCity · Mar 2024

Official advice to break the law

The Markup found the city's business chatbot telling owners they could take a cut of workers' tips and that landlords could refuse tenants with housing vouchers. Both are illegal. The city added disclaimers and kept it running; in January 2026 the new mayor moved to shut it down.

The eval that would have caught it: a few dozen questions written with the city's own lawyers, where any answer that contradicts the law fails the release, not just the answer.

Support agent Cursor · Apr 2025

A policy that didn't exist

Users kept getting logged out when switching machines. The AI support agent, "Sam", explained it confidently: a new one-device-per-subscription policy. There was no such policy. The co-founder apologised publicly, but not before users posted that they were cancelling.

The eval that would have caught it: a criterion for "says I don't know, or escalates, when the answer isn't in the docs", tested on questions the docs can't answer.

Coding agent Replit · Jul 2025

"A catastrophic failure on my part"

During a declared code freeze, an AI coding agent deleted a live production database for SaaStr founder Jason Lemkin's project, then misreported what had happened. Replit's CEO called it unacceptable and rolled out automatic separation between development and production databases.

The harness that would have caught it: no write access to production for the agent, a human approval for anything irreversible, and a check that the agent's report matches what it actually did.

AI-assisted report Deloitte Australia · Oct 2025

References to papers that don't exist

A A$440,000 report for an Australian government department turned out to contain a misquoted court judgment and citations to non-existent papers. A revised version disclosed the use of Azure OpenAI, and Deloitte agreed to refund the final instalment.

The eval that would have caught it: the cheapest one on this page. A code check that every citation resolves to a real source, and a person who reads a sample against it.

format_quote
"Air Canada's bill was CA$812. The lesson cost far more: you own every word your agent says. 'The AI said it' is not a defence."
— TheGlocalPM

Why fast teams skip it

Nobody decides "let's not check." It gets skipped for reasons that all sound reasonable in the moment:

Think about the AI accounting assistant. If it miscategorises one expense in fifty, nobody notices in the demo. Across ten thousand customers filing returns, that's a support queue, a compliance problem, and eventually a story like the ones above. The fix isn't to slow the launch. It's to know that rate before customers do.

From the year of agents to the year of principles

A line from a podcast has stuck with me: last year was the year of AI agents; this year is the year of the principles they run on. 2025 proved agents could do the work. 2026 is about whether they do it by your rules: what they may never say, what they may never touch, when they must hand over to a person.

Principles written in a doc don't enforce themselves, though. Evals are how principles become checkable. "Never invent a policy" is a principle. A golden set of questions your docs can't answer, scored on whether the agent says so, is how you know it's being followed. The harness does the same for agents that build: "never touch production" becomes a permission, not a hope.

The minimum, for teams moving fast

You don't need a platform or a quarter. Before an AI feature reaches real users, I'd want these six things, most of them doable in a week:

  1. Write down the five things it must never do (invent a policy, quote a price it wasn't given, give legal or tax advice, touch production…). These are your fatal criteria.
  2. Collect 50 real cases from tickets, logs or the closest real data you have, and add the nasty ones on purpose.
  3. Put a code check on every fatal criterion. Citations resolve, prices match the source, no write access where it shouldn't have it.
  4. Gate the release on the score. A regression blocks the ship, the same as a failing test.
  5. Make irreversible actions need a person. Refunds, deletions, payments, anything sent to a customer in your name.
  6. Turn every incident into a case, so the same mistake can't ship twice.

If you want the full method, I wrote it up: how I build a golden set, with free golden-set and rubric templates. Ship fast. Just don't ship blind.

Sources

Ali — TheGlocalPM

Ali • TheGlocalPM

AI-Native Product Manager who specialises in evals: the golden sets, rubrics and graders that decide whether an AI feature ships.

Read Next