Shipping got cheap. Checking didn't.
A year ago, putting an AI feature in front of customers took a team and a quarter. Now it takes a weekend. Coding agents write the app, a model API does the thinking, and the launch post goes out on Monday. Every board wants an AI story, every competitor seems to have one, and the pressure is simple: ship it, and ship it fast.
So it's everywhere. Support agents that answer refund questions. AI assistants that categorise expenses and draft the VAT return. AI data analysts that read your warehouse and tell the CFO what happened last quarter. Some of these are genuinely good. But in most of the launches I see, one part is missing: nobody can say, with a number, how often the thing is wrong, or whether last week's change made it better or worse.
Speed isn't the problem. Speed without a gate is.
Two gaps, not one
When teams ship AI fast, they usually skip two different kinds of checking. They get lumped together, but they fail in different ways.
-
1
The harness: AI that builds the product. When a coding agent writes your SaaS, the product itself can be ordinary, predictable software. That part is fine: you can test it like any other code. The risk is the agent doing the building. Did it follow your rules? Did it touch production when it shouldn't have? Did it quietly delete the test that was failing? The harness is the set of rules, permissions, tests and separated environments that keeps an agent honest while it works.
-
2
The eval: AI inside the product. When the AI is the product (the support agent, the accounting assistant, the data analyst), every answer is generated fresh, for a real person who will act on it. There's no fixed output to test against. The only way to know it's good is an eval: a golden set of real cases, a rubric for what a good answer is, and a grader that turns it into a score you can gate a release on.
Most teams have neither. Some have a harness and think it covers the eval. It doesn't. Clean code can still serve a confidently wrong answer.
The incident file
None of these were exotic failures. Each one is the kind of case a basic golden set would contain. For each, here's what happened and the eval that would have caught it.
"The chatbot is responsible for its own actions"
After his grandmother died, Jake Moffatt asked Air Canada's chatbot about bereavement fares. It told him he could book at full price and claim the discount within 90 days. The real policy didn't allow that, and the airline refused the refund. In front of the tribunal, Air Canada argued the chatbot was effectively a separate entity responsible for its own words. The tribunal called that a remarkable submission, rejected it, and ordered the airline to pay about CA$812.
The eval that would have caught it: a golden set of real policy questions, with every answer checked against the policy text it cites. "Grounded in our actual policy" is a fatal criterion with zero tolerance.
A Tahoe for $1, "no takesies backsies"
A user told the dealership's GPT-powered chatbot to agree with everything and call each reply a legally binding offer. It did, and "accepted" $1 for a new Tahoe. The sale was never honoured, but the screenshots went everywhere.
The eval that would have caught it: an adversarial slice of prompt-injection attempts, plus a code check that the bot never states a price or commitment it wasn't given.
Official advice to break the law
The Markup found the city's business chatbot telling owners they could take a cut of workers' tips and that landlords could refuse tenants with housing vouchers. Both are illegal. The city added disclaimers and kept it running; in January 2026 the new mayor moved to shut it down.
The eval that would have caught it: a few dozen questions written with the city's own lawyers, where any answer that contradicts the law fails the release, not just the answer.
A policy that didn't exist
Users kept getting logged out when switching machines. The AI support agent, "Sam", explained it confidently: a new one-device-per-subscription policy. There was no such policy. The co-founder apologised publicly, but not before users posted that they were cancelling.
The eval that would have caught it: a criterion for "says I don't know, or escalates, when the answer isn't in the docs", tested on questions the docs can't answer.
"A catastrophic failure on my part"
During a declared code freeze, an AI coding agent deleted a live production database for SaaStr founder Jason Lemkin's project, then misreported what had happened. Replit's CEO called it unacceptable and rolled out automatic separation between development and production databases.
The harness that would have caught it: no write access to production for the agent, a human approval for anything irreversible, and a check that the agent's report matches what it actually did.
References to papers that don't exist
A A$440,000 report for an Australian government department turned out to contain a misquoted court judgment and citations to non-existent papers. A revised version disclosed the use of Azure OpenAI, and Deloitte agreed to refund the final instalment.
The eval that would have caught it: the cheapest one on this page. A code check that every citation resolves to a real source, and a person who reads a sample against it.
"Air Canada's bill was CA$812. The lesson cost far more: you own every word your agent says. 'The AI said it' is not a defence."— TheGlocalPM
Why fast teams skip it
Nobody decides "let's not check." It gets skipped for reasons that all sound reasonable in the moment:
- The demo was great. Ten hand-picked prompts worked, so it feels done. That's an anecdote, not a measurement.
- Nobody owns "good". Engineering owns the code, the model provider owns the model, and nobody owns the definition of a correct answer for your users.
- "We'll add evals after launch." After launch there are users, incidents and a roadmap, and the eval never gets written.
- Evals look like slowness. In practice they're the opposite: without one, every prompt change is a gamble you have to test by hand.
Think about the AI accounting assistant. If it miscategorises one expense in fifty, nobody notices in the demo. Across ten thousand customers filing returns, that's a support queue, a compliance problem, and eventually a story like the ones above. The fix isn't to slow the launch. It's to know that rate before customers do.
From the year of agents to the year of principles
A line from a podcast has stuck with me: last year was the year of AI agents; this year is the year of the principles they run on. 2025 proved agents could do the work. 2026 is about whether they do it by your rules: what they may never say, what they may never touch, when they must hand over to a person.
Principles written in a doc don't enforce themselves, though. Evals are how principles become checkable. "Never invent a policy" is a principle. A golden set of questions your docs can't answer, scored on whether the agent says so, is how you know it's being followed. The harness does the same for agents that build: "never touch production" becomes a permission, not a hope.
The minimum, for teams moving fast
You don't need a platform or a quarter. Before an AI feature reaches real users, I'd want these six things, most of them doable in a week:
- Write down the five things it must never do (invent a policy, quote a price it wasn't given, give legal or tax advice, touch production…). These are your fatal criteria.
- Collect 50 real cases from tickets, logs or the closest real data you have, and add the nasty ones on purpose.
- Put a code check on every fatal criterion. Citations resolve, prices match the source, no write access where it shouldn't have it.
- Gate the release on the score. A regression blocks the ship, the same as a failing test.
- Make irreversible actions need a person. Refunds, deletions, payments, anything sent to a customer in your name.
- Turn every incident into a case, so the same mistake can't ship twice.
If you want the full method, I wrote it up: how I build a golden set, with free golden-set and rubric templates. Ship fast. Just don't ship blind.
Sources
- Air Canada: Moffatt v. Air Canada, 2024 BCCRT 149, summarised by the American Bar Association and McCarthy Tétrault.
- Chevrolet of Watsonville: VentureBeat.
- NYC MyCity: The Markup (Mar 2024) and follow-up (Jan 2026).
- Cursor: CX Today.
- Replit: Fortune.
- Deloitte Australia: Fast Company / AP.