Your AI feature got worse, and nobody noticed

Traditional software tends to fail loudly. A broken button throws an error, a test goes red, a page won't load. Someone notices within minutes.

AI features fail differently. The page loads, the answer arrives, it's well written, and it's wrong a little more often than last week. No error, no red test. The first sign is often a customer screenshot.

Why AI regressions are quiet

A model's quality can change even when nobody touches the code that calls it:

• Prompt edits. A sentence added to fix one case can quietly break five others.
• Model upgrades. A newer model is better on average and worse on the cases your product depends on.
• Data changes. New documents in a retrieval index, a retrained classifier, a changed tokenizer.
• Cost and speed tweaks. A smaller, cheaper model or a lower temperature trades quality you didn't measure.

Each of these can pass code review, pass unit tests and ship on time. The ticket says Done. Whether the feature still works as well is a separate question, and usually nobody owns it.

Unit tests can't see it

Unit tests check that code does exactly what it says. AI quality is statistical: it's a score over many examples, and "a bit worse" is still a passing build. What you need instead is an eval: a fixed set of examples, a way to score the answers, and a number you track over time.

Many teams already have evals. The trouble is where they live: a notebook, a script someone runs before big releases, a spreadsheet of scores. The score exists, but it isn't connected to the work. Nobody looks at it when an issue moves to Done.

What catches it: make the score part of Done

The fix is less about better evals and more about where their results go:

1. Agree on a line. For each model, a baseline it must not fall below, for the overall score and for the slices that matter, like refunds or non-English questions.
2. Run evals on every change. In CI on each pull request, and on a schedule for things that change without a pull request, like data.
3. Send the score to where work is tracked. When a change lands below the line, that's a bug, filed with the failing examples, not a number in a log.
4. Hold the work until it passes. The issue isn't Done because the code merged. It's Done when its eval passes.

That last step matters most. It turns quality from something someone checks into something the workflow enforces.

How Evalban does it

This is the idea Evalban is built on. Your eval script sends each run's score with two lines of code. Evalban checks it against the baseline and the slices. If it regressed, it files a bug with the failing examples attached, tells Slack, and keeps the linked issue in an Evaluating column until a run passes. If a failure you fixed comes back later, the old bug reopens.

You keep your eval code and your tools. The difference is that the next quiet regression gets noticed by your board, not your customers.

Try it with the quickstart, or read how to add evals to your pull requests.