Definition
AI evaluation is the practice of testing a probabilistic system against a set of real examples with known-good outcomes, scoring the results, and tracking that score as prompts, models, and data change.
Why "it seems to work" is not a measurement
Conventional software is tested by asserting that a known input produces a known output. Run the test, get pass or fail. That does not transfer to AI, because the same input can produce different phrasings, both of which may be right, or one of which is subtly, expensively wrong.
So AI systems get evaluated the way students do: a set of questions with known-good answers, a grading scheme, and a score you can track over time. Without it you have no way to answer the two questions that matter most. Is this good enough to deploy? And did the change we just made improve it or quietly break something?
That second question is the one that gets teams. Prompt changes are not local. Rewriting an instruction to fix one category of error routinely degrades another, and without evaluation nobody finds out until a customer does.
What to actually measure
The metric depends on the job, and picking the wrong one produces a system that scores well and performs badly.
| Task | What to measure |
|---|---|
| Extraction | Field-level accuracy, and whether errors are flagged |
| Classification | Precision and recall per class, not overall accuracy |
| Retrieval (RAG) | Did the right passage come back at all |
| Summarization | Faithfulness to source, coverage of key points |
| Agents | Task completion, and what it did when it could not finish |
Two cross-cutting measures matter regardless of task. Calibration: when the system is unsure, does it say so? A model that is 90% accurate and knows which 10% is far more useful than one that is 93% accurate and uniformly confident. Failure mode: when it is wrong, is it obviously wrong or plausibly wrong? Plausible errors are the expensive kind.
Building an evaluation set that is worth having
Take real cases from your actual work, not invented examples. Include the awkward ones deliberately: the ambiguous request, the badly scanned document, the customer who asked three things in one sentence, the case your best person had to think about. A set of only easy cases will tell you your system is excellent right up until it meets Tuesday.
Fifty to two hundred well-chosen cases is usually enough to catch regressions. Have a person who knows the work define the correct answer, and write down why it is correct. That reasoning is what lets someone else grade consistently six months later.
Then run it on every change. An evaluation set that is not run automatically stops being run.
How Automathing approaches it
We build the evaluation set before the system, from real cases the client supplies, and we treat the first score as the honest baseline rather than something to be embarrassed about. Every prompt, model, or retrieval change is measured against it. If a change cannot be shown to improve the score, it does not ship. And if the score says the system is not good enough for unattended use, we say that plainly rather than shipping it with a disclaimer.
Frequently asked questions
How accurate does an AI system need to be?
There is no universal threshold. It depends on what a mistake costs and whether you catch it. A system at 85% accuracy that flags its uncertain cases for review can be entirely safe. A system at 97% that acts silently on the remaining 3% may not be. Design the review path first, then ask what accuracy makes that path affordable.
Can we just use the benchmark scores published for models?
They tell you very little about your use case. Public benchmarks measure general capability on standardized tasks; your system's performance depends on your documents, your terminology, your edge cases, and the surrounding engineering. A model that leads a benchmark can underperform a smaller one on your specific job.
How often should evaluation run?
On every change to the prompt, model, retrieval setup, or data. That is what catches regressions. Beyond that, periodically on fresh production cases, because the world drifts: new products, new phrasing, new document formats. A system that was measured once at launch and never since is unmeasured.
Who should define the correct answers?
The person who does the work today, not the team building the system. Builders unconsciously grade toward what the system produces. Whoever owns the outcome should define what "right" means and review disagreements, because their judgment is the standard the system is trying to reach.