A test harness for AI, not just for code
catch a worse answer before your customers do
Most teams ship a prompt change the way they would never ship a code change. No test, just a quick manual check and a hope it did not break anything. We build the evaluation harness that scores an AI feature against real examples before it goes live. It is the same discipline behind a seven-channel sales agent with hundreds of automated tests running underneath it.
What the harness actually scores
An evaluation harness is a dataset of real, representative inputs paired with scoring criteria for what a good output looks like. It runs automatically against an AI feature whenever a prompt changes, a model gets swapped, or a new version ships. Instead of a developer reading five outputs and deciding they look fine, the harness scores a meaningful set of cases against defined criteria. It flags a regression before it reaches a real customer.
When guessing stops being good enough
You need this once an AI feature is important enough that a quality regression has a real cost. A sales agent giving wrong pricing information. A content generator producing off-brand copy. A classification task that starts misrouting requests. It is also essential once you are comparing model providers or prompt versions. You need a real answer to “is this better,” not an impression from reading a handful of examples side by side.
You do not need a full harness for a low-stakes, experimental AI feature still being prototyped, where iteration speed matters more than rigor at that stage. The investment makes sense once the feature is heading toward, or already in, production use that real users depend on.
How we build it
The dataset comes first. It has to be built from real or realistic examples specific to your use case, not generic benchmark questions that miss what your feature actually faces. A sales agent’s evaluation set looks nothing like a content generator’s. Scoring criteria get defined per task. A factual question has a clear right or wrong answer that is easy to score automatically. A tone or style judgment often needs a structured rubric, sometimes scored by a second model acting as a judge, sometimes by a human reviewing a sample.
We deliberately include edge cases and adversarial examples, the question someone will eventually ask that the happy path never anticipated. Those are exactly the cases that get skipped when evaluation relies on manual spot-checking. Results get tracked over time, so quality drift from a provider’s silent model update shows up on a dashboard, not in a customer complaint. This is the exact discipline behind a seven-channel sales agent now backed by hundreds of automated tests, built up case by case as real production situations surfaced.
Where evaluation setups quietly rot
An evaluation harness is only as good as its dataset. The most common failure is building it once at launch and never adding to it. Six months later, it stops catching the exact problems that show up in actual production use. We treat the dataset as a living artifact. Every real failure that reaches production becomes a new test case, not just a one-off fix. Automated scoring with a model-as-judge approach carries its own bias and cost. We weigh that against human review for anything where the judgment is genuinely subjective, not factual. Cost of ownership is mostly the discipline of maintaining the dataset. The harness itself, once built, runs cheaply and quickly compared to the cost of a regression reaching real customers undetected.
We also track cost alongside quality in the same harness. A prompt change that improves accuracy by a small margin while tripling token usage is not obviously worth shipping. That tradeoff is easy to miss when quality and cost get measured separately, by different people, on different schedules. A harness that reports both numbers side by side turns that into a visible, deliberate decision rather than a surprise on next month’s invoice.
Price and timeline
| Scope | Price | Timeline |
|---|---|---|
| Core dataset and scoring | from $1,500 | 2 to 3 weeks |
| Multi-model comparison, tracking dashboard | from $3,500 | 3 to 4 weeks |
Related
Built as part of AI agents and custom development. Pairs with an LLM gateway with cost control for comparing providers and prompt and knowledge versioning for tracking what changed. See the testing discipline behind a seven-channel AI sales agent and an AI product card designer bot. Tell us what AI feature needs real quality tracking: get in touch.
FAQ
How much does an evaluation harness cost?
A harness with a solid initial dataset and basic scoring starts at $1,500. A fuller setup with adversarial examples, multi-model comparison and a tracking dashboard runs $3,000 to $6,000.
How long does it take?
2 to 4 weeks. Building the evaluation dataset from real examples takes longer than writing the scoring code, and it is the part that actually determines whether the harness catches real problems.
Can this replace manual review entirely?
No, and it should not try to. An evaluation harness catches regressions and lets you compare options with real numbers. A human still reviews a sample of real production outputs regularly, because some quality problems only show up in actual use.
What is the stack?
A Python-based evaluation framework, sometimes a library like promptfoo, sometimes a custom harness depending on how complex your scoring criteria are. Results get stored and tracked over time, not run once and forgotten.
Who owns the evaluation dataset?
You do. It is built from your own real examples and lives in your repository. That matters because it becomes more valuable over time, as edge cases from production get added to it.