No prompt change ships
until it clears the same test set every time
Changing a prompt or switching a model is easy to do and hard to judge. It feels better on the three examples someone tried, and nobody finds out it got worse on a case that matters until a customer hits it. A prompt and model evaluation agent runs a fixed set of real, recorded cases against every version before it ships, scoring accuracy, hallucination rate and tone. It shows the comparison plainly instead of a gut feeling.
Why “it feels better” is not a result
A team tweaks a prompt, tries it on a few examples, it looks better, and ships it. What that quick check rarely catches is the case that mattered six weeks ago, the edge case that used to work and now quietly does not. Nobody re-ran the full set of scenarios the original prompt was built to handle.
Comparing two model versions, or two providers, by feel is unreliable in both directions. A version can feel better because the few examples tried happened to favor it, while actually performing worse on the cases that show up most in real traffic. Without a fixed eval set, “it works better now” is not a claim anyone can check later. That makes every prompt change a one-way decision nobody can confidently roll back from or build on.
What the agent checks before a version ships
The agent runs a fixed set of real, recorded cases, actual conversations or scenarios your agent has handled, against every candidate prompt or model version. This happens before it goes anywhere near live traffic. It scores each version against a rubric covering accuracy, hallucination rate, and tone, the same measures we use on our own agent builds. The result is a clear side-by-side comparison, not a single pass or fail.
Every run is logged with the exact prompt and model version it tested. A regression is traceable to the exact change that caused it, not just noticed after the fact.
Typical scope: evaluating prompt changes, model version upgrades, and provider comparisons before anything ships to live traffic. Writing the rubric itself, what counts as a good answer for your specific use case, is a joint step done with your team.
What stays with humans
Writing the scoring rubric, and deciding what a good answer actually looks like for your business, is your team’s call, built with us but owned by you. Judgment on a genuinely ambiguous case, one a human reviewer would also debate, gets escalated rather than scored unilaterally. Deciding whether a version is good enough to ship stays with your team, informed by the eval, not replaced by it.
Guards
No version goes to live traffic until it clears the eval bar your team sets. The eval set is kept separate from anything used to tune the prompt itself, to avoid testing against the same cases a version was optimized for. Every eval run is logged with the exact prompt and model version, fully traceable after the fact.
Price and timeline
| Option | Price | What it covers | Timeline |
|---|---|---|---|
| Agency runs it | from $2,200 + support plan | Eval pipeline built and run by us, reviewed before every version ships | 2 to 3 weeks |
| Full control, handover-ready | from $3,200 | Same pipeline on your own infrastructure, documented eval set, your team runs it | 3 to 4 weeks |
Running cost is usually $15 to $60 a month in model usage, depending on eval set size and run frequency.
Related
See the AI agents service page and development for the surrounding build. Within this group: the critic and verifier agent applies similar scrutiny to live output, and the QA and test agent and synthetic data agent cover adjacent testing ground. For a related one-time setup, see automate synthetic test data generation and automate agent cost and quality monitoring. The seven-channel AI sales agent case study was tested against real recorded conversations before launch. The two-brand analytics hub case study shows the same discipline at warehouse scale.
Shipping prompt changes on a feeling rather than a number? Get in touch and we will look at what you would test it against first.
FAQ
How much does a prompt and model evaluation agent cost?
From $2,200 to build an eval set and test pipeline for one agent, live in 2 to 3 weeks. A larger test set or evaluation across multiple agents usually runs $3,500 to $5,500.
How long before it is evaluating real versions?
2 to 3 weeks. Building a representative eval set from your real conversations or cases takes most of it. The eval is only as good as the cases it is tested against.
Which models and prompts does it work with?
Any model accessible through an API: Claude, GPT, Gemini, and any prompt version you want compared. It is model-agnostic by design, useful specifically when you are deciding between providers or versions.
What if the eval itself is wrong, or a case is genuinely ambiguous?
Borderline cases where even a human would disagree are flagged for your team to decide. They get added to the rubric, rather than scored by the agent's own judgment alone.
Does this replace a human reviewing agent outputs?
No. It replaces guessing whether a change made things better or worse before shipping. A human still writes the rubric and decides what the eval set should contain, and still reviews flagged borderline cases.