Evals for voice agents: 103 test cases and an LLM judge on a budget

Voice agents fail differently from chat agents: in real time, out loud, to someone who didn't opt into being a beta tester. Yet most voice deployments I've seen test by calling the bot a few times before launch. I built an open-source eval harness for voice agents, think Promptfoo, but for phone calls, because that testing gap kept producing production incidents.

The harness runs 103 test cases distilled from five production agents. Each case is a conversation scenario with expected behaviors: does the agent handle an interruption mid-sentence, recover when the caller changes their mind, escalate when it hits a question it must not answer, stay coherent when the caller switches language? Real failure modes, harvested from real calls, replayed against every change.

Scoring is an LLM judge with a hard budget guardrail. The judge reads transcripts and scores against rubrics, but eval suites that cost more than the feature get skipped, so the harness enforces a spend ceiling and fails loudly rather than silently burning budget. An eval nobody runs is documentation, not protection.

Two lessons from building it. First, the rubric matters more than the judge: vague criteria produce confident nonsense scores; behavioral, observable criteria ('agent confirmed the date before booking') produce signal. Second, judge drift is real, pin judge model versions and keep a small human-labeled set to calibrate against, or your quality bar moves without anyone deciding it should.

It's Apache-2.0 on my GitHub. If you're shipping voice agents without regression tests, you're one model update away from your agent cheerfully booking appointments on the wrong calendar.