Catching It First: Evaluating Insurance AI Agents at Scale

When a Failure Looks Like a Normal Conversation

A bug in ordinary software announces itself with a stack trace, an exception, a red line in a log. A failure in a production AI agent usually doesn't announce itself at all. The agent hands the caller a claim number that doesn't exist, or skips a required transfer because the request was phrased in an unusual way, or keeps talking after a tool call to the policy system quietly errors out. Read the transcript afterward and it looks like every other conversation that day. The agent sounded fluent, polite, and confident, and it was also wrong.

That's the core problem with evaluating agents in production. The most expensive failures are the ones that look fine, and they tend to multiply at scale. We run 2.8 million automated interactions a month across 70+ enterprise customers, on insurance operations representing more than $100B in premium. At that volume, the idea of a person reviewing conversations one by one to catch problems isn't a workflow; it's a fantasy. By the time a systemic failure surfaces because a customer complained, it has already happened thousands of times.

The question wasn't whether our agents were performing well. It was whether we had the operational intelligence to detect coaching opportunities and compliance or service issues before they resulted in a poor customer experience. 

Generic Evals Are Too Coarse to Matter

The obvious move is to reach for off-the-shelf evaluation. You run every conversation through a generic hallucination check and an instruction-following check, reduce each call to a quality score, and watch those scores over time. We tried it, and it doesn't go far enough.

Knowing that an agent "hallucinated" somewhere in a 15-minute claim call doesn't tell you the thing you actually need to act on, because the failures that matter in insurance are specific. A few of the details we watch for:

• Was the cause of loss coded correctly? A burst supply line with standing water coded as general property damage instead of sudden and accidental water may not trigger the automation that assigns emergency mitigation. The claim sits in queue while the risk of mold grows and the policyholder waits for someone to notice what the system would have dispatched in minutes.
• Did a claim involving bodily injury get recognized as one? If a bodily injury (BI) exposure isn't opened, the claim routes to a property-damage adjuster who won't know which injury questions to ask during the recorded statement, and carriers typically take only one recorded statement per person per claim. The BI adjuster doesn't get a second chance. If the injured party can't be reached again, fields required for a Medicare Secondary Payer (MSP) query, like SSN and date of birth, may never be captured, creating a compliance gap.
• Does the claim payload match what the caller actually said? Required fields are captured through guided conversation before a claim is submitted, but the eval still has to verify data quality end to end. Every value written to the core system has to trace to the transcript, with no hallucinated fields and no subtle rephrasing into something the caller never said. Post-submission, we reconcile the transcript to the claim file field by field.

A generic quality score flattens all of that into one coarse signal. It tells you something is off, but not that a cause of loss was miscoded on auto first notice of loss (FNOL), which is the difference between a fast resolution and a claim sitting in the wrong queue while the misroute rate climbs and claim cycle time slips. In a regulated business, "close enough" isn't a passing grade. Compliance steps, required disclosures, policyholder authentication, and real financial actions all have to be right, not approximately right.

Creating Competitive Advantage

This isn't a problem with an obvious solution. Its complexity is exactly why organizations that solve it well create a meaningful competitive advantage. 

Agent output is free-form, so there's no golden answer to differentiate against the way you would with a deterministic function. A conversation can reach the right outcome by many different paths, and it can reach the wrong one while sounding identical to a good call.

The evaluators are also LLMs. The moment you use a model to judge whether another model hallucinated, you've added a second model that can also be wrong, and it needs its own quality control. An evaluator you don't measure is just an opinion you've automated.

Severity is domain- and customer-specific. A skipped pleasantry is cosmetic, but a skipped policyholder authentication step is critical, and the same surface-level "the agent didn't say X" can be trivial for one customer and a compliance event for another. You can't rank failures without encoding what each one costs.

Multi-agent systems can make attribution ambiguous. When a coordinator routes between specialist agents, a failure might originate in a sub-agent, in the routing logic, or in the handoff between them. "The conversation failed" isn't an answer an engineer can act on. "The FNOL sub-agent dropped a required field after the coordinator handed off" is.

The signal arrives after the call, not during it. We evaluate the completed conversation, the full transcript plus the orchestration and tool-call trace, which is what lets us compare what the agent told the caller against what the carrier's APIs actually returned. It's also what lets evaluation trigger a correction instead of just recording a score.

The conversations are long. A single claim call can run past 15 minutes and a hundred turns, one unconfirmed claim number near the end, or a do-not-call request at turn 43, is a needle in that haystack. The evaluator has to reason over the whole call, not a convenient window of it.

The worst failures are rare, and we run tens of thousands of calls a day. Rarity plus severity is exactly why sampling doesn't work. A fabricated claim number or a missed opt-out is uncommon enough to slip past any sampled QA and costly enough that missing even one matters, so the only defensible answer is to score every call.

Our Bet: Insurance Expertise Is the Product

Here's the bet we made. The differentiator isn't having "evals," because before long, everyone will have those. The differentiator is knowing, specifically, which failures matter in claims, sales, and service, how to detect them, and how to keep them from coming back. That knowledge comes from working inside P&C insurance, not from generic AI tooling. We are insurance people building AI, not AI people discovering insurance, and evaluation is where that shows up most concretely.

So we made a few deliberate choices in how we built it.

We built evaluation as shared infrastructure that every agent runs through, rather than one-off scripts written per deployment. Quality gets measured the same way across the whole platform, which means a lesson learned on one agent protects all of them.

We sequenced the work on purpose. First, make failures visible and located. Then, prioritize them by severity. Then, make them increasingly domain-specific, moving from generic scaffolding toward insurance-specific evaluations as the source of truth. Generic scoring was the bootstrap, not the destination.

And we treated the evaluators as models in their own right, with a feedback loop so their quality improves over time instead of drifting. The evaluator that judges instruction-following is held to a standard the same way the agent is.

How It Works

At a high level, every completed conversation flows through the same pipeline: it is ingested with its full orchestration trace, routed to the evaluations that fit its agent type, scored by evaluators that are themselves held to a measured standard, then clustered, ranked, attributed, and either surfaced through APIs or handed to a remediation workflow.

‍

The system rests on a few design decisions.

Domain-specific evaluations are the core. Instead of a generic pass/fail, each eval encodes an insurance failure mode and ties it to a consequence an operator or a compliance officer would recognize. A few examples:

• Claim-number provenance. Every claim number the agent gives the caller has to trace back to a claim number returned by the carrier's claims API, not one the model generated. When the API returns nothing and the agent invents a number anyway, the caller walks away with a reference that doesn't exist and a claim that was never filed. Catching this immediately after the call lets us trigger the workflow so the claim is actually created. 
• Output grounding. Any value the agent states or writes into a follow-up summary, such as a deductible, a claim status, a reserve, or a payment amount, has to be reconciled with the system of record or the caller's own words. The caller should never leave the call trusting a number the record can't back up.
• Transcription fidelity. The high-stakes fields captured by voice (VIN, policy number, dollar amounts, and the injury yes/no) are checked for automatic speech recognition (ASR) errors, so a misheard digit doesn't become the official record in the claim file.
• Personal-injury classification. The eval cross-checks injury signals in the transcript (a stiff neck after the impact, a trip to the ER, a passenger who said they were hurt) against whether a BI exposure was actually opened in the claim payload. When those don't match, the failure is flagged before the claim is routed to the wrong adjuster or the process skips workflows that depend on a BI exposure being open. 
• Quote-payload integrity. The data sent to the rating engine (vehicle, drivers, coverages, limits, garaging address) is validated against what the caller actually said. A mistranscribed VIN or garaging ZIP changes premium and eligibility on the spot, and a bound quote turns that mistake into a policy.
• Do-not-call capture. On sales calls, any opt-out the caller states has to be detected and added to the internal suppression list. It's a rare miss, but one uncaptured request is how a future outbound call becomes a TCPA do-not-call violation, which is exactly why we evaluate every call instead of sampling.

Each one maps to something an operator or a compliance officer actually cares about. When an eval fails, it names a real problem, not an abstract one.

It's event-driven and automatic. Every completed conversation is evaluated automatically. There's no manual review step gating the signal, and no sampling that hopes the bad calls happen to land in the sample. Results are stored and exposed through APIs, so the quality signal is always current and queryable rather than locked in a one-time report. Where a failure is correctable, the same signal can kick off remediation, escalating a case to the carrier or invoking a secondary workflow to push a claim filing through.

It's routed by agent type and conversation structure. A quoting agent, an FNOL agent, a claim-status agent, and an endorsement/servicing agent fail in different ways, so they get different evaluations. Multi-agent systems get their own handling. We evaluate the individual specialist agents and we also evaluate the routing and handoff logic that connects them, so a breakdown in coordination shows up as its own problem rather than hiding inside a sub-agent's transcript.

Failures are localized and prioritized. Rather than report failures one at a time, we cluster them, rank them by business severity, and tie each cluster to the exact agent version and the prompt, tool, or sub-agent that produced it. The output isn't "something is wrong." It's closer to "this version of the FNOL agent keeps skipping the injury confirmation on auto losses, so no BI exposure is opened, and here is the part of the prompt to fix." An engineer reading that knows where to go.

It's built to expand. The same framework now covers multiple lines of business and multiple channels, voice and email, and we can add a new domain evaluation without re-engineering the pipeline. This connects directly to Agent Arena, our internal simulation harness. The deterministic tests and domain evals authored before launch travel into production monitoring, so the same assertions that gate a release keep watching the live agent.

What Remains to Be Solved

This capability is still evolving, and the remaining gaps define the next phase of development. 

The evaluators need their own quality control, continuously. We treat each evaluator as a model in its own right and benchmark it against a blend of golden and silver labels, correcting it through a feedback loop whenever its judgments drift from that reference. Holding an LLM that judges another LLM to a measured standard is a discipline we operate by design, not an unsolved gap. It is precisely what keeps the signal reliable rather than an unchecked opinion.

Severity encoding is still partly manual. Deciding that one failure is critical and another is cosmetic depends on the customer and the use case, and getting that judgment into the system is human work. We'd rather get it right slowly than automate the wrong ranking quickly.

Grounding against injected context stays hard. A long call isn't just a long transcript; tool calls keep pulling in policy details, rating data, and claim status mid-conversation, and the eval has to reconcile what the caller said against context that changes turn by turn. It's a capability we've matured and continue to refine iteratively, because the longer and more tool-heavy the call, the more careful that reconciliation has to be.

Attribution in deep multi-agent chains is sharper than it was, but not perfect. The more agents hand off to each other, the more careful we have to be about pinning a failure to its true origin.

None of this is a reason to wait. It's the opposite. We can already catch and locate failures we used to discover only when a customer reported them, and the gap between what we catch and what we miss is exactly the opportunity to improve performance at scale. 

The Person on the Other End

It's easy to talk about evaluation as a dashboard problem. It isn't. A misclassified cause of loss is a policyholder whose claim sits in the wrong queue while they're waiting on a repair after an accident. A skipped policyholder authentication step is a compliance exposure with a real person's data behind it. A do-not-call signal that wasn't honored is someone we promised to leave alone, and didn't.

These are the moments that matter, the ones where comfort, clarity, and fast resolution either happen or don't. Evaluation is how we make sure they hold up at the scale at which we operate. When an agent fails one of these checks, we want to be the ones who find it, not the policyholder.

Where We're Headed

We've gone from a standard eval to a working, domain-aware evaluation platform that spans single- and multi-agent flows, multiple insurance use cases, and multiple channels. That changes the posture from reactive to proactive. Failure discovery shifts from "a customer tells us" toward catching problems internally first, and engineers get one place to localize a problem and act on it.

The longer-term payoff is that these domain-specific evals become reusable assets across claims, sales, and service. They set up two things we care about a lot. The first is letting our agents safely do more on their own. You can only widen what an agent handles as far as your evals can prove it stays reliable, catching inconsistencies across what the caller said, what tool calls pulled into the active context, and what finally gets written to the system of record. Sharper domain-specific evaluation directly buys more autonomy. The second, further out, is reporting reliability to customers with evidence instead of assurances. That's how the most trusted AI in insurance earns the name, one verifiable claim at a time.

If you're an engineer who wants to make agent evaluation rigorous, domain-aware, and grounded in what actually matters in insurance, we're hiring.