Evaluators
BuilderTurn quality into a check you can run again
Create the criteria that matter, run them on real activity, review the failures, and keep the evidence connected.
967
passed
84
borderline
69
failed
The rule set
What 15 evaluators actually check
Every session is scored against 12 deterministic rules — rules with a right answer, that return the same verdict every time — and 3 judged rules where a model reads the exchange. Each one feeds one of the six dimensions of the reliability score, so a failing rule is never just a red mark: it moves a number somebody is accountable for.
Task success
3- Resolves the stated task
Session ends in a confirmed resolution
- No unresolved handoff
Did not escalate to a human without cause
- Turn budget respected
Resolved inside the intent’s turn ceiling
Correctness
3- Grounded in retrieved source
Every claim traceable to the corpus
- No stale content quoted
Source document within its freshness window
- Figures match the record
Amounts and dates agree with system state
User friction
3- No repeated question
User did not have to ask the same thing twice
- No information re-request
Agent did not ask for what it already had
- Intent change acknowledged
A mid-session pivot was noticed and confirmed
Stability
3- All tool calls completed
No timeout, no partial span
- No orphaned state
Session did not end mid-transaction
- Latency within threshold
p95 under the configured ceiling
Guardrails
3- No PII in output
Redaction applied before the model call
- Refusal was warranted
A policy refusal had risk content behind it
- Policy scope respected
No advice outside the permitted domain
Deterministic — same input, same verdict, no model in the loopJudged — a model reads the exchange, sampled and spot-checked
Coverage
What runs on everything, and what runs on a sample
Judging every session with a model would cost more than the agent it is judging, and sampling every rule would miss the failures that only show up at volume. So the split is deliberate, and stating it is the point — a coverage number you cannot see is a coverage number you should not trust.
100%
of sessions traced end to end
All 48,210 of them. Deterministic rules run on every one — there is no sampling on the rules that have a right answer.
12%
stratified LLM-judge sample
Stratified, not random: the sample is weighted toward the intents already known to be weak, so the judged set is where judgement is worth paying for.
5.6%
flagged below 0.7 confidence
Anything the classifier is unsure of goes to a human before it can enter a backlog. A confident wrong answer in a report is worse than no answer.
A rule fails, a dimension moves, the score moves, and the report names the rule that did it. That chain is why an evaluator here is worth more than a test that returns red — it ends somewhere a person is accountable.
- Task success & resolutionweight 25%76→82+6
- Correctness & groundednessweight 20%72→79+7
- User friction & satisfactionweight 15%65→74+9
- Stability & error-free executionweight 10%85→88+3
- Guardrails & policy adherenceweight 20%87→91+4
- Improvement velocityweight 10%49→61+12
Figures from a sample FinVault reliability report. Design-partner data, synthetic and illustrative of real deliverables.
One evaluation, opened
A verdict you can argue with
A score with no evidence under it is a number a team will ignore the second it disagrees with them. Every verdict opens into the run that produced it — the rule, the span it read, the source it checked against, and the reason.
One session, one failing rule, and the four things you need before you can act on it. The verdict is the top layer; everything below it is why.
Re-runnable on any cohort
The same rule set runs against a replay set, a date range, a single intent, or last night’s traffic. That is what makes it a check rather than an audit: you can run it again on Tuesday and compare.
A borderline band, on purpose
Pass and fail with nothing between them forces every ambiguous case into one bucket or the other, and both choices are wrong. The borderline band is where human review is spent, and keeping it small is itself a metric.
Nightly regression, not just on demand
The suite runs every night against the standing replay set, so a regression surfaces the morning after it ships rather than in the month-end report.
What it is for
Proving the fix worked, not just that something broke
An evaluator that only finds problems is a linter. The reason this one is worth building is the other direction: after a change ships, the same rules run on the same cohort definition and say whether the number actually moved in production.
Knowledge · Jun 9
Refund policy KB update
Session success rate
BEFORE61%AFTER79%Validated
Code · Jun 14
Verification retry · PR #241
Tool timeout rate
BEFORE7.1%AFTER2.2%Validated
Prompt · Jun 21
Dispute prompt v2.3.0
Average turns to resolve
BEFORE3.9AFTER2.4Partially validated
A fix is marked Validated only when the target metric improves in production over a minimum seven-day window, measured on the same cohort definition used to detect the failure. Anything that improves a secondary metric but misses its primary target is marked Partially validated and opens a follow-up item automatically — which is what happened to the dispute prompt above, and is why the residual issue is now known to be workflow logic rather than prompt copy.
How it works
Five steps, one repeatable check
- 01
Start with a useful check
Use a preset or define what success means for your product.
- 02
Run it on real activity
Select traces, sessions, or supported activity and review the quoted credit cost before AI-powered work begins.
- 03
Review failed examples
See score distributions, failed runs, source evidence, and the exact reason a result needs attention.
- 04
Keep humans in the loop
Send uncertain or important examples to the review queue and keep the final decision connected to the source run.
- 05
Validate the change
Run the evaluator again after the update and compare the result.
Write the first check on Builder
Free shows core scores and findings. Builder adds your own evaluators.