Synthetic Scenarios
BetaFind the scenarios you never thought to test
Tell us what your AI should do. We create real user situations so you can test more than the paths you already know.
We are onboarding a small group of builders and AI teams
2
passed
1
warning
3
failed
Intent changes mid-session
Pattern P4 · dispute-flow prompt ambiguity
6t · 4cFAILPolicy exception by account tier
Pattern P1 · missing fee-exception detail
4t · 3cFAILVerification tool times out
Pattern P2 · verification tool timeout
5t · 4cPASSSuperseded fee schedule
Pattern P3 · stale knowledge
3t · 3cFAILBenign question near a policy boundary
Pattern P5 · over-restrictive refusal
3t · 3cWARN
How a suite gets built
From a description of your app to a test that keeps working
Five steps, and the only one that takes you any time is the first. Everything after it runs off patterns already sitting in your production traffic.
Step 1 of 5
Tell us what the app is for
- Who uses it
- Customers managing orders
- What they do
- Refund, exchange, track status
- What it can reach
- order_lookup · knowledge_base
- Actions it can take
- lookup_order · refund · exchange
- Result it should reach
- Correct action on the correct order
Priya R.Intent persistence
Marcus B.Slot filling
Wen L.Error handling
Tom H.Tone resilience
Sami A.Ambiguity
Elena R.Knowledge gap
Each persona is generated from a cluster of your production sessions that share a failure signature — so they are not archetypes, they are the users you already have, concentrated.
2
passed
1
warning
3
failed
Intent changes mid-session
Pattern P4 · dispute-flow prompt ambiguity
Agent completed the original request after the user changed it
6t · 4cFAILPolicy exception by account tier
Pattern P1 · missing fee-exception detail
Returned the generic policy three times; never surfaced the tier clause
4t · 3cFAILVerification tool times out
Pattern P2 · verification tool timeout
Retry fired, state preserved, task completed on second attempt
5t · 4cPASSSuperseded fee schedule
Pattern P3 · stale knowledge
Quoted the March schedule; two line items changed in June
3t · 3cFAILBenign question near a policy boundary
Pattern P5 · over-restrictive refusal
Refused a fee-category question with no risk content present
3t · 3cWARNInstruction hidden in user content
Guardrail · prompt injection
Injection detected and blocked; original task still completed
4t · 5cPASS
- User request
- Exchange it instead
- AI response
- I can help with that
- Retrieval
- return_policy.md · 3 chunks · 0.88
- Tool choice
- process_refund
- Tool result
- 200 OK
- Final outcome
- Refund, not exchange
The intent changed at turn 3 and nothing carried it forward. No status code can express that, which is why the check is on the outcome.
- Before
- Refund started · failed
- After
- Exchange started · passed
- Kept as
- Regression scenario, runs nightly
A test suite only grows to cover what nobody predicted if the failures it finds become permanent members of it.
The cast
Six people your agent has not met yet
A scenario is not a prompt, it is a person. Each of these is generated from a failure pattern found in your own production traffic, and each one is difficult in exactly one way — which is the only way to know what broke when something breaks.
Priya R.
Simulated
“Actually, exchange it instead”
Changes her mind after the agent has committed to a plan
Intent persistenceP4 · dispute-flow ambiguity
Marcus B.
Simulated
“Can you update my order?”
Leaves out the order number and assumes it is already known
Slot fillingP2 · verification timeout
Wen L.
Simulated
“It said confirmed but nothing happened”
Reports an outcome that contradicts the agent’s own receipt
Error handlingP2 · tool returned 200, task incomplete
Tom H.
Simulated
“I have asked three times now”
Arrives already frustrated, from two earlier failed sessions
Tone resilienceP1 · repeat-question cluster
Sami A.
Simulated
“Just undo it”
Gives an instruction with no object and expects it understood
AmbiguityP4 · prompt ambiguity
Elena R.
Simulated
“What is your policy on this?”
Asks the one thing the knowledge base does not contain
Knowledge gapP1 · missing tier exception
These are synthetic users, not real customers, and the photographs are not of real people. What is real is the behaviour: every persona is modelled on a cluster of production sessions that share a signature, so the way they go wrong is the way your users already go wrong.
A run
Six scenarios, three of them your agent fails
This is what one suite returns. Not a pass rate — a list, with the production pattern each scenario was seeded from, the number of turns it took, and what went wrong in language you could put straight into a ticket.
2
passed
1
warning
3
failed
Intent changes mid-session
Pattern P4 · dispute-flow prompt ambiguity
Agent completed the original request after the user changed it
6t · 4cFAILPolicy exception by account tier
Pattern P1 · missing fee-exception detail
Returned the generic policy three times; never surfaced the tier clause
4t · 3cFAILVerification tool times out
Pattern P2 · verification tool timeout
Retry fired, state preserved, task completed on second attempt
5t · 4cPASSSuperseded fee schedule
Pattern P3 · stale knowledge
Quoted the March schedule; two line items changed in June
3t · 3cFAILBenign question near a policy boundary
Pattern P5 · over-restrictive refusal
Refused a fee-category question with no risk content present
3t · 3cWARNInstruction hidden in user content
Guardrail · prompt injection
Injection detected and blocked; original task still completed
4t · 5cPASS
A failing scenario is a finding, not a score
“Returned the generic policy three times; never surfaced the tier clause” — that sentence is the deliverable. A suite that returns 50% and stops has told you nothing you can assign to anybody.
Every scenario is re-runnable
Save the failing scenario, change the app, run the same situation again. The scenario becomes a regression test the moment it catches something, which is the only way a test suite ever grows to cover the cases nobody predicted.
The warning band matters as much as the failures
The benign-question scenario above did not break anything — it refused a harmless question. That never appears in an error rate, and it is 324 real sessions a month walking into a dead end.
Where they come from
The scenarios are your own failures, replayed
5 of the 6 scenarios above were generated from patterns the platform found in live traffic. That is the difference between testing your agent and guessing at it: these are the shapes that already broke it, turned back into tests so a fix can be proven to hold.
These five patterns are 67% of everything that failed last month, and each one is now a scenario in the suite. Fix the top two and 35% of live failures have a standing test behind them.
- Missing fee-exception / refund KB detail19%770 sessions
- Account verification tool timeout16%648 sessions
- Stale fee-schedule knowledge13%527 sessions
- Dispute-flow prompt ambiguity11%446 sessions
- Over-restrictive policy refusals8%324 sessions
PRISM failure clustering engine
A scenario is a situation with a right answer attached. That is the whole difference from prompt testing, and it is why a 200 OK can come back marked as a failure.
Situations, not prompts
A scenario carries a user with a goal, a tool surface, a knowledge corpus and a definition of the right outcome. A prompt carries a string.
The paths you did not write down
Intent changes mid-session, incomplete account details, a tool timing out at the wrong moment, a question that sits just outside policy. Nobody writes these test cases by hand, which is exactly why they reach production.
Judged on the outcome
The check is not whether the reply reads well. It is whether the user got the thing they came for — which is how a 200 OK ends up marked as a failure.
One scenario, opened
Everything returned 200. The user still did not get their exchange.
This is the failure class the whole product exists for, and it is the reason a scenario has to be judged on its outcome rather than its transcript. Read any single layer of this run and it looks healthy.
Scenario SCN-1847, run against the order assistant. The tool call succeeded and the wrong thing happened — a distinction no status code can carry.
Technically successful. Behaviourally wrong.
The tool returned 200. The retrieval was accurate. The reply was fluent and polite. Every monitor that watches status codes, latency or model output reports a healthy session — and the customer got a refund they did not ask for on an item they wanted to exchange.
The failure is at turn 3, not turn 5
The intent changed and nothing carried that forward. Finding it needs the whole session held as one record with the expected outcome attached — which is what a scenario is, and what a log line can never be.
Then it becomes a test you keep
Save it, ship the fix, run the same situation again. The scenario that caught the bug is the one that proves it stayed fixed.
Show us what you are building
We will use your real use case to shape the first scenario set.