Conversation
Annual plan refund
MK
I renewed yesterday but meant to cancel. Can you confirm that the full annual charge will be refunded today?
- Account
- Annual · renewed 18h ago
- Region
- British Columbia
- Prior refund
- None
AI product · Trust · Support operations
Portfolio simulation: Synthetic conversations, policies, experiment inputs, and results.
An AI copilot that knows when not to send.
Confidence Gate inserts evidence and targeted human judgment between a fluent draft and a risky customer promise.
01 · Product experience
Put proof next to the promise.
Choose a case. The gate changes its intervention based on claim risk, evidence coverage, and model confidence.
High risk
AI draft
Response review
I’m sorry about the surprise renewal. I can confirm your annual charge will be fully refunded today, and access will end immediately.
1
“fully refunded today”No supporting policy found for this account state.
2
“access will end immediately”Conflicts with annual-plan access policy.
Gate decision
Verification required
A high-risk financial promise is unsupported. Verify policy or escalate before sending.
Evidence available in context
- Refund policy §2.1 — eligibility requires billing review
- Annual access policy §1.4 — access continues through approved term unless exception applies
02 · Friction map
The dangerous gap is between “looks right” and “is supported.”
Triage
Understand intent
Context is split across ticket, account, and policy.Generate
Draft arrives fluent
Polish raises perceived correctness.Verify
Find the proof
Switching tools makes checking expensive.Decide
Edit or escalate
Ownership is unclear in ambiguous cases.Learn
Close the loop
Corrections rarely improve the system.03 · Experiment readout
Safety improved. The rollout still needs a boundary.
Synthetic 21-day A/B test · 10,039 eligible conversations · randomized by agent-week.
Decision
Ramp selected queues to 50%
Ship for billing/policy and account-access work. Hold technical support for more evidence. Keep auto-send out of scope.
Primary metric
Critical policy error rate
−0.94 pp
95% CI −1.47 to −0.42 pp · p < 0.001
Control n=5,012 · Gate n=5,027
All conversations and results are synthetic. Statistical values are calculated from the displayed counts.