.png?fit=max&auto=format&n=hWBn50UiAZnOwoHI&q=85&s=6b3d6f29082e9ecfe13a87ffa36688d4){kind=icon}
A/B test: Silvia Analyst 1.0 vs 1.1 (quant_analyst tool)
Status: CALLED 2026-10-07: 1.1 for everyone, with longer timeouts. Ran 2026-09-29 01:43 UTC to 2026-10-07 22:34 UTC at 50/50 in production. The flag module is deleted; see Result.How the test ran (historical)
The sections up to Runbook record the test as it ran from 2026-09-29 to 2026-10-07. None of them is a live instruction: there is no split, no exposure and no 1.0 judge any more. The current controls are under Runbook, and the outcome is under Result.The two arms
Only the
quant_analyst tool is split, bucketed by clerkId. Every failure (PostHog down,
unknown variant, no clerkId) serves 1.0.
Why it could run alongside chat-model-v02
- Assignments are independent. PostHog hashes flag key plus distinct id, so each test is balanced across the other’s arms. Opus 5.5 users land 50/50 in the analyst arms and the other way round. That adds some noise to each test but no bias.
- The chat judge stays on 1.0. chat-model-v02 decides on
judge_overall_mean, and 1.1 scored the same turns +0.42 higher in the replay. Moving the judge mid-test would shift that score for both arms at once, and the result would stop meaning anything. The judge moves to 1.1 in a follow-up after chat-model-v02 ends. - The CSV mapper and CSV mapping judge move to 1.1 for everyone. Neither is part of either test. 1.1 tied 1.0 on mapping (57/60 each) and passed more correct mappings clean in the judge (30/38 vs 22/38).
Volume
Last 7 days (to 2026-09-24): 14,000 external runs from 2,947 users, 94.7% success. At 50/50, one week gives about 7,000 runs per arm.What was instrumented
Exclude internal users from the read (
public.internal) and test or deleted accounts.
Decision criteria (as signed off before the ramp)
Ship 1.1 if all three hold after one week at 50/50:- Run success rate is not lower than control by more than 1 point.
- Median
elapsed_msis no more than 1.5x control anddeadline_exceededdoes not rise. - The judge score on turns that called
quant_analystis not lower than control.
Runbook
The flag, its Redis kill switch and the internal-user env var are gone. The steps below are the only working controls now. Roll back to 1.0 (every caller: quant_analyst tool, chat judge, CSV mapper and judge). Works only while the Modal appquant-analyst-llm is still up. Set two production env vars
in Vercel and redeploy:
SILVIA_ANALYST_URL=https://silvia--quant-analyst-llm-server-serve.modal.run/v1QUANT_ANALYST_MODAL_MODEL=quant-analyst
quant_analyst_runs.model then reads quant-analyst again, which confirms it took. Unset
both and redeploy to return to 1.1. Reverting this PR also works but takes the staging to
main path.
Retire 1.0. After 1.1 has served all traffic for a full week with no rollback, stop the
Modal app: modal app stop -y quant-analyst-llm. From then on the only rollback is a new 1.0
deploy, so check the abort rate and median elapsed_ms before stopping it.
Result (2026-10-07)
External users only, internal, test and deleted accounts dropped. Arm read fromquant_analyst_runs.model.
1.1 failed gate 1 on timeouts alone. 301 of its 345 failures were the executor’s own aborts
(“This operation was aborted”): 203 hit the 240 s run deadline and 98 hit the 120 s per-call
timeout (1.0: 51 of 117). 1.1 takes about 9 s per turn against 5 s for 1.0, so normal-length
runs ran out of clock. Excluding aborts, success is 99.3% for 1.1 and 99.0% for 1.0, and 1.1
overflows context less (14 vs 35) and never truncated a tool call (1.0: 9).
Decision: ship 1.1 to everyone and raise the budgets 50% (run 240 s to 360 s, call 120 s to
180 s). Watch the abort rate and median latency for a week after the deploy. The chat judge
moves to 1.1 in the same change (chat-model-v02 has ended), so nothing asks for 1.0.
quant-analyst-llm stays up for a week as the rollback target, then stops.