> ## Documentation Index > Fetch the complete documentation index at: https://docs.cfosilvia.com/llms.txt > Use this file to discover all available pages before exploring further. # Silvia analyst 1 1 ab test # A/B test: Silvia Analyst 1.0 vs 1.1 (quant\_analyst tool) Status: **CALLED 2026-10-07: 1.1 for everyone, with longer timeouts.** Ran 2026-09-29 01:43 UTC to 2026-10-07 22:34 UTC at 50/50 in production. The flag module is deleted; see Result. ## How the test ran (historical) The sections up to **Runbook** record the test as it ran from 2026-09-29 to 2026-10-07. None of them is a live instruction: there is no split, no exposure and no 1.0 judge any more. The current controls are under **Runbook**, and the outcome is under **Result**. ### The two arms | Arm (PostHog variant) | Model | Endpoint | | - | - | - | | `control-analyst-v01` | Silvia Analyst 1.0, Qwen3.5-35B-A3B + LoRA, id `quant-analyst` | Modal `quant-analyst-llm` | | `analyst11-v01` | Silvia Analyst 1.1, Qwen3.8-27B, id `quant-analyst-1.1` | Modal `silvia-analyst-llm` | Only the `quant_analyst` tool is split, bucketed by clerkId. Every failure (PostHog down, unknown variant, no clerkId) serves 1.0. ### Why it could run alongside chat-model-v02 * **Assignments are independent.** PostHog hashes flag key plus distinct id, so each test is balanced across the other's arms. Opus 5.5 users land 50/50 in the analyst arms and the other way round. That adds some noise to each test but no bias. * **The chat judge stays on 1.0.** chat-model-v02 decides on `judge_overall_mean`, and 1.1 scored the same turns +0.42 higher in the replay. Moving the judge mid-test would shift that score for both arms at once, and the result would stop meaning anything. The judge moves to 1.1 in a follow-up after chat-model-v02 ends. * **The CSV mapper and CSV mapping judge move to 1.1 for everyone.** Neither is part of either test. 1.1 tied 1.0 on mapping (57/60 each) and passed more correct mappings clean in the judge (30/38 vs 22/38). ### Volume Last 7 days (to 2026-09-24): 14,000 external runs from 2,947 users, 94.7% success. At 50/50, one week gives about 7,000 runs per arm. ### What was instrumented | Signal | Where | | - | - | | Arm, per run | `quant_analyst_runs.model`: `quant-analyst` = control, `quant-analyst-1.1` = treatment. Unserved runs are recorded as `unserved:modal:` (or `unserved:none:` when the arm has no configured backend), so an outage counts against the arm that asked for it | | Exposure | `$feature_flag_called` for `silvia-analyst-v01`, once per enrolled run | | Outcome | `quant_analyst_runs.success`, `.elapsed_ms`, `.deadline_exceeded`, `.tool_parse_failures`, `.turns` | | Answer quality | `ai_chat_judge_runs` on turns that called `quant_analyst`, scored by the 1.0 judge in both arms | Exclude internal users from the read (`public.internal`) and test or deleted accounts. ### Decision criteria (as signed off before the ramp) Ship 1.1 if all three hold after one week at 50/50: 1. Run success rate is not lower than control by more than 1 point. 2. Median `elapsed_ms` is no more than 1.5x control and `deadline_exceeded` does not rise. 3. The judge score on turns that called `quant_analyst` is not lower than control. The offline evidence (72.5% vs 20.1% judged correct on 193 real prod questions) is the reason to expect a win. The online test checks that it holds on live traffic and that 1.1 is fast enough. ## Runbook The flag, its Redis kill switch and the internal-user env var are gone. The steps below are the only working controls now. **Roll back to 1.0** (every caller: quant\_analyst tool, chat judge, CSV mapper and judge). Works only while the Modal app `quant-analyst-llm` is still up. Set two production env vars in Vercel and redeploy: * `SILVIA_ANALYST_URL=https://silvia--quant-analyst-llm-server-serve.modal.run/v1` * `QUANT_ANALYST_MODAL_MODEL=quant-analyst` `quant_analyst_runs.model` then reads `quant-analyst` again, which confirms it took. Unset both and redeploy to return to 1.1. Reverting this PR also works but takes the staging to main path. **Retire 1.0.** After 1.1 has served all traffic for a full week with no rollback, stop the Modal app: `modal app stop -y quant-analyst-llm`. From then on the only rollback is a new 1.0 deploy, so check the abort rate and median `elapsed_ms` before stopping it. ## Result (2026-10-07) External users only, internal, test and deleted accounts dropped. Arm read from `quant_analyst_runs.model`. | Gate | 1.0 | 1.1 | | | - | - | - | - | | Runs (users) | 6,461 (1,442) | 6,467 (1,472) | | | Run success | 98.2% | 94.7% | fail, -3.5 pts | | Median `elapsed_ms` | 38 s | 51 s | pass, 1.34x | | `deadline_exceeded` | 0.28% | 0.43% | up | | Judge overall on matched turns | 3.48 (n=640) | 3.43 (n=630) | -0.05, 95% CI \[-0.21, +0.11] | 1.1 failed gate 1 on timeouts alone. 301 of its 345 failures were the executor's own aborts ("This operation was aborted"): 203 hit the 240 s run deadline and 98 hit the 120 s per-call timeout (1.0: 51 of 117). 1.1 takes about 9 s per turn against 5 s for 1.0, so normal-length runs ran out of clock. Excluding aborts, success is 99.3% for 1.1 and 99.0% for 1.0, and 1.1 overflows context less (14 vs 35) and never truncated a tool call (1.0: 9). Decision: ship 1.1 to everyone and raise the budgets 50% (run 240 s to 360 s, call 120 s to 180 s). Watch the abort rate and median latency for a week after the deploy. The chat judge moves to 1.1 in the same change (chat-model-v02 has ended), so nothing asks for 1.0. `quant-analyst-llm` stays up for a week as the rollback target, then stops. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.