> ## Documentation Index > Fetch the complete documentation index at: https://docs.cfosilvia.com/llms.txt > Use this file to discover all available pages before exploring further. # Opus 5 5 ab test # A/B test: Opus 4.8 vs Opus 5.5 (chat model) **Concluded 2026-10-08** (experiment 470502, `chat-model-v03`, control at 100%). Scaffolding removed in CFO-9381 per Teardown below, including the internal Opus 5.5 pin; the `chat-model-effort.ts` template stays. Status (historical): **BUILT, NOT LAUNCHED** (2026-09-22). No external user is enrolled until the flag is ramped, so their turns serve today's config. **Internal users are the exception**: the team runs Opus 5.5 from deploy via `CHAT_MODEL_V02_INTERNAL_TREATMENT`, which does not read the flag at all. That is real 5.5 traffic before launch, so the pricing migration below is a prerequisite for *merging*, not for ramping. Owner: chat / AI platform Flag: `chat-model-v02` · Related: `docs/plans/thinking-effort-ab-test.md` (the concluded v01 test, whose flag module this one is forked from) ## The question Production serves `claude-opus-4-8` @ `medium` — the winning arm of the v01 model × effort test. Opus 5.5 is a newer model at lower cost. Is it at least as good for Silvia's chat workload? ## The cost case, measured Every rate is 20% lower on 5.5 **except cache reads, which are 60% lower**: | \$/MTok | input | output | cache write (5m) | cache write (1h) | **cache read** | | - | - | - | - | - | - | | Opus 4.8 | 5.00 | 25.00 | 6.25 | 10.00 | **0.50** | | Opus 5.5 | 4.00 | 20.00 | 5.00 | 8.00 | **0.20** | That asymmetry is the whole story for Silvia, because of what its token mix looks like. Measured over 64,587 completed Opus 4.8 turns (28 days to 2026-09-22): | category | share of tokens | avg/turn | | - | - | - | | **cache read** | **81.9%** | 316,601 | | cache write | 16.2% | 62,673 | | output | 1.9% | 7,226 | | input | 0.01% | 37 | A \~115KB system prompt re-read on every loop of an agentic turn means spend is almost entirely cache reads — exactly the line with the 60% cut. Pricing that identical traffic both ways: **$47,202 at 4.8 rates vs $33,672 at 5.5 rates — a 28.7% saving** ($0.731 → $0.521 per turn), not the 20% the headline rates suggest. At current volume that is \~\$13.5k per 28 days. Token MIX is held fixed in that calculation, so it isolates the price change. 5.5 may also shift the mix — it consolidates tool calls, which moves loop count and therefore cache reads — and that second-order effect is what the experiment measures. **Do not compare `total_tokens` between arms as a cost proxy.** The four categories span a 100× price range, so mix moves cost independently of the total. Cost is priced per row by the realized model through `model_cost_usd`. ## The two arms | Arm (PostHog variant) | Model | Effort | | - | - | - | | `control-v02` | `claude-opus-4-8` | `medium` | | `opus55-medium-v02` | `claude-opus-5-5` | `medium` | Exactly one factor varies: the **model**. Effort is held at `medium` on both sides, and both arms read it from the same source (`DEFAULT_LLM_CONFIG.defaultThinkingEffort`) so they cannot drift apart even if the production default moves. `control-v02` is *derived* from `DEFAULT_LLM_CONFIG`, not restated as literals. The v01 template hardcoded its control and went stale when production moved off `xhigh` — a control arm that has quietly stopped matching production is an experiment measuring nothing. **Why the arm labels carry `-v02`:** `ai_chat_logs.experiment_arm` is one column shared by every experiment, and v01 also had an arm called `control` — meaning Opus 4.8 @ *xhigh*. Unversioned labels would make `WHERE experiment_arm = 'control'` silently union two different populations six weeks apart. ## Pre-flight findings (verified against the live API) Three probe scripts, run 2026-09-22. Outputs in `docs/plans/opus-5-5-ab-test/`. **`claude-opus-5-5` works with our exact request shape** — `__tests__/opus-5-5/probe-opus-5-5-compat.ts`. Adaptive thinking @ medium, `context-1m-2025-08-07`, `compact-2026-01-12`, and tools are all accepted. A nonsense model ID returns 404, so those ACCEPTs are meaningful. `__tests__/opus-5-5/probe-opus-5-5-identity.ts` confirms `response.model` echoes `claude-opus-5-5`. **Narration survives — do NOT "fix" it** — `__tests__/opus-5-5/probe-opus-5-5-narration.ts`. The Anthropic migration guide says text between tool calls moves into progress-update `thinking` blocks on 5.5, which would blank Silvia's working strip (it is fed by `text` blocks before tool calls — `lib/anthropic-stream.ts::TurnEndEvent`). **That did not reproduce.** Measured with the real system prompt *and* the request shape production actually sends — betas `context-1m-2025-08-07` + `compact-2026-01-12` and the `compact_20260112` edit, all of which `chat-turn.ts` sets on every non-Haiku turn — across 3 runs: | config | pre-tool narration (3 runs) | | - | - | | 4.8 control, prod shape | 109 / 259 / 129 chars | | **5.5, prod shape** | **82 / 81 / 112 chars** | | 5.5 @ `updates` (beta) | 0 / 88 / 106 chars, thinking blocks always empty | | 5.5, no betas or compaction | 0 / 86 / 91 chars | 5.5 still narrates in `text` blocks. `display: "updates"` returns empty thinking blocks and is no better than `summarized`, so it was rejected. No narration code change; both arms send an identical request differing only in the model ID. The last row is the control for the betas themselves: it overlaps the prod-shape row completely, so compaction and the context-window beta do not affect whether 5.5 narrates. An earlier version of this probe sent no betas at all, which measured a request shape production never sends — the numbers above replace it. The first version of this probe sent **no system prompt** and concluded 5.5 stops narrating entirely. That was unsound: `system-prompt.ts:162` instructs "Write ONE sentence per tool-using turn", so without it 4.8 was narrating spontaneously and 5.5 simply chose not to. Re-run with the real prompt before trusting any behavioral claim here. **5.5 consolidates tool calls.** It issued 6 parallel calls where 4.8 made 3 across two turns. Fewer loops means fewer narration beats and different latency, independently of answer quality. This is a treatment effect to measure (`avg_loop_count`, `tool_count`), not a defect — and it is the expected direction, already flagged in `llm_config.ts` when the default dropped to `medium`. ## What's instrumented All of this already existed or was added with the flag; no new telemetry is pending. | Signal | Where | | - | - | | Assigned arm, per turn | `ai_chat_logs.experiment_arm` (completed **and** failed rows) | | Arm on the analytics event | `chat_response_completed_server.experiment_arm` + `$feature/chat-model-v02` | | Exposure | `$feature_flag_called`, fired once per turn at enrollment | | Realized model / effort | `ai_chat_logs.model`, `.thinking_effort`, `.thinking_effort_source` (`'experiment'` when enrolled) | | Latency | `completed_at - created_at`; `duration_ms` on the event | | Cost | `ai_chat_logs.{input,output,cache_read,cache_creation,total}_tokens` | | **Quality** | `ai_chat_judge_runs` — 1–5 rubric, `source='live'`, rubric `v1` | | The read | `chat_ab_test_results`, filtered to `test_key = chat-model-v02` | The judge is the material upgrade over v01, which shipped with no quality metric and leaned on thumbs. It is what makes judge non-inferiority the ship gate rather than an aspiration. **It must be on** (`JUDGE_ONLINE_ENABLED` not `'false'`) for the whole run. ## Decision criteria **These are proposed and need sign-off BEFORE anyone is enrolled.** The flag already exists at 0% enrollment, so the deadline is the ramp, not the flag. Choosing them after seeing data is how you talk yourself into a result. **Ship `opus55-medium-v02` if all three hold:** 1. `judge_overall_mean` is non-inferior to control within a margin of **0.10** on the 1–5 scale (2% of range). 2. `judge_poor_pct` (overall ≤ 2) and `judge_fabricated_pct` do not rise materially. 3. `avg_cost_usd_per_turn` improves — measured over **priced turns** (completed and uncontaminated, count in `n_cost_priced`), not all completed turns — **or** p50 and p95 latency both improve without a cost regression. If `contamination_pct` is asymmetric between arms, that kill condition fires first and this gate does not apply. Cost is priced, not counted — read `avg_cost_usd_per_turn`, never `avg_total_tokens`. See the cost section above: the 5.5 arm can spend **more tokens and still cost less**. Note the cost columns are the one **per-protocol** metric here; everything else is intent-to-treat. A turn that fell back mid-loop has its cumulative tokens priced at whatever model finished it, so its dollar figure is wrong rather than merely diluted, and averaging it would corrupt the ship-gate number. Contaminated turns are therefore excluded from cost only. Read `avg_cost_usd_per_turn` next to `n_cost_priced` and `contamination_pct` — contamination is not random (heavy turns 529 more), so those dollars describe the turns that stayed on their arm. **Kill early, without waiting for significance, if:** * `failure_rate_pct` separates between arms * `judge_fabricated_pct` rises in the treatment arm — on financial answers this is the failure that matters most * `contamination_pct` becomes **asymmetric** between arms (see below). This one also invalidates the cost gate rather than merely warning about it: the dollars are measured on turns that stayed on their arm, so an arm losing systematically more turns is not comparable on cost. **Do not peek-and-stop on latency.** It separates almost immediately and will tempt an early call before the quality signal has any power. **Open — needs a number before launch:** sample size. The binding constraint is judge `overall` at a 0.10 margin. Compute from current completed-turns-per-day and the observed variance of `ai_chat_judge_runs.overall`. If the answer is "months", widen the margin or accept a cost-only decision. ## Contamination, and the one thing to watch A turn can leave its arm mid-flight. The fallback chain is: primary → retry → **Sonnet 5** (Anthropic API) → **Bedrock** (last resort). Analysis is intent-to-treat — group by assigned arm, report contamination separately — so these turns are detected, not silently re-bucketed. **Bedrock stays on Opus 4.8 for both arms. Deliberate.** The Bedrock path deletes `thinking` and `output_config` outright (`chat-turn.ts` \~552), so a Bedrock turn runs with no thinking at all. Pointing the treatment arm at a 5.5 Bedrock ID would produce a *third* configuration belonging to neither arm, not the treatment. Those turns are already caught: the row records `us.anthropic.claude-opus-4-8`, which matches neither arm's expected model. **The asymmetry risk is the real one.** Both arms converge on Sonnet 5 @ `max` identically, so contamination is symmetric *by construction* — unless 5.5 429s/529s at a different rate than 4.8. A new model plausibly has tighter capacity. If the treatment arm falls back more often, contamination stops being noise and becomes a confound. Per-arm `contamination_pct` is the guardrail; check it in the first days, not at the end. *Known record-accuracy bug, pre-existing, out of scope:* on a Bedrock hop the fallback clamp writes `thinking_effort = 'max'` while the request had thinking deleted. Those turns are excluded from this experiment either way. ## Internal (team) users The team runs **Opus 5.5** and is **not in the experiment**. Two separate things, kept separate on purpose: * `getInternalTreatmentModel()` pins internal users to the treatment model. Opt-out via `CHAT_MODEL_V02_INTERNAL_TREATMENT=off`. Not routed through the arm table, Redis, or PostHog — it outlives the experiment and must survive the flag being killed. * `getChatModelV02Arm()` returns `null` for internal users, so no exposure event fires and `experiment_arm` stays NULL. Every arm-grouped query drops them **by construction** rather than by someone remembering `WHERE NOT internal`. Their turns stay identifiable via `model` + `router_is_internal`. ## Router precedence The arm wins over a router downgrade while the test runs, stamped `router_reason = 'live_suppressed:experiment'` (internal: `live_suppressed:internal_treatment`). Applied to **both** arms — this is the part that is easy to get wrong. `control-v02`'s model is `DEFAULT_LLM_CONFIG.model`, so control users get pinned to Opus 4.8 too. Suppressing the router for the treatment arm only would leave control taking Sonnet downgrades, and the arms would differ by model *mix* as well as by model. The suppression is derived from enrollment, so when the experiment ends the router is live again with nothing to remember to flip back. ## Rollout 1. Apply the two migrations. **`_01` must land before the app deploys**, not just before the ramp: * `20260922_01_chat_ab_tests.sql` — `used_fallback` and `experiment_key` on `ai_chat_logs`, the `chat_ab_tests` / `chat_ab_test_arms` tables, both tests seeded into them, and the `chat_ab_test_results` view. Also drops `ai_chat_experiment_arms`, v01's bespoke view, which this replaces. One file because the view reads columns created in the same change. **Deploy this first or every chat turn fails to persist.** `markChatLogCompleted` sends `used_fallback` on every completed turn, enrolled or not. PostgREST rejects an update naming a column that does not exist, so the completion write would throw, the failure write would throw for the same reason, and the row would be stranded in `running` until the stale-run reaper closes it. This is not scoped to enrolled users the way `experiment_key` is — it is all chat traffic. * `20260922_02_model_pricing_opus_5_5.sql` — **required before this PR merges**, not before the ramp: internal users serve 5.5 from deploy. `normalize_model_key` maps anything containing "opus" to the 4.8 rate sheet, so Opus 5.5 gets billed at 4.8 rates. Per category that is 25% over on input, output and cache writes, and 150% over on cache reads. Blended on Silvia's measured token mix — 82% cache reads — it is **1.40× actual**. That is not just analytics: the USD cap gates Pro and Free members through the same function, so at 1.40× a member on 5.5 reaches their cap after 71% of the usage they are owed, i.e. **\~29% early**. 2. Sign off the decision criteria above and fix the sample size. 3. Verify the wiring with **enrollment still at 0%**: a real turn resolves unenrolled, the arm survives a workflow slice, and fail-closed works with PostHog unreachable. 4. Smoke the treatment arm **on staging**: `redis SET chat-model-v02:arm opus55-medium-v02`, send a few turns, confirm the working strip still narrates and no 400s. Unset when done. The pin is environment-wide — on production it would put every user on one arm. 5. **Launch the experiment** (sets `start_date`), then move enrollment 0 → 100. That order matters: PostHog counts exposures from `start_date`, so ramping first enrolls members whose early turns fall outside the stats window. The variant split is already 50/50 and does not change; enrollment is the only number that moves. 6. Watch `contamination_pct` asymmetry and `failure_rate_pct` daily, and run to the pre-declared sample size. Redis kill: `SET chat-model-v02:arm off` — propagates in ≤5s, no deploy. ## Teardown Per v01's precedent, remove the scaffolding rather than leaving the flag at 100% winner. What this test added, and what to do with each: * `lib/flags/chat-model-v02.ts` — **keep** as the template for the next chat experiment, wired to nothing. Bump the flag key off `-v02` and re-derive the arms before reuse. * `getInternalTreatmentModel` + `CHAT_MODEL_V02_INTERNAL_TREATMENT` — remove once the winner is the production default; the pin is then a no-op. * The arm read, exposure event, router suppression, `experimentArm` on the checkpoint / `markChatLogCompleted` / `markChatLogFailed`, and the `experiment_arm` / `$feature/` event properties — remove. * The `'experiment'` rung of `resolveThinkingEffort` and the `ThinkingEffortSource` member — **leave**. It has been removed and re-added once already; historical rows carry the value, and the column has no CHECK constraint, so read it as an open set. * `ai_chat_logs.experiment_arm`, `chat_ab_tests` / `chat_ab_test_arms`, `chat_ab_test_results`, and the probes — **keep** as the historical record. If 5.5 wins, shipping it means changing `DEFAULT_LLM_CONFIG.model`, and at that point re-check `lib/claude-generate.ts`: it uses forced `tool_choice` on the first iteration, which **returns a 400 on Opus 5.5**. It is not on the chat path and is unaffected by this experiment, but it would block a global default change. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.