​

A/B test: Opus 4.8 vs Opus 5.5 (chat model)

Concluded 2026-10-08 (experiment 470502, chat-model-v03, control at 100%). Scaffolding removed in CFO-9381 per Teardown below, including the internal Opus 5.5 pin; the chat-model-effort.ts template stays. Status (historical): BUILT, NOT LAUNCHED (2026-09-22). No external user is enrolled until the flag is ramped, so their turns serve today’s config. Internal users are the exception: the team runs Opus 5.5 from deploy via CHAT_MODEL_V02_INTERNAL_TREATMENT, which does not read the flag at all. That is real 5.5 traffic before launch, so the pricing migration below is a prerequisite for merging, not for ramping. Owner: chat / AI platform Flag: chat-model-v02 · Related: docs/plans/thinking-effort-ab-test.md (the concluded v01 test, whose flag module this one is forked from)

​

The question

Production serves claude-opus-4-8 @ medium — the winning arm of the v01 model × effort test. Opus 5.5 is a newer model at lower cost. Is it at least as good for Silvia’s chat workload?

​

The cost case, measured

Every rate is 20% lower on 5.5 except cache reads, which are 60% lower: That asymmetry is the whole story for Silvia, because of what its token mix looks like. Measured over 64,587 completed Opus 4.8 turns (28 days to 2026-09-22): A ~115KB system prompt re-read on every loop of an agentic turn means spend is almost entirely cache reads — exactly the line with the 60% cut. Pricing that identical traffic both ways: $47,202 at 4.8 rates vs$33,672 at 5.5 rates — a 28.7% saving ($0.731 →$0.521 per turn), not the 20% the headline rates suggest. At current volume that is ~$13.5k per 28 days. Token MIX is held fixed in that calculation, so it isolates the price change. 5.5 may also shift the mix — it consolidates tool calls, which moves loop count and therefore cache reads — and that second-order effect is what the experiment measures. Do not compare total_tokens between arms as a cost proxy. The four categories span a 100× price range, so mix moves cost independently of the total. Cost is priced per row by the realized model through model_cost_usd.

​

The two arms

Exactly one factor varies: the model. Effort is held at medium on both sides, and both arms read it from the same source (DEFAULT_LLM_CONFIG.defaultThinkingEffort) so they cannot drift apart even if the production default moves. control-v02 is derived from DEFAULT_LLM_CONFIG, not restated as literals. The v01 template hardcoded its control and went stale when production moved off xhigh — a control arm that has quietly stopped matching production is an experiment measuring nothing. Why the arm labels carry -v02: ai_chat_logs.experiment_arm is one column shared by every experiment, and v01 also had an arm called control — meaning Opus 4.8 @ xhigh. Unversioned labels would make WHERE experiment_arm = 'control' silently union two different populations six weeks apart.

​

Pre-flight findings (verified against the live API)

Three probe scripts, run 2026-09-22. Outputs in docs/plans/opus-5-5-ab-test/. claude-opus-5-5 works with our exact request shape — __tests__/opus-5-5/probe-opus-5-5-compat.ts. Adaptive thinking @ medium, context-1m-2025-08-07, compact-2026-01-12, and tools are all accepted. A nonsense model ID returns 404, so those ACCEPTs are meaningful. __tests__/opus-5-5/probe-opus-5-5-identity.ts confirms response.model echoes claude-opus-5-5. Narration survives — do NOT “fix” it — __tests__/opus-5-5/probe-opus-5-5-narration.ts. The Anthropic migration guide says text between tool calls moves into progress-update thinking blocks on 5.5, which would blank Silvia’s working strip (it is fed by text blocks before tool calls — lib/anthropic-stream.ts::TurnEndEvent). That did not reproduce. Measured with the real system prompt and the request shape production actually sends — betas context-1m-2025-08-07 + compact-2026-01-12 and the compact_20260112 edit, all of which chat-turn.ts sets on every non-Haiku turn — across 3 runs: 5.5 still narrates in text blocks. display: "updates" returns empty thinking blocks and is no better than summarized, so it was rejected. No narration code change; both arms send an identical request differing only in the model ID. The last row is the control for the betas themselves: it overlaps the prod-shape row completely, so compaction and the context-window beta do not affect whether 5.5 narrates. An earlier version of this probe sent no betas at all, which measured a request shape production never sends — the numbers above replace it. The first version of this probe sent no system prompt and concluded 5.5 stops narrating entirely. That was unsound: system-prompt.ts:162 instructs “Write ONE sentence per tool-using turn”, so without it 4.8 was narrating spontaneously and 5.5 simply chose not to. Re-run with the real prompt before trusting any behavioral claim here. 5.5 consolidates tool calls. It issued 6 parallel calls where 4.8 made 3 across two turns. Fewer loops means fewer narration beats and different latency, independently of answer quality. This is a treatment effect to measure (avg_loop_count, tool_count), not a defect — and it is the expected direction, already flagged in llm_config.ts when the default dropped to medium.

​

What’s instrumented

All of this already existed or was added with the flag; no new telemetry is pending. The judge is the material upgrade over v01, which shipped with no quality metric and leaned on thumbs. It is what makes judge non-inferiority the ship gate rather than an aspiration. It must be on (JUDGE_ONLINE_ENABLED not 'false') for the whole run.

​

Decision criteria

These are proposed and need sign-off BEFORE anyone is enrolled. The flag already exists at 0% enrollment, so the deadline is the ramp, not the flag. Choosing them after seeing data is how you talk yourself into a result. Ship opus55-medium-v02 if all three hold:

1. judge_overall_mean is non-inferior to control within a margin of 0.10 on the 1–5 scale (2% of range).
2. judge_poor_pct (overall ≤ 2) and judge_fabricated_pct do not rise materially.
3. avg_cost_usd_per_turn improves — measured over priced turns (completed and uncontaminated, count in n_cost_priced), not all completed turns — or p50 and p95 latency both improve without a cost regression. If contamination_pct is asymmetric between arms, that kill condition fires first and this gate does not apply.

Cost is priced, not counted — read avg_cost_usd_per_turn, never avg_total_tokens. See the cost section above: the 5.5 arm can spend more tokens and still cost less. Note the cost columns are the one per-protocol metric here; everything else is intent-to-treat. A turn that fell back mid-loop has its cumulative tokens priced at whatever model finished it, so its dollar figure is wrong rather than merely diluted, and averaging it would corrupt the ship-gate number. Contaminated turns are therefore excluded from cost only. Read avg_cost_usd_per_turn next to n_cost_priced and contamination_pct — contamination is not random (heavy turns 529 more), so those dollars describe the turns that stayed on their arm. Kill early, without waiting for significance, if:

• failure_rate_pct separates between arms
• judge_fabricated_pct rises in the treatment arm — on financial answers this is the failure that matters most
• contamination_pct becomes asymmetric between arms (see below). This one also invalidates the cost gate rather than merely warning about it: the dollars are measured on turns that stayed on their arm, so an arm losing systematically more turns is not comparable on cost.

Do not peek-and-stop on latency. It separates almost immediately and will tempt an early call before the quality signal has any power. Open — needs a number before launch: sample size. The binding constraint is judge overall at a 0.10 margin. Compute from current completed-turns-per-day and the observed variance of ai_chat_judge_runs.overall. If the answer is “months”, widen the margin or accept a cost-only decision.

​

Contamination, and the one thing to watch

A turn can leave its arm mid-flight. The fallback chain is: primary → retry → Sonnet 5 (Anthropic API) → Bedrock (last resort). Analysis is intent-to-treat — group by assigned arm, report contamination separately — so these turns are detected, not silently re-bucketed. Bedrock stays on Opus 4.8 for both arms. Deliberate. The Bedrock path deletes thinking and output_config outright (chat-turn.ts ~552), so a Bedrock turn runs with no thinking at all. Pointing the treatment arm at a 5.5 Bedrock ID would produce a third configuration belonging to neither arm, not the treatment. Those turns are already caught: the row records us.anthropic.claude-opus-4-8, which matches neither arm’s expected model. The asymmetry risk is the real one. Both arms converge on Sonnet 5 @ max identically, so contamination is symmetric by construction — unless 5.5 429s/529s at a different rate than 4.8. A new model plausibly has tighter capacity. If the treatment arm falls back more often, contamination stops being noise and becomes a confound. Per-arm contamination_pct is the guardrail; check it in the first days, not at the end. Known record-accuracy bug, pre-existing, out of scope: on a Bedrock hop the fallback clamp writes thinking_effort = 'max' while the request had thinking deleted. Those turns are excluded from this experiment either way.

​

Internal (team) users

The team runs Opus 5.5 and is not in the experiment. Two separate things, kept separate on purpose:

• getInternalTreatmentModel() pins internal users to the treatment model. Opt-out via CHAT_MODEL_V02_INTERNAL_TREATMENT=off. Not routed through the arm table, Redis, or PostHog — it outlives the experiment and must survive the flag being killed.
• getChatModelV02Arm() returns null for internal users, so no exposure event fires and experiment_arm stays NULL. Every arm-grouped query drops them by construction rather than by someone remembering WHERE NOT internal.

Their turns stay identifiable via model + router_is_internal.

​

Router precedence

The arm wins over a router downgrade while the test runs, stamped router_reason = 'live_suppressed:experiment' (internal: live_suppressed:internal_treatment). Applied to both arms — this is the part that is easy to get wrong. control-v02’s model is DEFAULT_LLM_CONFIG.model, so control users get pinned to Opus 4.8 too. Suppressing the router for the treatment arm only would leave control taking Sonnet downgrades, and the arms would differ by model mix as well as by model. The suppression is derived from enrollment, so when the experiment ends the router is live again with nothing to remember to flip back.

​

Rollout

1. Apply the two migrations. _01 must land before the app deploys, not just before the ramp:
   • 20260922_01_chat_ab_tests.sql — used_fallback and experiment_key on ai_chat_logs, the chat_ab_tests / chat_ab_test_arms tables, both tests seeded into them, and the chat_ab_test_results view. Also drops ai_chat_experiment_arms, v01’s bespoke view, which this replaces. One file because the view reads columns created in the same change. Deploy this first or every chat turn fails to persist. markChatLogCompleted sends used_fallback on every completed turn, enrolled or not. PostgREST rejects an update naming a column that does not exist, so the completion write would throw, the failure write would throw for the same reason, and the row would be stranded in running until the stale-run reaper closes it. This is not scoped to enrolled users the way experiment_key is — it is all chat traffic.
   • 20260922_02_model_pricing_opus_5_5.sql — required before this PR merges, not before the ramp: internal users serve 5.5 from deploy. normalize_model_key maps anything containing “opus” to the 4.8 rate sheet, so Opus 5.5 gets billed at 4.8 rates. Per category that is 25% over on input, output and cache writes, and 150% over on cache reads. Blended on Silvia’s measured token mix — 82% cache reads — it is 1.40× actual. That is not just analytics: the USD cap gates Pro and Free members through the same function, so at 1.40× a member on 5.5 reaches their cap after 71% of the usage they are owed, i.e. ~29% early.
2. Sign off the decision criteria above and fix the sample size.
3. Verify the wiring with enrollment still at 0%: a real turn resolves unenrolled, the arm survives a workflow slice, and fail-closed works with PostHog unreachable.
4. Smoke the treatment arm on staging: redis SET chat-model-v02:arm opus55-medium-v02, send a few turns, confirm the working strip still narrates and no 400s. Unset when done. The pin is environment-wide — on production it would put every user on one arm.
5. Launch the experiment (sets start_date), then move enrollment 0 → 100. That order matters: PostHog counts exposures from start_date, so ramping first enrolls members whose early turns fall outside the stats window. The variant split is already 50/50 and does not change; enrollment is the only number that moves.
6. Watch contamination_pct asymmetry and failure_rate_pct daily, and run to the pre-declared sample size.

Redis kill: SET chat-model-v02:arm off — propagates in ≤5s, no deploy.

​

Teardown

Per v01’s precedent, remove the scaffolding rather than leaving the flag at 100% winner. What this test added, and what to do with each:

• lib/flags/chat-model-v02.ts — keep as the template for the next chat experiment, wired to nothing. Bump the flag key off -v02 and re-derive the arms before reuse.
• getInternalTreatmentModel + CHAT_MODEL_V02_INTERNAL_TREATMENT — remove once the winner is the production default; the pin is then a no-op.
• The arm read, exposure event, router suppression, experimentArm on the checkpoint / markChatLogCompleted / markChatLogFailed, and the experiment_arm / $feature/<flag> event properties — remove.
• The 'experiment' rung of resolveThinkingEffort and the ThinkingEffortSource member — leave. It has been removed and re-added once already; historical rows carry the value, and the column has no CHECK constraint, so read it as an open set.
• ai_chat_logs.experiment_arm, chat_ab_tests / chat_ab_test_arms, chat_ab_test_results, and the probes — keep as the historical record.

If 5.5 wins, shipping it means changing DEFAULT_LLM_CONFIG.model, and at that point re-check lib/claude-generate.ts: it uses forced tool_choice on the first iteration, which returns a 400 on Opus 5.5. It is not on the chat path and is unaffected by this experiment, but it would block a global default change.