​

ABC test: model × thinking effort (Opus 4.8 / Opus 5)

​

Outcome

• The arm read, router override (live_suppressed:experiment), $feature_flag_called exposure, and every experiment_arm / $feature/<flag> property on the completion and failure events.
• The experimentEffort rung of resolveThinkingEffort and the 'experiment' member of ThinkingEffortSource.
• experimentArm from the checkpoint, markChatLogCompleted, markChatLogFailed, and reportChatError.

• lib/flags/chat-model-effort.ts and its test suite, as the reference template for the next chat-side experiment. It is wired to nothing — see the header there for the two things to fix before reusing it (the EXPERIMENT_ARMS literals describe this concluded test, and the flag key must be bumped off -v01).
• ai_chat_logs.experiment_arm, its partial index, and the ai_chat_experiment_arms view — the historical record. Nothing writes the column any more, so it stays NULL on all new rows. 20260817_01_create_ai_chat_judge_runs.sql also references it.
• thinking_effort / thinking_effort_source / thinking_supported / duration_ms on chat_response_completed_server, and the streamStartAtMs checkpoint carry. Per §5 these stand on their own; the latter was a real bug fix (multi-slice runs reported only their last slice).

• thinking_effort_source rows written before 2026-08-20 can carry the string 'experiment', which is no longer in the TS union. The column ships no CHECK constraint, so read it as an open set.
• Router downgrades (short turns → Sonnet 5) were suppressed for enrolled users while the test ran and are live again as of 2026-08-20. Latency and model-mix comparisons that straddle that date are not apples-to-apples.

​

Original plan follows

​

The question we’re answering

​

The three arms

• control → opus48-medium isolates effort (same model).
• opus48-medium → opus5-medium isolates model (same effort).

​

What we already capture (verified against the code)

1. chat_response_completed_server carries neither thinking_effort, thinking_effort_source, nor a turn duration_ms. Without them the PostHog side cannot split latency by arm at all.
2. No variant property on any server event, so PostHog would not attribute metrics to an arm.
3. No persisted arm label on the row, so the SQL side can’t group by arm either (model+effort is a proxy for the arm, and a contaminated turn breaks the mapping).
4. No accuracy signal.

​

Design

​

1. The flag

chat-model-effort-v01
  control        -> claude-opus-4-8 @ xhigh   (today)
  opus48-medium  -> claude-opus-4-8 @ medium
  opus5-medium   -> claude-opus-5   @ medium

​

2. Resolution module — lib/flags/chat-model-effort.ts (NEW, server-only)

1. Redis override chat-model-effort:arm = off | variant name — ≤5s TTL emergency kill with no deploy. Non-negotiable here: this flag can change the model, so a bad arm is a product-quality incident.
2. Non-prod: return null (fall through to today’s config). Do not randomize on staging — it makes QA non-reproducible (“why did my answer change?”). To exercise an arm off-prod, set the Redis key from step 1 — it is read before this bail-out, which is what makes a staging smoke test possible at all. There is deliberately no per-request ?arm= override. That would be a client-supplied model + effort selector, which is exactly what was stripped from ChatRequest once already (“client-trusted LLM-config overrides that no legitimate client ever sent” — app/api/claude/types.ts). Scoped production targeting is what a percentage rollout is for. If per-session pinning is ever genuinely needed, it belongs on an internal-only debug route keyed by clerkId, not on the public chat contract.
3. PostHog (production only): client.getFeatureFlag(key, clerkId) via getPostHogFlagClient(). Bound with withTimeout(…, 2_000), cached 60s per clerkId, single-flight — a chat turn must never block on /decide. posthog-node has no personalApiKey configured here, so every cache miss is a remote round-trip (documented in lib/posthog-server.ts).
4. Fail-closed: any error, timeout, unknown variant, or missing client → null → caller keeps DEFAULT_LLM_CONFIG.model + defaultThinkingEffort. Fail-closed must equal control, so a PostHog outage degrades to today’s behavior rather than into a treatment arm.

export const EXPERIMENT_ARMS = {
  control:         { model: 'claude-opus-4-8', effort: 'xhigh'  },
  'opus48-medium': { model: 'claude-opus-4-8', effort: 'medium' },
  'opus5-medium':  { model: 'claude-opus-5',   effort: 'medium' },
} as const satisfies Record<string, { model: string; effort: ThinkingEffort }>

​

3. Wiring into the turn — three load-bearing details

• Haiku still wins — sending output_config.effort to Haiku is a 400.
• Internal users still win (see §4).
• The provider fallback clamp still wins. If the primary 529s and we swap to Sonnet 5, the re-clamp at ~line 1693 must keep running; xhigh is not valid on Sonnet 5.

​

4. Internal users: excluded

​

5. Telemetry — closing the four gaps

thinking_effort: finalThinkingEffort,          // post-clamp, what actually ran
thinking_effort_source: finalThinkingEffortSource,
thinking_supported: turnConfig.thinkingSupported,
duration_ms: Date.now() - streamStartAtMs,     // server-truth wall clock
experiment_arm: experimentArm,                 // null when not enrolled
'$feature/chat-model-effort-v01': experimentArm,

ALTER TABLE ai_chat_logs ADD COLUMN IF NOT EXISTS experiment_arm TEXT;

• provider fallback (529 → Sonnet 5 → effort re-clamped to max),
• Bedrock last-resort hop,
• Haiku capability gate forcing effort: 'off'.

​

6. Accuracy — LLM-as-judge, offline

• Nightly script sampling completed ai_chat_logs rows per arm (status = 'completed', response non-null), stratified by work_class so a tool-heavy arm isn’t judged against a chatty one.
• Judge input: prompt + response (+ interim_narration tool list for context). Start ungrounded — a rubric score (1–5) on answers-the-question / correct-use-of-the-data / no-hedging, plus a short rationale. Grounding against the user’s actual financial data is a later iteration; ungrounded is enough to detect a regression between arms, which is the decision at hand.
• Blind the judge: strip model name and effort from the input, and randomize presentation order for pairwise comparisons. A judge told it is looking at “the cheaper config” will find it worse.
• Prefer pairwise on matched prompts over absolute scoring where we can get it — pairwise preference is far more sensitive at small n than a 1–5 scale, and n is our binding constraint.
• Write scores to a chat_judge_scores table keyed by ai_chat_logs.id, so scoring can be re-run and re-versioned without touching the chat table. Store the judge model + rubric version on every row; a judge upgrade invalidates cross-period comparisons otherwise.
• Calibrate once against feedback_type before trusting it: if judge scores don’t correlate with real thumbs, the judge is measuring its own taste.

​

7. Reading the result

• Ship a treatment arm if p50 and p95 latency both improve materially, total tokens/turn do not regress, and judge score is non-inferior to control within a pre-declared margin.
• Kill an arm early if thumbs-down rate or judge score drops beyond a pre-set floor — quality regressions do not need statistical significance to be worth stopping.
• Latency-only wins with a quality regression are not a ship.

​

8. Rollout

1. Ship telemetry first (§5, gaps 1–3) with no flag. Confirm thinking_effort and duration_ms land on the event and the column populates. This alone answers “are we capturing what we need?” and is independently useful.
2. Create the flag at 100% control. Verify a real turn resolves control, that the checkpoint carries the arm across a workflow slice, and that fail-closed works with PostHog unreachable.
3. Smoke the Opus 5 arm on staging by setting the Redis key chat-model-effort:arm = opus5-medium, then sending a few turns. Confirms chat accepts claude-opus-5 with output_config.effort: 'medium' — we keep no per-model effort support table on purpose (see the ThinkingEffort docstring), so an unsupported level would be a 400 on every turn in that arm. Also confirms the arm overrides a router downgrade. Unset the key when done. Note the pin is environment-wide, not per-session: on staging that’s fine, in production it would put every enrolled user on one arm and destroy the comparison while it’s set. The production equivalent of a smoke test is step 4’s canary, not a Redis pin.
4. Canary in production: opus5-medium at 1–2%, control holding the rest. Real traffic, real config, small blast radius, clean data from turn one, and the Redis kill is still 5s away.
5. Ramp 34/33/33. Watch guardrails daily.
6. Run to the pre-declared sample size. Do not peek-and-stop on latency — latency will separate almost immediately and will tempt an early call before the quality signal has any power.
7. Ship the winner by changing DEFAULT_LLM_CONFIG, then rewrite the flag to 100% winner and leave it as a kill switch (the pattern post-signup-onboarding-experiment.ts documents), then remove it.

​

9. Tests

• Redis off beats a PostHog opus5-medium.
• Unknown/garbage variant → fall through to control config.
• PostHog timeout / throw / null client → control, never a treatment arm.
• Internal user → never calls getFeatureFlag.
• Cache + single-flight: N concurrent turns for one clerkId → one eval.

• Arm effort applies for an enrolled external user.
• Internal cohort still beats the arm.
• Haiku capability gate still beats the arm (off, supported: false).
• Fallback clamp still re-clamps an arm’s effort on a Sonnet 5 swap.

​

Open questions

1. Router precedence — recommendation is arm-wins; needs a call, because it temporarily degrades the router’s own live-mode coverage.
2. Sample size — at current chat volume, how long to detect a pairwise-preference difference? If the answer is “months”, drop to two arms (control vs opus5-medium) and test effort separately afterward.
3. Cost ceiling for the judge — nightly sample size × judge model is a real line item; cap it explicitly.