> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cfosilvia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Thinking effort ab test

# ABC test: model × thinking effort (Opus 4.8 / Opus 5)

Status: **CONCLUDED 2026-08-20 — `opus48-medium` shipped as the default.**
Owner: chat / AI platform

## Outcome

`opus48-medium` (Claude Opus 4.8 @ `medium`) won and is now the production
default: `DEFAULT_LLM_CONFIG.defaultThinkingEffort` moved `'xhigh'` → `'medium'`
with the model held at `claude-opus-4-8`. Effort was the lever that mattered;
the model contrast (`opus5-medium`) did not justify a switch.

Per §8.7, the experiment scaffolding has been removed rather than left at 100%
winner. What that entailed, and what deliberately survives:

**Removed** (all of it dead code once the default changed):

* The arm read, router override (`live_suppressed:experiment`), `$feature_flag_called`
  exposure, and every `experiment_arm` / `$feature/<flag>` property on the
  completion and failure events.
* The `experimentEffort` rung of `resolveThinkingEffort` and the
  `'experiment'` member of `ThinkingEffortSource`.
* `experimentArm` from the checkpoint, `markChatLogCompleted`,
  `markChatLogFailed`, and `reportChatError`.

**Kept:**

* `lib/flags/chat-model-effort.ts` and its test suite, as the reference
  template for the next chat-side experiment. It is wired to nothing — see the
  header there for the two things to fix before reusing it (the `EXPERIMENT_ARMS`
  literals describe this concluded test, and the flag key must be bumped off
  `-v01`).
* `ai_chat_logs.experiment_arm`, its partial index, and the
  `ai_chat_experiment_arms` view — the historical record. Nothing writes the
  column any more, so it stays NULL on all new rows.
  `20260817_01_create_ai_chat_judge_runs.sql` also references it.
* `thinking_effort` / `thinking_effort_source` / `thinking_supported` /
  `duration_ms` on `chat_response_completed_server`, and the `streamStartAtMs`
  checkpoint carry. Per §5 these stand on their own; the latter was a real bug
  fix (multi-slice runs reported only their last slice).

**Two caveats for anyone querying old data:**

* `thinking_effort_source` rows written before 2026-08-20 can carry the string
  `'experiment'`, which is no longer in the TS union. The column ships no CHECK
  constraint, so read it as an open set.
* Router downgrades (`short` turns → Sonnet 5) were suppressed for enrolled
  users while the test ran and are live again as of 2026-08-20. Latency and
  model-mix comparisons that straddle that date are not apples-to-apples.

## Original plan follows

Related: `supabase/migrations/20260803_01_add_thinking_to_ai_chat_logs.sql`,
`lib/flags/workflow-dispatch.ts`, `app/api/claude/config/llm_config.ts`,
`lib/router/model-router.ts`

## The question we're answering

Today every external chat turn runs `DEFAULT_LLM_CONFIG.model`
(`claude-opus-4-8`) at `defaultThinkingEffort = 'xhigh'`. Internal (team)
turns run at `internalThinkingEffort` (default `'medium'`, env-tunable via
`INTERNAL_THINKING_EFFORT`). That internal-vs-external split is an
*observational* comparison, not an experiment: the cohorts differ in who they
are, what they ask, and how they use the product, so it cannot answer "should
we ship a cheaper/faster configuration to everyone?".

This plan makes it a randomized ABC test over the two things we can actually
change — **which model serves the turn** and **how hard it thinks** — measured
on latency, token cost, and answer quality.

## The three arms

| Arm (PostHog variant) | Model | Effort | What it is |
| - | - | - | - |
| `control` | `claude-opus-4-8` | `xhigh` | Today's production behavior, unchanged |
| `opus48-medium` | `claude-opus-4-8` | `medium` | Effort cut, model held fixed |
| `opus5-medium` | `claude-opus-5` | `medium` | New model, effort held fixed vs. the arm above |

Adjacent pairs each vary exactly one factor, which is what makes the result
readable:

* `control` → `opus48-medium` isolates **effort** (same model).
* `opus48-medium` → `opus5-medium` isolates **model** (same effort).

We deliberately do NOT run the fourth cell (Opus 5 @ `xhigh`). Three arms at
our traffic volume already stretches the time-to-significance; the two
contrasts above answer the actual decision ("can we drop to medium?" and "is
Opus 5 better at medium?") without it. If the chain shows an interaction
worth chasing, the fourth cell is a follow-up experiment, not a fourth arm on
this one.

## What we already capture (verified against the code)

| KPI | Where it already lives | Gap |
| - | - | - |
| Effort actually used | `ai_chat_logs.thinking_effort`, `.thinking_supported`, `.thinking_effort_source` (migration `20260803_01`) | Not on the PostHog event |
| Model actually served | `ai_chat_logs.model`, `.router_effective_model` | Not enough alone — see contamination |
| Latency (wall clock) | `completed_at - created_at`; view `ai_chat_thinking_latency` | Not on the PostHog event |
| Latency (TTFT) | client `chat_response_completed` (`hooks/use-claude-chat.ts`, `requestSentAt` → `firstTokenAt`) | Client-only — dies on tab close / SSE drop |
| Tool + loop volume | `ai_chat_logs.loop_count`, `interim_narration.totalElapsedMs` | — |
| Token usage | `ai_chat_logs.{input,output,cache_creation,cache_read,total}_tokens` **and** the `chat_response_completed_server` PostHog event (`lib/claude/chat-turn.ts` \~4985) | — |
| Thumbs feedback | `ai_chat_logs.feedback_type`, client `chat_message_feedback` | Too sparse to power a metric alone |
| Accuracy | nothing | Needs an LLM judge (§5) |

**Four gaps** to close before the experiment is readable:

1. `chat_response_completed_server` carries neither `thinking_effort`,
   `thinking_effort_source`, nor a turn `duration_ms`. Without them the
   PostHog side cannot split latency by arm at all.
2. No variant property on any server event, so PostHog would not attribute
   metrics to an arm.
3. No persisted arm label on the row, so the SQL side can't group by arm
   either (model+effort is a *proxy* for the arm, and a contaminated turn
   breaks the mapping).
4. No accuracy signal.

Closing (1) is a \~10-line change and is worth doing whether or not we run
this experiment.

## Design

### 1. The flag

One PostHog **multivariate** flag whose variant selects a `(model, effort)`
pair — not two independent flags. Two flags would let a user land in the
un-tested Opus 5 @ `xhigh` cell and would make the cohorts un-analyzable.

```
chat-model-effort-v01
  control        -> claude-opus-4-8 @ xhigh   (today)
  opus48-medium  -> claude-opus-4-8 @ medium
  opus5-medium   -> claude-opus-5   @ medium
```

Rollout by `clerkId`: PostHog hashes the distinct id, so a user consistently
gets one arm — the same property `lib/flags/workflow-dispatch.ts` relies on,
and it matters more here than in a UI test. Chat is conversational, and
flipping model or reasoning depth *between turns of one conversation* would
both confuse the user and mix arms inside a single session.

Start `control` 100 / others 0, then ramp to 34/33/33.

The variant → `(model, effort)` mapping lives in **our** code, never in the
flag payload, so a fat-fingered PostHog edit can't send an invalid model
alias or an invalid `output_config.effort` to Anthropic.

### 2. Resolution module — `lib/flags/chat-model-effort.ts` (NEW, server-only)

Copy the layered shape of `lib/flags/workflow-dispatch.ts`; it is the house
pattern and already solved the hard parts (single-flight, bounded cache,
timeouts, kill switch):

1. **Redis override** `chat-model-effort:arm` = `off` | variant name — ≤5s
   TTL emergency kill with no deploy. Non-negotiable here: this flag can
   change the model, so a bad arm is a product-quality incident.
2. **Non-prod**: return `null` (fall through to today's config). Do *not*
   randomize on staging — it makes QA non-reproducible ("why did my answer
   change?"). To exercise an arm off-prod, set the Redis key from step 1 —
   it is read *before* this bail-out, which is what makes a staging smoke
   test possible at all.

   There is deliberately **no** per-request `?arm=` override. That would be
   a client-supplied model + effort selector, which is exactly what was
   stripped from `ChatRequest` once already ("client-trusted LLM-config
   overrides that no legitimate client ever sent" —
   `app/api/claude/types.ts`). Scoped *production* targeting is what a
   percentage rollout is for. If per-session pinning is ever genuinely
   needed, it belongs on an internal-only debug route keyed by clerkId,
   not on the public chat contract.
3. **PostHog** (production only): `client.getFeatureFlag(key, clerkId)` via
   `getPostHogFlagClient()`. Bound with `withTimeout(…, 2_000)`, cached 60s
   per clerkId, single-flight — a chat turn must never block on `/decide`.
   `posthog-node` has no `personalApiKey` configured here, so every cache
   miss is a remote round-trip (documented in `lib/posthog-server.ts`).
4. **Fail-closed**: any error, timeout, unknown variant, or missing client →
   `null` → caller keeps `DEFAULT_LLM_CONFIG.model` +
   `defaultThinkingEffort`. Fail-closed must equal `control`, so a PostHog
   outage degrades to today's behavior rather than into a treatment arm.

Validate against an allowlist before anything reaches the API — the same
defensive posture as `readInternalThinkingEffortEnv()`:

```ts theme={null}
export const EXPERIMENT_ARMS = {
  control:         { model: 'claude-opus-4-8', effort: 'xhigh'  },
  'opus48-medium': { model: 'claude-opus-4-8', effort: 'medium' },
  'opus5-medium':  { model: 'claude-opus-5',   effort: 'medium' },
} as const satisfies Record<string, { model: string; effort: ThinkingEffort }>
```

An unrecognized variant returns `null` (fall through), never a raw
passthrough.

### 3. Wiring into the turn — three load-bearing details

Resolve the arm **once per turn**, next to the existing `isInternalUser()`
lookup in `lib/claude/chat-turn.ts` (\~line 2985), then thread it exactly the
way `isInternal` already is.

**a. Snapshot it into the checkpoint.** `ChatTurnCheckpoint` (\~line 2027)
carries `isInternal?: boolean` for precisely this reason: a resumed workflow
slice re-derives the turn config, and a fresh flag read mid-turn could hand
slice 2 a different model than slice 1. Add `experimentArm?: string` beside
it, set it in the checkpoint literal (\~line 4746), and restore it in the
resume branch (\~line 3542) with the same `?? null` back-compat treatment.

**b. Compose with `resolveThinkingEffort`, don't bypass it.** That function
(`llm_config.ts`) is the single authority, with precedence: capability gate
(Haiku can't think) → internal cohort → fallback clamp → default. The arm
slots in **below** the capability gate and **below** the internal cohort:

* Haiku still wins — sending `output_config.effort` to Haiku is a 400.
* Internal users still win (see §4).
* The provider fallback clamp still wins. If the primary 529s and we swap to
  Sonnet 5, the re-clamp at \~line 1693 must keep running; `xhigh` is not
  valid on Sonnet 5.

Add a `ThinkingEffortSource` value (`'experiment'`) so the persisted
`thinking_effort_source` records *why* a turn ran where it ran. The column
already distinguishes `default` / `internal_lower` / `fallback_clamp`, and
the migration deliberately ships no CHECK constraint — so this needs no
schema change.

**c. Decide who wins between the arm and the router — and write it down.**
`decideChatModel` already overrides the model for short turns in `live` mode.
If both are live, an `opus5-medium` user whose turn is classified `short`
gets Sonnet and the arm means nothing for that turn. Recommendation: **the
arm wins over a router downgrade while the test runs**, stamped as
`router_reason = 'live_suppressed:experiment'` so the suppression is visible
and the router's own accuracy analysis can exclude those turns.

### 4. Internal users: excluded

Internal users stay pinned to `internalThinkingEffort` and are **not**
enrolled. The cohort is a few dozen accounts whose usage looks nothing like a
customer's (smoke tests, deliberate edge cases), so they add noise rather
than power — and they'd otherwise lose the low-latency config the pin exists
to give them. `resolveThinkingEffort` already checks the internal cohort
first, so this falls out of the precedence above for free; the enrollment
call should *also* skip the PostHog read entirely for internal users, keeping
the experiment's denominator clean (the "short-circuit before
`getFeatureFlag`" rule the sidebar-CTA hook documents).

### 5. Telemetry — closing the four gaps

**Gap 1 + 2 — the PostHog event.** Extend the existing
`CHAT_EVENTS.RESPONSE_COMPLETED` capture in `chat-turn.ts` (\~line 4985); do
not add a new event. It already carries model, tokens, loop count, and tool
counts, so we only need:

```ts theme={null}
thinking_effort: finalThinkingEffort,          // post-clamp, what actually ran
thinking_effort_source: finalThinkingEffortSource,
thinking_supported: turnConfig.thinkingSupported,
duration_ms: Date.now() - streamStartAtMs,     // server-truth wall clock
experiment_arm: experimentArm,                 // null when not enrolled
'$feature/chat-model-effort-v01': experimentArm,
```

That `$feature/<key>` property is what makes PostHog attribute the event to
an arm — `posthog-node` writes exactly this shape internally
(`dist/client.js`), and without it an experiment metric built on this event
shows no variant breakdown. Note `chat_response_completed_server` fires via
`after()` with an explicit `timestamp`, so this stays off the user's critical
path.

Set `send_event: false` on the flag read and fire exposure explicitly at
enrollment, so exposure is recorded once per turn at a known point rather
than incidentally on every cache miss.

**Gap 3 — persist the arm on the row.** Add one column:

```sql theme={null}
ALTER TABLE ai_chat_logs ADD COLUMN IF NOT EXISTS experiment_arm TEXT;
```

Written by `markChatLogCompleted` through the same
omit-the-key-when-absent pattern the `thinking` block already uses, so
pre-migration rows and non-enrolled turns stay NULL. This is what makes the
SQL analysis possible: `model` + `thinking_effort` is only a *proxy* for the
arm, and it breaks exactly on the contaminated turns we most need to find.

**Contamination — the reason the arm label matters.** A turn can leave its
arm mid-flight and the row will still look plausible:

* provider fallback (529 → Sonnet 5 → effort re-clamped to `max`),
* Bedrock last-resort hop,
* Haiku capability gate forcing `effort: 'off'`.

With `experiment_arm` persisted alongside the realized `model` /
`thinking_effort`, those turns are *detectable* (assigned arm ≠ realized
config) and can be reported as a contamination rate rather than silently
diluting the treatment effect. Analyze intent-to-treat (group by assigned
arm) and report contamination separately.

### 6. Accuracy — LLM-as-judge, offline

Judge **offline in a batch job, never inline.** Inline judging would add
latency to the very metric we're measuring, double the token bill, and put a
second model call on the critical path of every turn.

Shape:

* Nightly script sampling completed `ai_chat_logs` rows per arm (`status =
  'completed'`, response non-null), stratified by `work_class` so a
  tool-heavy arm isn't judged against a chatty one.
* Judge input: `prompt` + `response` (+ `interim_narration` tool list for
  context). Start ungrounded — a rubric score (1–5) on
  answers-the-question / correct-use-of-the-data / no-hedging, plus a short
  rationale. Grounding against the user's actual financial data is a later
  iteration; ungrounded is enough to detect a *regression* between arms,
  which is the decision at hand.
* **Blind the judge**: strip model name and effort from the input, and
  randomize presentation order for pairwise comparisons. A judge told it is
  looking at "the cheaper config" will find it worse.
* Prefer **pairwise** on matched prompts over absolute scoring where we can
  get it — pairwise preference is far more sensitive at small n than a 1–5
  scale, and n is our binding constraint.
* Write scores to a `chat_judge_scores` table keyed by `ai_chat_logs.id`, so
  scoring can be re-run and re-versioned without touching the chat table.
  Store the judge model + rubric version on every row; a judge upgrade
  invalidates cross-period comparisons otherwise.
* Calibrate once against `feedback_type` before trusting it: if judge scores
  don't correlate with real thumbs, the judge is measuring its own taste.

### 7. Reading the result

**SQL view** — extend the existing `ai_chat_thinking_latency` pattern rather
than inventing a new shape. Group by `experiment_arm` (assigned) instead of
`model × thinking_effort` (realized), keep the `loop_count <= 5 → 'short'`
work-class bucketing that view already uses, and keep p50/p95 alongside the
mean. Per arm, per work class: `n`, `p50_seconds`, `p95_seconds`,
`avg_output_tokens`, `avg_total_tokens`, `avg_loop_count`, contamination
rate, thumbs rate, and mean judge score.

The work-class split is load-bearing for the same reason the existing view
documents: a turn that ran 20 tool loops is slow regardless of arm, so
without bucketing you measure work mix, not the treatment.

**PostHog experiment** — create it against `chat-model-effort-v01` with
`chat_response_completed_server` as the metric source. Use PostHog for the
"is this significant" read and the funnel/retention side effects; use SQL for
the latency percentiles and token math, which is where the detail lives.

**Decision criteria — write these down before launching**, not after seeing
the data:

* Ship a treatment arm if p50 and p95 latency both improve materially, total
  tokens/turn do not regress, **and** judge score is non-inferior to control
  within a pre-declared margin.
* Kill an arm early if thumbs-down rate or judge score drops beyond a
  pre-set floor — quality regressions do not need statistical significance
  to be worth stopping.
* Latency-only wins with a quality regression are **not** a ship.

**Guardrail metrics** to watch regardless: error/timeout rate
(`chat_response_failed_server`), `chat_model_fallback` rate (a new model may
529 at a different rate, which would confound everything), and
`chat_checkpoint_too_large`.

### 8. Rollout

1. Ship telemetry first (§5, gaps 1–3) with **no flag**. Confirm
   `thinking_effort` and `duration_ms` land on the event and the column
   populates. This alone answers "are we capturing what we need?" and is
   independently useful.
2. Create the flag at 100% `control`. Verify a real turn resolves `control`,
   that the checkpoint carries the arm across a workflow slice, and that
   fail-closed works with PostHog unreachable.
3. Smoke the Opus 5 arm **on staging** by setting the Redis key
   `chat-model-effort:arm = opus5-medium`, then sending a few turns. Confirms
   chat accepts `claude-opus-5` with `output_config.effort: 'medium'` — we
   keep no per-model effort support table on purpose (see the
   `ThinkingEffort` docstring), so an unsupported level would be a 400 on
   every turn in that arm. Also confirms the arm overrides a router
   downgrade. Unset the key when done.

   Note the pin is environment-wide, not per-session: on staging that's
   fine, in production it would put every enrolled user on one arm and
   destroy the comparison while it's set. The production equivalent of a
   smoke test is step 4's canary, not a Redis pin.
4. Canary in production: `opus5-medium` at 1–2%, `control` holding the rest.
   Real traffic, real config, small blast radius, clean data from turn one,
   and the Redis kill is still 5s away.
5. Ramp 34/33/33. Watch guardrails daily.
6. Run to the pre-declared sample size. Do not peek-and-stop on latency —
   latency will separate almost immediately and will tempt an early call
   before the quality signal has any power.
7. Ship the winner by changing `DEFAULT_LLM_CONFIG`, then rewrite the flag to
   100% winner and leave it as a kill switch (the pattern
   `post-signup-onboarding-experiment.ts` documents), then remove it.

### 9. Tests

Mirror `lib/flags/__tests__/workflow-dispatch.test.ts` — it already
establishes the mocking shape for `getPostHogFlagClient` + `redis`:

* Redis `off` beats a PostHog `opus5-medium`.
* Unknown/garbage variant → fall through to control config.
* PostHog timeout / throw / null client → control, never a treatment arm.
* Internal user → never calls `getFeatureFlag`.
* Cache + single-flight: N concurrent turns for one clerkId → one eval.

Plus, in the `llm_config` thinking suite (19 tests today):

* Arm effort applies for an enrolled external user.
* Internal cohort still beats the arm.
* Haiku capability gate still beats the arm (`off`, `supported: false`).
* Fallback clamp still re-clamps an arm's effort on a Sonnet 5 swap.

And a golden-path assertion in `chat-turn-golden.test.ts` that the arm
survives a checkpoint round-trip (the `isInternal` snapshot tests are the
template).

## Open questions

1. **Router precedence** — recommendation is arm-wins; needs a call, because
   it temporarily degrades the router's own live-mode coverage.
2. **Sample size** — at current chat volume, how long to detect a
   pairwise-preference difference? If the answer is "months", drop to two
   arms (control vs `opus5-medium`) and test effort separately afterward.
3. **Cost ceiling for the judge** — nightly sample size × judge model is a
   real line item; cap it explicitly.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.