Phase 2 — Chat stream continuity (resume, chat-switching, multi-tab)
Successor todocs/plans/vercel-workflows-rollout.md (W0–W5, chat + radar both
at 100% workflow as of 2026-07-25). That plan listed “resumable chat streams
(getReadable({startIndex})) replacing 2s-partial-persist recovery” as Phase 2.
This plan scopes it — and widens it, because the same missing primitive is
behind three separate user-facing problems, not one.
One-line thesis: the durable stream already exists and is already
re-attachable; what’s missing is that nothing outside the current page
session knows its address. Persist the address, and reload-resume,
chat-switching, and multi-tab all fall out of the same change.
1. What already exists (do not rebuild)
W4.5 shipped the hard half of resumable streams to serve the sliced transport.
The load-bearing invariant, already enforced: 1 SSE frame == 1 durable
chunk (the workflow sink frames before writing). So the client’s count of
complete frames received is the chunk index to resume from — no separate
bookkeeping.
The single missing fact
runId and the resume token are read from response headers into a local
variable inside parseAnthropicStreamWithResume (lib/anthropic-stream.ts:345).
They are never persisted — not in localStorage, not in sessionStorage, not
server-side. Verified by grep across use-claude-chat.ts,
anthropic-stream.ts, shared-chat-context.tsx.
Consequence: re-attach works only within one continuous page session. Any
event that destroys the JS context — reload, tab close, navigation, crash —
also destroys the only pointer to a stream that is still running and still
readable on the server.
2. Why it’s worth doing — measured, not assumed
Turns are long
7 days, 20,549 completed turns (ai_chat_logs, completed_at - created_at):
Half of all turns take over two minutes. We are asking users to hold one tab
open, foregrounded, untouched, for that long.
They don’t — and we can count it
chat_response_failed, 7 days, by error_type:
~282/day. 1,612 of the 1,976 had partial text (
partial: True) — the user
watched an answer begin, then break.
(chat_response_completed_server also runs ~17–19%/day above the client-side
chat_response_completed. Suggestive, but discounted as evidence: client
events are systematically undercounted by ad blockers and unflushed unloads.
The recoverable_disconnect count is the number to trust.)
What today’s fallback does (be fair — it is not broken)
On disconnect,use-claude-chat.ts:1770 classifies the error as recoverable
and deliberately shows no error: partial text stays, status returns to
idle, and onRecoverableDisconnect (shared-chat-context.tsx:573) starts a
poller. On reload, loadSession sees status === 'running' and self-schedules
the same poll every 1500ms (shared-chat-context.tsx:1268-1279).
So this work is an upgrade, not a bug fix. Two things are genuinely wrong
with the poller, and they are the actual justification:
- Text arrives in lumps. 1.5s poll granularity stacked on the 2s partial-persist throttle. After a disconnect the user stops watching Silvia think and starts watching paragraphs appear. Live tool activity and the work-trace pill do not update at all.
- It is a full session re-fetch, repeated. Every tick re-selects every
row of the session with the full ~25-column list
(
app/api/chat/sessions/[sessionId]/route.ts:57). A turn with 60s remaining costs ~40 complete session queries, per disconnected client. Cost scales with users and with session length — the same shape we rejected for Stop.
3. The three problems, one primitive
3a. Reload / tab-close during a turn
Covered above. Today: lumpy DB polling. Target: re-attach to the live stream.3b. Switching chats mid-turn
Already partly solved — do not regress it. CFO-6624 madeloadSession
deliberately not abort, and scoped the streaming flag to the session on
screen:
SharedChatProvider is mounted once
(app/(app)/providers.tsx:34), so there is exactly one useClaudeChat
instance, and starting a new send aborts the previous stream by design:
3c. Multi-tab
Each tab is its own JS context, its own provider, its own stream. Today that means tab B has no idea what tab A is doing: it sees an in-flight turn only through 1.5s polling, and its sidebar goes stale against sessions created elsewhere. The AI SDK’s resumable-streams documentation states the property that makes this tractable: “Multiple clients can connect to the same stream simultaneously.” Multi-tab is not a fight against the design — it is the design, once the stream has a durable address. Both tabs re-attach to the same run and both stream live. Its stated caveat is about the write side, not the read side: concurrent sends race, and the mitigation is to clear the active stream id when a new stream starts. That maps directly onto ourrun_id uniqueness.
4. Prior art — vercel/ai-chatbot and the AI SDK
Checked because we should not invent an architecture that a reference implementation has already settled.vercel/ai-chatbot package.json carries "resumable-stream": "^2.2.12" and
"redis": "^6.1.0". In app/(chat)/api/chat/route.ts:
GETs /api/chat/[id]/stream on mount, the
server looks up the chat’s activeStreamId and calls resumeExistingStream(),
returning 204 No Content when there is nothing active. Redis holds the
stream body with a configurable TTL.
Two conclusions:
- Our proposed shape is the canonical one — persist a stream address on the chat row server-side; resume by looking it up on load. We arrived at it independently; this is confirmation, not a new dependency.
- We should NOT adopt
resumable-stream+ Redis. That package exists to manufacture a durable, replayable stream on top of ephemeral functions. Vercel Workflows already gives us exactly that, chunk-indexed, with auth we have already built. Adding Redis here would be re-solving a solved problem and adding a second source of truth.
return new Response(null, { status: 204 }) — the write
side persists stream ids, the read side is disabled. Read that as “this is
fiddly enough that the reference impl left it dark,” not as “it doesn’t work.”
It raises the bar on our verification, it does not change the design.
The docs also restate, independently, the conclusion we reached empirically
with the cancel probe: client aborts only disconnect, they do not cancel work,
so a dedicated stop endpoint is required. We already have /api/claude/stop.
4b. Prior art — pingdotgg/t3code
Read because it is our problem with the difficulty turned up: an agent harness driving many concurrent sessions, observed from web, mobile and desktop at once. Source read directly (apps/server/src/ws.ts,
packages/client-runtime/src/state/threads.ts).
Their shape. subscribeThread(threadId, afterSequence?) filters a global
domain-event PubSub by aggregateId, so N clients are N independent
subscribers — multi-client is structural, not bolted on. The client loads a
thread snapshot over HTTP, then subscribes over WebSocket passing that
snapshot’s sequence; the server replays persisted events after it, then goes
live.
The bug they document twice — and our transport is immune to. Two long
comments warn that live delivery must be attached BEFORE reading the
snapshot/replay, or events published during the load window fall into a hole
(past the persisted tail the replay read, before the live subscription
exists). Their fix: fork live into an unbounded scoped queue first, emit
catch-up, drain the buffer, and dedupe by sequence on the client.
We have no such seam. The Workflows durable stream is one ordered,
chunk-indexed log covering history and live, so getReadable({startIndex})
returns both in a single stream. This is a reason NOT to be tempted into
splitting our handshake into “snapshot, then subscribe” — the most
error-prone part of their design is something we get for free.
Three policies worth importing:
- A gap threshold deciding replay vs snapshot.
SHELL_RESUME_MAX_GAP = 1_000— past that, per-event replay costs more than a fresh snapshot, so they snapshot instead; and a cursor ahead of server state is treated as invalid and reset. We replay from index 0 unconditionally, which on a long turn is thousands of frames — the very thing that forces the damper catch-up in P2.3. We cannot adopt their rule yet because our DB snapshot carries no frame index; persistingframes_writtennext to the partial text would unlock snapshot + attach-at-index. Cheap, and worth doing if replay-from-0 proves slow. - An explicit synchronizing→live state.
requestCompletionMarker: truemakes the server inject a{kind:"synchronized"}sentinel after catch-up; the client renders a distinct synchronizing state and flips to live on it. Better than P2.3’s current heuristic (snap the damper when the buffer exceeds a threshold), which only guesses where replay ends. Reconsider the UX: a visible “catching up” beats a silent fast-forward. - Re-subscribe on app-foreground.
resubscribefiltered on"application-active". A backgrounded tab or phone kills the connection with no reload, so nothing re-attaches until the user acts — a trigger this plan had missed entirely. Avisibilitychangere-attach is a few lines and likely covers a common mobile case. Added to P2.3.
5. Locked design decisions
- The workflow run is the stream. No
resumable-stream, no Redis stream storage, no second durability layer.getReadable({startIndex})is the transport. - The address is persisted server-side, on the row — a new
workflow_run_idcolumn onai_chat_logs. NotlocalStorage. Server-side survives cache clears, works cross-device, keeps no bearer token in browser storage, and makes ownership checkable. - Resume tokens are minted on demand, never stored. Session load returns
a freshly minted token for a running row the caller owns. Tokens stay
short-lived and bound to
(workflowRunId, clerkId)exactly as today. - Re-attach from
startIndex=0on a cold client. After a reload the client has no text, so there is no index to reconcile — replay the whole stream. Index tracking stays only for the warm in-page loop that already works. - A running row with a run id renders EMPTY and lets the replay fill it. Never render DB partial text and replay into the same bubble. (See Risks — this is the highest-probability bug in the plan.)
- The poller stays as the fallback, and a run id is NOT a promise that
re-attach will work. S1 explicitly allows retention to be shorter than a
turn, so a row can carry an address whose stream is already gone. The
client therefore treats re-attach as an attempt:
attachToRunperforms its handshake BEFORE writing any hook state and returns'unavailable'on 404 / 403 / 204 / network error, and the caller then starts the poller exactly as it does today. A stranded turn — neither replay nor polling — is the failure this rule exists to prevent, and it is an E2E case, not just a unit one. The poller is deleted in the last PR, not the first. - Chat-switching keeps CFO-6624’s semantics. Background generation must
continue;
isLoadingstays scoped tostreamingSessionId === sessionId. Re-attach makes the abort recoverable; it does not remove the abort. - Read is multi-client; write is unserialized, and that is explicit.
Any number of tabs may re-attach to a run. Concurrent sends are NOT
serialized:
run_iduniqueness only makes each workflow invocation distinct, so two tabs sending into one session produce two independent running rows, both legal. This plan does not add a session-scoped lock. What it does guarantee is a deterministic read: the handshake keys on the latest row bycreated_at, so the newest send is the one a reloading client re-attaches to. The consequence to accept, in writing: an older concurrent turn keeps running and persists normally, but no client will re-attach to its stream — it lands via the completed row on the next load. Serializing sends is a separate project (see Out of scope). - Flagged, kill-switchable, same control plane as W0–W5
(
lib/flags/workflow-dispatch.tspattern: Redis override → PostHog → env).
6. Spikes — answer before building (P2.0)
Both are load-bearing and neither is documented. The PR-0 cancel probe is the precedent: one afternoon of probing prevented building the wrong Stop architecture. Extendworkflows/smoke.ts with a ?mode=resumeprobe run, then record answers
in this doc:
- S1 — Retention. How long after a run completes does
getReadable()still return the frames? And while it is still running, is there any cap? This defines whether re-attach is “in-flight only” (fine) or also covers just-finished turns. - S2 — Concurrent readers. Two simultaneous
getReadable({startIndex: 0})on one run: do both receive every frame independently, or does the second steal/disturb the first’s position? This one gates 3c entirely. If readers are exclusive, multi-tab needs a different design (one leader tab + BroadcastChannel fan-out) and P2.4 changes shape. - S3 — Cost. A re-attached invocation held open for the remaining turn duration, vs ~40 full session queries. Measure both; the poller removal should be a saving, and we should know the number before claiming it.
How to run it — BUILT
The probe exists (workflows/smoke.ts::resumeProbeWorkflow plus the reader
half in app/api/internal/workflow-smoke/route.ts). It needs a Preview
deploy: getReadable() is a server API against real workflow
infrastructure, so this cannot be answered locally or in a unit test.
frame N/12 once per second, waits 3s so the run is
genuinely mid-flight, then attaches two readers concurrently at
startIndex: 0 and drains both. The verdict field is independentReaders:
true→ every tab may re-attach; P2.4’s read path is free.false→ readers interfere. P2.4 needs a leader tab + BroadcastChannel fan-out, and P2.3 needs rework, because on reload the old POST pump may still be draining when the reloaded client attaches.
gone: true
means the stream is no longer readable, which bounds what the handshake may
advertise.
RESULT — S2 ANSWERED: readers ARE independent (2026-07-29)
getReadable({startIndex: 0}) calls on one mid-flight run
each received every frame, in order, neither stalled. The gate is cleared,
in the good direction:
- P2.4’s read path is free. Every tab re-attaches on its own; no leader election, no BroadcastChannel fan-out of frames. PR F shrinks to sidebar invalidation only.
- P2.3 stands as designed. The reload case is safe: a still-draining POST pump does not block or disturb a newly attached reader, which was the risk that would have forced a rework.
- Keep the handshake a stateless mint (decision 8). Nothing needs to arbitrate who reads.
/api/claude/stream invocation, which is the same shape — but the first live
multi-tab test is still an E2E case, not something this probe discharged.
An earlier run of this probe returned independentReaders: false with both
readers at count: 0. That was a false negative from the probe’s own decoder
(the durable stream is an object stream of strings, not bytes — see the
asText note in the route). Investigating it also confirmed the production
resume path is healthy: prod logs show the same run id re-attaching at
increasing startIndex (82 → 155, 68 → 84) across many users, all 200.
Still open: S1 (retention window) and S3 (cost). Neither gates the design;
S1 bounds what the handshake may advertise, and decision 6 already assumes a
run id is not a promise.
S1.5 — the tail probe (CFO-7088, added 2026-07-30)
Prod evidence (CFO-7088, investigated in a parallel session) split S1 open: a re-attach to a COMPLETED run atstartIndex: 107 returned 200 and then
hung with zero frames, repeatedly, pinned at the same index — while this
probe’s own startIndex: 0 replay of a completed run returns everything in
285 ms. The user saw a truncated answer, the client silently finalized the
partial as success, and the re-asks burned allowance.
Two candidate causes, needing different fixes:
- Client frame-count drift — the client believed it had consumed 107
frames but the durable log held fewer, so it asked for a chunk past the
end, and a reader positioned at-or-past the end waits forever. This would
violate the
1 SSE frame == 1 durable chunkinvariant the resume index rests on. - SDK mid-log replay broken on completed runs — the index was right,
but
getReadable({startIndex: N>0})on a finished run never serves the tail, whilestartIndex: 0does.
&startIndex=N. On a completed 12-frame subject:
stalled: true at 11 → cause 2 (fix: the stream route
serves the tail + done from the persisted row when the turn is already
completed). Clean at 11 but stalled at 12/13 → at-or-past-the-end waits
forever → prod’s client overshot the real chunk count → cause 1 (fix the
counting, plus the same row-fallback as defense). Whether close() occupies
its own chunk index is also settled by which of 11/12 yields the close.
RESULT (2026-07-30, all three run against the day-old completed subject):
A closed durable log never hangs any reader, at any index — including an
overshooting one. Both simple causes are eliminated: mid-log replay works,
and at/past-the-end readers terminate immediately. (Bonus: retention ≥ ~26h,
so S1 closes — the 15-min reaper window, not stream retention, is the binding
constraint on what the handshake may offer.)
Surviving hypothesis: the prod runs’ streams were never closed. The only
state in which a reader waits forever is an UNCLOSED stream with the reader
at-or-past its current tail — which is ordinary live-tailing behavior, hung
off the end of a log whose producer never wrote the end. That matches prod
exactly: 200 + zero frames at a pinned index, on a second hop two minutes
later, minutes after
ai_chat_logs said completed. It also means the client’s
107 may have been CORRECT (107 frames, 107 chunks, waiting for a 108th that
never comes) — the frame==chunk invariant is no longer the prime suspect.
The producer difference fits too: the probe subject writes and closes in ONE
step invocation; the chat workflow releases the writer between slices and
closes in the final slice, and that close’s failure path is a swallowed
console.warn. Whether the writer-handoff close lands durably for LATE readers
is exactly what the next test measures — using ?mode=sliced, which already
exists and reproduces chat’s handoff pattern (3 slice writes across separate
step invocations + a final close step; 4 chunks, indices 0–3):
wrun_01KYTJ7V46MWA1K5NTQEF1NGTZ): fully healthy. startIndex:0 live-tailed
through completion and returned all 4 frames including the final step’s write;
startIndex:3 returned exactly the end line; startIndex:4 (at end) closed
cleanly in 119 ms. The writer-handoff pattern — release between slices, close
from a later step — is exonerated end to end. (An earlier subject produced
confusing zero-frame results; startIndex:0 revealed it was a PLAIN smoke run
— the mode=sliced param had been dropped from the minting request — and all
its results were ordinary past-the-end behavior. Count chunks with a full
replay before interpreting tail results.)
ROOT CAUSE, found in code (workflows/chat.ts): every generic mechanism is
healthy, and the defect is a race the probes cannot reproduce because the
smoke subjects AWAIT their close. The chat sink’s close() is
fire-and-forget:
sink.close() followed immediately by the
step function returning. The checkpoint path carefully drains
(await lastWrite; await writer.ready) before releasing the writer — the
final path awaits nothing: neither the queued tail writes (lastWrite)
nor the close. When the invocation freezes before those flush, the durable
log is left UNCLOSED (and possibly missing tail frames). A late re-attacher
then tails an unclosed log at its end and waits forever — exactly prod’s
signature: 200, zero frames, pinned index, still hanging minutes later,
intermittent (a race most closes win), and perfectly silent (a frozen
invocation logs nothing; prod shows zero [Chat Workflow] close warnings).
Fix — BUILT INTO #3047 (2026-07-30), both halves + regression tests proven
to fail without them:
workflows/chat.ts(~6 lines): capture the close promise in the sink; in the final-slice branch,await lastWritethen await the close before the step returns — the exact discipline the checkpoint path already applies.- Client guard, unconditional (same guard P2.3 needs — below).
- Defense in depth, optional:
/api/claude/streamchecks the row by wrun — if the turn is terminal, bound the read instead of trusting an unclosed log; covers any future close-loss cause.
done event
must NEVER finalize as success; it routes to the recoverable-disconnect /
history-hydration path. The give-up paths in parseAnthropicStreamWithResume
currently return silently, so use-claude-chat.ts finalizes partial text as
a completed answer — that is the user-visible truncation. And P2.3’s planned
404/403/204 fallback would sail past a 200-with-empty-stream the same way:
attachToRun must also treat zero-frames-no-done as a failed attempt.
Findings — terminated (2026-07-27)
Not a multi-tab bug. 38 occurrences in 3 days across 36 distinct users,
only 8 of which coincided with a concurrent turn by the same user. The two-tab
test that surfaced it almost certainly hit this rather than caused it.
It is an upstream stream termination, not a client disconnect. The value is
a raw error.message reaching markChatLogFailed through the generic else
branch in chat-turn.ts (userFriendlyError = error.message || "Stream error", errorType = "stream_error"). TypeError("terminated") is what undici
throws when a fetch body dies — here the server→Anthropic connection. A client
disconnect cannot produce it: on the workflow transport the turn writes to the
durable stream, not to the client’s socket, so it is entirely decoupled from
whether anyone is listening.
Open question, deliberately NOT fixed here. isTransientConnectionError
(chat-turn.ts) checks err.code ?? err.cause?.code against a socket-code set
including ECONNRESET, and lib/claude-service.ts explicitly documents that
TypeError("terminated") carries cause.code === 'ECONNRESET'. So these
should already be retried. Two possibilities, and the fix differs:
- Retries exhausted — classification works and the upstream connection is
genuinely flaky. Adding
terminatedto the message regex changes nothing. - Cause stripped in transit — the error arrives without
cause.code, falls past the socket-code check, and the message regex (request timed out|connection error|socket hang up|fetch failed) does not matchterminated, so it is never retried.
connection_timeout (the
retryable path) against stream_error counts for these turns, and check
whether retry attempts were logged. Shipping the one-line regex addition
without that evidence is a guess.
Also worth reconciling: all 38 rows have completed_at IS NULL, because
markChatLogFailed does not stamp it — unlike the completion and stop paths,
which both do.
7. PR sequence
P2.1 — Persist the run id (ships dark) — BUILT (migration 20260728_03)
- Migration:
workflow_run_id textonai_chat_logs. NO INDEX — an earlier draft of this plan called for a partial index on(clerk_id, workflow_run_id); that was dropped before shipping and the migration deliberately has none. Every path reaches the row another way (the write is by primary key; the read has the column in the target list with its predicate already served byidx_ai_chat_logs_clerk_session_status), so an index would only make the populating write non-HOT on every workflow turn in exchange for zero reads. Addable later,CONCURRENTLY, with no code change. Migration lands and is verified applied BEFORE the code that READS it — the Preferences-widget incident (release #2333) shipped a feature ahead of its column and broke in prod. The WRITER is exempt: it is best-effort and warns-and-continues on a missing column, which is why P2.1 and P2.2 could ship together. - Write the id when the workflow starts. Nothing reads it yet.
- Backfill: none. Old rows resume via the poller.
P2.2 — Mint-on-load + the resume handshake (flagged, dark)
GET /api/chat/sessions/[sessionId]additionally returns, for arunninglatest row owned by the caller and carrying aworkflow_run_id, a freshly minted resume token. Ownership is enforced by the query — scope to(clerk_id, session_id), the lesson from the stop route’s 1,378 ms unscoped lookup.- No client change yet. Verify the token verifies.
P2.3 — Client re-attaches on load (the user-visible one)
- On session load, running row + run id + token → ATTEMPT re-attach to
/api/claude/stream?runId=…&startIndex=0, rendering into an empty bubble. The poller is skipped only once the handshake has actually succeeded: the GET runs before any hook state is written, so agonerun (retention expired, never dispatched to a workflow, reaped) leaves the historical view untouched and falls straight through to the poller. Never assume an address implies a readable stream — see decision 6. - Reuse
parseAnthropicStreamWithResumeunchanged — same budget, same frame accounting. - Also re-attach on
visibilitychange→ visible, not only on load. A backgrounded tab or phone kills the connection without a reload, so without this nothing reconnects until the user navigates (t3code’s"application-active"resubscribe, §4b). Same guard set as the load path: running row + run id, cooldown, andalready-streamingshort-circuit. - Fixes 3a. Fixes 3b as a side effect: returning to a chat whose stream was detached now re-attaches instead of polling.
- Canary: internal → 25% → 100%. Gate on
recoverable_disconnectfalling and no rise instream_error.
P2.4 — Multi-tab (gated on S2)
- If readers are independent: each tab re-attaches on mount; no coordination needed for the read path.
- Session-list staleness across tabs is a separate, smaller problem —
BroadcastChannelinvalidation of the sessions query. Scope it here only if S2 came back clean; otherwise it is its own ticket.
P2.5 — Delete the poller
- Remove the 1.5s
loadSessionself-reschedule and its state. - Only after P2.3 is at 100% for ≥1 week with the disconnect metric down.
- Sequencing note: this can only be complete once every turn has a run id,
i.e. after the inline path is gone (W5) and
envelope_invalidis fixed.
8. Verification
- Golden transcripts (existing harness,
lib/claude/__tests__/chat-turn-golden.test.ts): a replayed-from-0 stream must produce byte-identical rendered text to the live stream, includingneedsSpaceBetweenjoins and citationendOffsets. - The double-render test — running row with persisted partial text + replay from 0 must yield the text once. Written to fail against decision 5’s absence.
- Ownership — a resume token minted for user A must not verify for user B; a session-load for a foreign session must return no token (not a 403 that confirms existence).
- Preview E2E: send → reload mid-turn → stream resumes live and completes; send → switch chat → return → resumes; send → second tab → both stream. Plus the stranded-turn case, which is the one that silently regresses: force the re-attach to fail (point at an expired/unknown run id) on a row that DOES carry an address, and assert the turn still lands — the poller takes over and the answer appears. Neither-replay-nor-polling must be unreachable.
- Metric gate:
recoverable_disconnect/day materially down from the ~282 baseline;chat_response_completedclient/server gap narrowing.
9. Risks
10. Out of scope
- Cross-tab send coordination (two tabs sending to one session).
run_iduniqueness already prevents duplicate runs; a nicer UX is its own project. - Mid-stream halt (Stop layer 3). Still deliberately deferred; note that P2.1
makes it possible — with the run id stored and ownership-checked,
run.cancel()becomes safely callable, which is exactly what we removed from the stop route for lack of a trustworthy id. - Step-per-model-call, remaining ~15 QStash pairs, per-user rate limiting — carried over from the W-series Phase 2 list, unrelated to stream continuity.
11. Dependencies on open W-series work
envelope_invalidfallbacks (~6–59/day) must be fixed before P2.5: those turns have no workflow run, so they need the poller.- W5 great deletion removes the inline path, after which every turn has a run id. P2.5 lands after W5.
- Radar graduation (direct POST, QStash removal) is independent of this plan and can proceed in parallel.