> ## Documentation Index > Fetch the complete documentation index at: https://docs.cfosilvia.com/llms.txt > Use this file to discover all available pages before exploring further. # Vercel workflows rollout # Vercel Workflows Rollout — Chat + Radar (feature-flagged) **Status:** PR 0 in review ([PR #2663](https://github.com/CFO-Silvia/Silvia-Web/pull/2663)) · **Owner:** Felipe · **Last updated:** 2026-07-10 > **Reviewed plan supersedes the PR framing below.** The approved implementation > plan (post external review) introduced these amendments: a new **PR 0** (SDK > integration + control plane + input schemas + spikes); runtime kill switch via > **Redis override** (`workflow-dispatch:`, ≤5s cache — Vercel env-var > changes require a redeploy and are NOT a kill path); **explicit per-step > retries** (chat turn `maxRetries=0`; radar analysis `maxRetries=2` = parity > with today's QStash whole-job retries; `FatalError` for business outcomes); > **radar terminal-failure design** (workflow posts FAILED through the existing > idempotent completion path — no stranded PROCESSING rows); **no chat delta > coalescing** in the compatibility canary (chunk boundaries are semantic: > `lib/needs-space-between.ts` is shared by the client and the server > accumulator + citation offsets); and PR 7 graduation is a **processor code + > CDK change**, not an env flip (`RADAR_WORKER_URL` is embedded in the QStash > publish URL, handler.py:1905). ## PR 0 — spike findings (2026-07-09, workflow\@4.6.0 pinned) 1. **Composition works:** `withBotId(withSentryConfig(withWorkflow(nextConfig)))` in `next.config.mjs` (withWorkflow innermost). 2. **SDK internal routes:** `/.well-known/workflow/v1/*` (manifest also emitted under `public/.well-known/workflow/v1` at build). The Clerk middleware matcher already excludes any dotted path (`.*\..*$`, middleware.ts:391) → **no middleware change needed**. 3. **`start()` has NO idempotency key** (`StartOptions` = `world` / `specVersion` / `deploymentId` only) → the Redis `SETNX` dispatch claim in PR 6 is confirmed necessary. Default `deploymentId` pins the current deployment (skew protection). 4. **Run API:** `runId`, `readable`, `getReadable({ startIndex })`, `status`, `returnValue`, `cancel()`, `wakeUp()`. For PR 8: client disconnect must NOT `cancel()` the run (durability is the point) — the route just stops piping; verify billing behavior on Preview. 5. **Retries:** step default is `maxRetries = 3` — every step we ship sets it explicitly (`fn.maxRetries = N` on the step function). `FatalError` / `RetryableError({ retryAfter })`, `getWritable`, `sleep`, `defineHook` all import from `'workflow'`. 6. **No concurrency-cap option** found on the SDK surface → radar burst mitigation stays jittered `sleep()` + app-level semaphore; measure queue behavior in the Preview burst probe. 7. **Shipped in PR 0:** `lib/flags/workflow-dispatch.ts` (Redis override → **non-prod default: workflow always ON for dev/preview incl. staging-branch deploys** → PostHog → env default — the flag gates PRODUCTION only; fail-through (env-default resting state); bounded single-flight caches; 25 tests. Redis kill works in every environment as the QA escape hatch), `lib/claude/workflow-input.ts` (`ChatWorkflowInputV1`/`RadarWorkflowInputV1`, strict + JSON-structural + iterative depth guard (64) + 800 KB size guard, all falling back inline; 14 tests), `workflows/smoke.ts` + `/api/internal/workflow-smoke` (`GET` for signed-in internal users — browser triggerable on previews; `POST` Bearer INTERNAL\_API\_TOKEN for automation; `?stream=1` exercises the `getWritable()` → `run.readable` transport chat will use). 8. **Staging's unbounded `"undici": ">=7.28.0"` override breaks two things** (it resolves undici 8.7.0, a major jump): `@workflow/world-local` passes a v7 dispatcher → every local queue delivery failed (`UND_ERR_INVALID_ARG`, workflows enqueued but never executed in dev); and jsdom\@28/@29 deep-import `undici/lib/handler/wrap-handler.js` (removed in v8) → the vitest jsdom environment collapsed (171 errors, 4/175 files running) — **a live problem on staging HEAD**. Fixed with scoped overrides (`"@workflow/world-local": {"undici": "7.28.0"}`, `"jsdom": {"undici": "^7.28.0"}`), both satisfying the ≥7.28.0 security floor. 9. **Webpack integration landmine (fixed):** importing `workflow/api` from a route crashes at module scope under webpack dev/build — `@vercel/queue → @vercel/oidc → xdg-app-paths` throws `path.parse(undefined)` when bundled into the RSC vendor chunk. `withWorkflow()` does not externalize its own runtime chain. Fix: add `'@vercel/queue'` AND `'@vercel/oidc'` to `serverExternalPackages` (directive-free runtime deps; installing `workflow` dedupes @vercel/oidc to 3.8.0 repo-wide — also under the pre-existing @vercel/functions — so the oidc external covers every import path, incl. `/api/ai-import`'s chunk). 10. **Local smoke results (dev server, filesystem world):** `start()` returns `wrun_*` ids; steps execute as separate HTTP invocations (`POST /.well-known/workflow/v1/flow` / `/step` — all 200 through Clerk middleware, confirming finding 2 empirically); streamed step output arrives via `run.readable`. **Caveat — CONFIRMED IN BOTH WORLDS:** `run.readable` does NOT close on workflow completion (local filesystem world AND Vercel preview — run `wrun_01KXE84JKYR0GTREF6YJ68KXHR`, 2026-07-13: content delivered in \~6s, then `Vercel Runtime Timeout Error: Task timed out after 60 seconds` on the GET holding `run.readable` as its response body — i.e. an unclosed stream bills the consumer function's FULL maxDuration). PR 8 MUST (a) have the producer step explicitly `writer.close()` when the turn finishes (pattern now demonstrated in `workflows/smoke.ts`), and (b) keep the SSE `done` event as the client-side terminator (it already is). 11. **Workflow-bundle purity (found in the W1 local e2e):** the workflow FUNCTION's top-level import graph is compiled into the deterministic replay context, where Node.js modules are FORBIDDEN — a top-level `import { helper } from '@/lib/radar-worker-core'` pulls claude-service → bedrock-sdk/crypto/fs into the workflow bundle and the dev/build fails with 6 `workflow-node-module-error`s. **Pattern (demonstrated in `workflows/radar.ts`):** top-level imports are TYPE-ONLY; each `'use step'` body does `await import('@/lib/...')` — dynamic imports land in the step bundle (Node context) only. W3's chat workflow MUST follow the same shape (`lib/claude/chat-turn.ts` will transitively import the Anthropic SDK, Supabase, etc.). Also verified live: step `fn.maxRetries = N` is enforced exactly (deliverResult failed with `attempt: 4, retryCount: 3`), and a step exhausting retries fails the run with `errorCode: USER_ERROR` — for radar that's acceptable because the FAILED-callback delivery itself exhausting means AWS is unreachable, and the existing stale-PROCESSING re-dispatch (handler.py:1121/2082) is the recovery of record. Adopt [Vercel Workflows](https://vercel.com/docs/workflows) as the durability layer for our two AI surfaces — **chat** (`/api/claude`) and **radar** (`/api/radar/run`) — behind per-surface feature flags so each migration ships as a canary with an instant, deploy-free kill switch. *** ## 1. Goals & non-goals **Goals** * One shared Claude agentic-loop core used by every consumer (chat SSE, radar, simulation, proactive prompts). No more duplicated loops/retry/fallback logic. * Workflows adoption on radar and chat, each behind its own flag, each independently canaried. * Identical business logic on both sides of every flag fork — the flag only ever selects *transport/durability*, never analysis behavior. This is what makes the canary comparison honest. * Zero client changes for the chat canary: the workflow path must emit the byte-identical SSE protocol that `hooks/use-claude-chat.ts` already speaks. * Zero AWS changes for the radar canary: QStash keeps delivering to `/api/radar/run`; the fork lives inside that route. **Non-goals (now)** * Migrating chat to `@ai-sdk/workflow` / `WorkflowAgent`. We keep the raw Anthropic SDK (citations beta, compaction, adaptive thinking effort, fast mode) and stream our own bytes. * Step-per-model-call decomposition of the chat loop (Phase 2 — see §8). * Migrating the other \~15 QStash dispatcher/consumer pairs (follow-on program, same template). *** ## 2. Target architecture ``` ┌──────────────────────────────────────────┐ │ lib/claude/ (shared core — PRs 1–3) │ │ shared.ts | stream-engine.ts | │ │ agentic-loop.ts (runAgenticLoop) │ └───────▲──────────────────▲───────────────┘ │ │ INLINE (today) │ │ WORKFLOW (flagged) │ │ /api/claude ── SSE controller ──┘ └── workflows/chat.ts │ flag: chat-workflow-transport step: runChatTurn → getWritable() │ route returns run.readable (same SSE) │ /api/radar/run ── inline 800s run ─┘ └── workflows/radar.ts flag: radar-workflow-dispatch steps: gate → analyze → callbackToAws (QStash delivery unchanged) start(radarWorkflow) then ack QStash ``` Key invariant: `runAgenticLoop()` is **pure library code** — injected deps (Anthropic/Bedrock clients, Supabase, tool context) and a transport-agnostic callback emitter. The same function runs inside an SSE route handler and inside a `'use step'` body without modification. *** ## 3. Patterns & conventions (the quality bar) 1. **Transport-agnostic emitter.** The loop never touches a `ReadableStream` controller or a workflow writable directly. It calls `LoopCallbacks`; adapters own transports: ```ts theme={null} interface LoopCallbacks { onEvent?(name: SseEventName, data: object): void // full SSE vocabulary passthrough onText?(delta: string): void onPartial?(accumulated: string): void // persistence throttle onUsage?(usage: LoopUsage, loop: number): void onModelChange?(model: string, reason: FallbackReason): void } ``` 2. **SSE event vocabulary is a versioned contract.** Event names + payload shapes live in one exported const (`lib/claude/events.ts`). Both adapters serialize from it. Golden-transcript tests (§7) enforce it. 3. **Workflows in `workflows/.ts`, one per file.** Workflow functions contain *only* orchestration (call steps, branch on results). All I/O, randomness, `Date.now()`, env reads live in steps — the workflow body must be deterministic for replay. 4. **Steps are thin wrappers over `lib/` functions.** A `'use step'` function does: build deps → call library function → return serializable result. No business logic in the step body itself, so inline and workflow paths can never drift. 5. **Idempotency at every entry that can be delivered twice.** Redis `SET NX EX` claim keyed by the natural id (`radar:{radarId}`, `chat:{run_id}`) before `start()`. Settled by the PR-0 spike: `start()` has NO idempotency key (`StartOptions` = `world`/`specVersion`/ `deploymentId` only), so the manual claim is required, not provisional. 6. **Path tagging everywhere.** `dispatch_path: 'workflow' | 'inline'` on: structured logs, the PostHog `workflow_dispatch` event, radar's AWS callback payload, and `ai_chat_logs` metadata. Every canary readout is a group-by on this field. 7. **Flag failures fall through, and the chain's resting default is inline.** A PostHog evaluation error, timeout, or missing flag does NOT hard-select inline — it falls through to the env default (`WORKFLOW_DISPATCH_`), which is unset in prod and therefore resolves inline today. Setting that env var to a percentage changes what "PostHog unavailable" means — treat it as part of the dispatch contract, not an emergency lever (env changes require a redeploy; the Redis override is the emergency path). 8. **No silent behavior merges.** Where the route loop and claude-service loop differ today (retry taxonomy, `rate_limit_switch_to_bedrock`, thinking-effort clamps, citations wrapping), the difference is reconciled *explicitly* in PR review with a documented decision — the route's implementation is presumed canonical. *** ## 4. Flag design Module: `lib/flags/workflow-dispatch.ts` ```ts theme={null} type Surface = 'chat' | 'radar' async function getDispatchPath(surface: Surface, clerkId: string): Promise<'workflow' | 'inline'> ``` **Implemented in PR 0** (`getDispatchDecision` also returns the deciding `source`). Resolution order — first authoritative answer wins: 1. **Redis override** `workflow-dispatch:` = `off | 0..100` — the RUNTIME kill switch, works in every environment, cached ≤5s per instance. This is the only instant, deploy-free kill path (Vercel env-var changes require a redeploy). 2. **Non-prod default:** dev + preview deploys (incl. the staging branch) always take the workflow path — dogfood everywhere except prod. Detection: `VERCEL_ENV`, `NODE_ENV` fallback (mirrors `lib/landing/section-order-experiment.ts`). 3. **PostHog flags (PRODUCTION ONLY):** `radar-workflow-dispatch`, `chat-workflow-transport`. Percentage rollout, **deterministic on `clerkId`**; internal users targetable first. Cached 60s per (surface, clerkId), bounded (5k entries, sweep-then-evict-oldest), single-flight on concurrent misses. Definitive true/false is authoritative; missing flag/client/timeout falls through. 4. **Env default (production, deploy-time only):** `WORKFLOW_DISPATCH_RADAR|CHAT=off|0..100`. NOT a kill path. Unset/invalid → `'workflow'` since W5a (was `'inline'`). **Defaults (W5a, 2026-09-22):** every "couldn't decide" case now resolves to `'workflow'` — an unset/invalid env default, and an unexpected resolver error (`source: 'resolution-error'`). During the rollout these resolved to `'inline'`, the trusted path at the time; once workflow carried all traffic that became backwards, and a PostHog timeout with the env var unset put 3 chat + 1 radar turns on inline in the 30 days before the flip. Only an explicit instruction selects inline now: Redis `off`, PostHog `false`, or env `off`/percentage. The exception is a missing `clerkId`, which stays `'inline'` (`source: 'fail-closed'`) because every layer and the workflow envelope are keyed on it. To get inline behavior on staging (e.g. transport-diff golden transcripts in PR 8): set the Redis key to `off`. **Start failures (W5b PR 1, CFO-9360, 2026-10-08):** a `start()` failure no longer falls back to inline on either surface. Over the 60 days before, `start()` failed on \~1 in 15,000 chat turns (13 of 197,072), isolated, all `fetch failed`; and on 9 of 359,514 radar jobs, clustered at the scheduled batch minutes. * **Chat** retries `start()` once after 500 ms with the same envelope, so the same canonical `runId`. If the first request secretly reached Vercel, the second run collapses on the `(clerk_id, run_id)` unique index as `duplicate_run` before any model call. A second failure returns a retryable 503 the client shows as-is (`dispatch_path: 'failed'`, `fallback_outcome: 'request_failed'`). * **Radar** answers 503 for `start_failed` and `claim_unavailable`, so QStash redelivers with backoff (`dispatch_path: 'redelivery'`). The AWS stale-PROCESSING sweep is the backstop if QStash's retries run out. Inline is still reachable through the explicit kill switches, an invalid envelope, and radar's unconfirmed held claim, until PR 2 deletes the inline transport. *** ## 5. PR sequence **REORDERED 2026-07-14 (Felipe):** workflows land on both surfaces FIRST (radar → chat), flags control the rollout, and the refactors move to the END — where, with both surfaces at 100% on workflows, they largely become deletions of the inline paths. Rationale: get testable value early; avoid refactoring code we intend to delete. One structural exception: chat's loop lives inside its route handler and a workflow step cannot call into a route (Next route files may only export HTTP methods), so PR W3 includes the **minimal mechanical move** of the loop body into a callable function — a cut-and-paste with an injected emitter, NOT the full unification (that stays in the final refactor). | PR | Scope | Key contents | Acceptance | | - | - | - | - | | **0 ✅** | Workflow SDK integration + control plane + input schemas *(shipped — PR #2663)* | `workflow@4.6.0` pinned; `withWorkflow()` innermost in `next.config.mjs`; `serverExternalPackages += ['@vercel/queue','@vercel/oidc']`; `lib/flags/workflow-dispatch.ts` (§4); `lib/claude/workflow-input.ts` (strict Zod V1 schemas, depth+size guards with inline fallback); smoke workflow + `GET`(internal users)/`POST`(token) trigger route. Spike findings recorded above. | tsc/vitest/build green; live smoke incl. streaming; SDK routes pass middleware. | | **W1** | **Radar workflow** (prod stays flag-gated) | `workflows/radar.ts` (steps: `claimAndGate` → `runRadarAnalysis` → `postResultToAws`, thin wrappers over the EXISTING `generateRadarAnalysis`/`lib/claude-service.ts` — no refactor needed, radar's loop is already a callable library, and both fork sides sharing it keeps the canary honest; explicit per-step `maxRetries` — analysis 2, `FatalError` for token-limit/missing-user); terminal-failure design (FAILED callback + Slack, never a stranded PROCESSING row); flag fork inside `/api/radar/run` (workflow path → Redis `SETNX` dispatch claim → `parseRadarWorkflowInput` → `start()` → ack QStash; guard errors → inline; inline path untouched); path tagging. `start()` has no idempotency key (PR 0 finding) — the Redis claim is required. | Prod dark (PostHog unset). **Non-prod default = staging runs every radar through workflows the moment this merges** (the intent: immediate dogfood). Manual internal trigger verifies Dynamo result + notifications identical to inline; forced failure → FAILED callback + alert. | | **W2** | Radar canary → graduation | Stage rollout (§6). After 100% + 2-week bake: Lambda dispatcher calls new thin `/api/radar/start` directly (auth: internal token), inline path + QStash publish removed (code+CDK change, handler.py:1905). | Per-path metrics parity or better (§6). QStash radar topic idle before removal. | | **W3** | **Chat workflow transport** (prod stays flag-gated; **NO coalescing**) | (a) **Mechanical loop move**: route.ts loop body (\~2728–3453) + completion (\~3455–3620) cut into `lib/claude/chat-turn.ts` `runChatTurn(input, emit)` — injected emitter replaces closed-over `sendEvent`; route's inline path calls it wired to the SSE controller (behavior-neutral, proven by golden transcripts recorded BEFORE the move). NOT the full extraction: claude-service untouched, helpers stay duplicated until W5. (b) `workflows/chat.ts` (`chatWorkflow` step, `maxRetries = 0`): same `runChatTurn` wired to a `getWritable()` serializer — **byte-identical SSE frames, same chunk boundaries** (`needsSpaceBetween` + citation offsets are boundary-sensitive); writer **explicitly closed** at turn end (PR 0 finding 9 — unclosed streams bill full maxDuration). (c) Flag fork in `/api/claude` after auth/gate/routing/chat-log setup: workflow → `parseChatWorkflowInput` (frozen snapshot) → `start(chatWorkflow)` → `new Response(run.readable, SSE_HEADERS)`; guard errors → inline. Zero client changes. | Prod dark. Golden transcripts: pre-move inline == post-move inline == workflow path, event-for-event **including chunk boundaries**; TTFT delta measured on internal users; \$-per-turn measured WITHOUT coalescing. | | **W4** | Chat canary | Stage rollout (§6) with chat gates: TTFT p95, per-turn cost (workflow events + data written), stream-error rate, `done` completion rate, damper behavior. | Kill-switch drill at 5%: Redis `workflow-dispatch:chat=off` → inline recovery ≤5s. | | **W5** | **The great deletion + unification** *(after BOTH surfaces at 100% + bake)* | Delete: chat's inline branch in `/api/claude` (route becomes start-workflow-and-stream); radar's inline branch (done in W2); orphaned `/api/chat`, `/api/chat/internal`, `/api/claude/internal` (+ dead Lambda sync code, `INTERNAL_API_URL`/`FALLBACK_API_URL` in `aws/radar/stack.py`, `examples/test_radar.py`, middleware carve-outs, BotId entry) after zero-traffic log check. Unify (now optional-scope, judged then): claude-service onto `runChatTurn`'s engine or vice versa; merge duplicated helpers (`isRetryableError` union w/ CFO-5042 checks, Bedrock translation — chat finally inherits the stream-termination retries); GenUI `ui-*` validation/repair/markdown-fallback at the loop choke point. | Zero-hit logs on deleted paths (7 days). Radar simulation + proactive-prompts smoke unchanged if unification lands. `generative_ui_schema_validation_failed` rate drops. | | **W6** | Stream-cost optimization *(only if W4's \$-gate demands it)* | Delta coalescing behind its own flag; acceptance = **semantic equivalence** (rendered text incl. `needsSpaceBetween` joins, citation `endOffset`s, terminal-error events). | Semantic-equivalence suite green; \$-per-turn improvement quantified. | **Known cost of the reorder (accepted):** the loop-drift bugs stay live until W5 — chat keeps missing the CFO-5042 ECONNRESET/`UND_ERR_*` retries, claude-service keeps its weaker Bedrock translation. Neither worsens: both fork sides of each surface share that surface's existing code, so canary comparisons remain apples-to-apples. **Phase 2 (post-rollout, separately flagged):** see §8. *** ## 6. Canary protocol (both surfaces) Stages: **internal users → 5% → 25% → 100%**, minimum 3 days per stage (radar: include at least one 13:00 UTC burst per stage). Baseline first (Clerk-v7 ritual): capture pre-flag metrics for one week before enabling anything. **Shared gates (per `dispatch_path`):** * Completion rate (`COMPLETED`/`done`) — no regression > 0.5pp * Error rate + retry counts — no regression * End-to-end duration p50/p95 * Token usage + cache-read rate parity (loop logic is shared, so drift here = bug) **Radar-specific:** callback delivery success; DynamoDB result-shape diff on sampled pairs; burst behavior at 13:00 UTC (queue absorption vs today's stampede). **Chat-specific:** TTFT p50/p95 (the decision metric); recoverable-disconnect rate; \$-per-turn including Workflows billing (events + data written — measured WITHOUT coalescing; the result decides whether PR 10 happens); no `use-claude-chat` client errors in Sentry attributable to the path. **Kill switch:** Redis `workflow-dispatch:=off` (≤5s propagation, works in every environment) or PostHog flag → 0% (production). Env vars are deploy-time defaults, NOT a kill path — Vercel env changes require a redeploy. Rollback ritual: flip the Redis key, confirm inline traffic recovers, then diagnose offline. ### Telemetry contract (implemented in lib/radar-run-telemetry.ts, 2026-07-14) Lifecycle events (PostHog, both transports, canonical context = radar\_id / schedule\_id / execution\_id / clerk distinct id / dispatch\_path / dispatch\_source / workflow\_run\_id): `workflow_dispatch` (every decision) · `workflow_fallback` (workflow-intended dispatch degraded, with `fallback_reason`) · `workflow_started` (entry step, attempt 1, with queue\_latency\_ms) · `workflow_step_failed` (every failed attempt: step\_name / attempt / max\_retries / terminal / duration\_ms / error\_class) · `workflow_callback_failed` (delivery attempts, with callback HTTP status) · `workflow_completed` (terminal, BOTH paths: status / duration\_ms / model / token+cache usage / error\_class). Sentry receives ACTIONABLE failures only, with searchable tags (surface, dispatch\_path, radar\_id, workflow\_run\_id, error\_class, stage/step\_name): terminal step exhaustion, permanent callback 4xx, workflow-start failures, claim-cleanup failures. Sentry `environment` now reports VERCEL\_ENV (preview/staging separable from prod). ### Alerts to create before the staging bake ends (W2 entry gate) | Signal | Rule | Why | | - | - | - | | Sentry tag `stage:claim_cleanup` | any event | Leftover pending claims degrade redeliveries to inline until TTL | | Sentry tag `error_class:permanent_callback_4xx` | any event | Auth/payload drift to the AWS gateway — every delivery is failing | | Sentry tag `stage:workflow_start` | any event in prod; spike in staging | Workflow transport down while flag routes traffic to it | | PostHog `workflow_fallback` | rate > \~1% of `workflow_dispatch` (workflow-intended) | Dispatch layer silently degrading | | PostHog `workflow_callback_failed` where `terminal=true` | > 0/day | Results not reaching DynamoDB | | PostHog `workflow_completed` FAILED-rate split by `dispatch_path` | workflow − inline > 2pp over 24h | The canary regression gate itself | | PostHog `workflow_dispatch` (workflow, non-dedupe) without matching `workflow_completed` within 1h | > 0 | Stranded runs (complements AWS stale-PROCESSING sweep) | *** ## 7. Verification harness * **Golden SSE transcripts** (built in PR 2, reused forever): fixed prompt set — plain answer, single tool, multi-tool loop, forced 529 fallback, compaction-length conversation, GenUI component answer — recorded as ordered event-name+shape sequences. Diffed on every PR touching the loop, and across inline vs workflow paths in PR 8. * **Radar parity run:** silvia-simulation against a fixture portfolio; structured output (`sendAlert`, `triggerMet`, `confidence`, summary shape) diffed before/after PRs 3, 4, 6. * **Load probe (pre-PR-7 graduation):** synthetic 200-job radar burst against the workflow path in preview, watching queue latency + Anthropic 429 behavior. *** ## 8. Phase 2 (after both canaries graduate) > **2026-07-21: chat's Phase-2 slice PULLED FORWARD as W4.5** — the "only if > 800s truncations are observed" condition fired on internal-canary turn #2 > (video build, step 504 at 800s). Chat turns now slice at loop boundaries > past 180s-per-invocation, with client re-attach for streams that outlive > the /api/claude pump. See docs/plans/w3-chat-extraction-map.md → "W4.5 > ADDENDUM". Radar analyze (single-step, minutes-scale) stays as-is. * **2a — Resumable chat streams:** client reconnect via `run.getReadable({ startIndex })`; retire the hand-rolled recovery (2s partial persistence + session rehydration). Own flag. * **2b — Step-per-model-call chat loop:** decompose so deep-research turns escape any single invocation's `maxDuration`. Only if 800s truncations are observed in practice. * **2c — QStash pair migration program:** template from PR 6 applied to document queue first, then burst-prone schedulers. * **2d — Rate limiting (from the audit; independent of workflows):** per-user request + concurrency caps on `/api/claude`; concurrency semaphore on radar analysis if the PR-6 spike finds no native per-workflow concurrency control. *** ## 9. Risks & open questions | Risk | Mitigation | | - | - | | Workflows step invocations still bounded by `maxDuration` — phase 1 doesn't lift the 800s ceiling | Accepted for parity; 2b is the fix if needed. Track `max_tokens_cutoff`/timeout rates per path. | | Chat TTFT regression from `start()` + queue hop + Redis-persisted chunks | Measured at internal-user stage before any % rollout; fallback position = workflows for async surfaces only, chat stays inline. | | Workflow stream billing on chat token volume | Measured in PR 9 (no coalescing in the compatibility canary); PR 10 owns coalescing under semantic-equivalence acceptance. | | ~~`start()` idempotency semantics unknown~~ **Answered (PR 0):** no idempotency key on `StartOptions` | Redis `SETNX` dispatch claim in PR 6 is required, not provisional. | | No native per-workflow concurrency cap → radar burst still hammers Anthropic | Jittered `sleep()` first step + 2d semaphore. | | Double-execution during canary (QStash redelivery after `start()` succeeded) | Redis `SET NX` claim before `start()`; claim TTL > max radar runtime. | | Route/claude-service fallback behavior reconciliation changes radar subtly | PR 2 requires an explicit documented decision per delta; radar parity run in §7. | | Skew protection pins runs to old deployments — a bad deploy's in-flight runs keep its bugs | Acceptable (matches QStash semantics today); note in ops runbook. | *** ## 10. File map (end state) ``` lib/claude/ shared.ts # PR 1 — cache helpers, error classification, Bedrock translation stream-engine.ts # PR 2 — stream create/consume + retry/fallback chain agentic-loop.ts # PR 3 — runAgenticLoop + LoopCallbacks events.ts # PR 3 — SSE event vocabulary (versioned contract) lib/flags/ workflow-dispatch.ts # PR 6 — flag resolution (PostHog + env override + cache, fail-through (env-default resting state)) workflows/ radar.ts # PR 6 — radarWorkflow + steps chat.ts # PR 8 — chatWorkflow + runChatTurn step app/api/claude/route.ts # thin adapter: auth, gate, routing, persistence, fork, SSE app/api/radar/run/route.ts # QStash consumer: verify, fork (workflow start | inline run) app/api/radar/start/route.ts # PR 7 — direct dispatch entry (post-graduation) lib/claude-service.ts # thin adapter over runAgenticLoop (radar/simulation/proactive) ``` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.