Vercel Workflows Rollout — Chat + Radar (feature-flagged)
Status: PR 0 in review (PR #2663) · Owner: Felipe · Last updated: 2026-07-10Reviewed plan supersedes the PR framing below. The approved implementation plan (post external review) introduced these amendments: a new PR 0 (SDK integration + control plane + input schemas + spikes); runtime kill switch via Redis override (workflow-dispatch:<surface>, ≤5s cache — Vercel env-var changes require a redeploy and are NOT a kill path); explicit per-step retries (chat turnmaxRetries=0; radar analysismaxRetries=2= parity with today’s QStash whole-job retries;FatalErrorfor business outcomes); radar terminal-failure design (workflow posts FAILED through the existing idempotent completion path — no stranded PROCESSING rows); no chat delta coalescing in the compatibility canary (chunk boundaries are semantic:lib/needs-space-between.tsis shared by the client and the server accumulator + citation offsets); and PR 7 graduation is a processor code + CDK change, not an env flip (RADAR_WORKER_URLis embedded in the QStash publish URL, handler.py:1905).
PR 0 — spike findings (2026-07-09, workflow@4.6.0 pinned)
- Composition works:
withBotId(withSentryConfig(withWorkflow(nextConfig)))innext.config.mjs(withWorkflow innermost). - SDK internal routes:
/.well-known/workflow/v1/*(manifest also emitted underpublic/.well-known/workflow/v1at build). The Clerk middleware matcher already excludes any dotted path (.*\..*$, middleware.ts:391) → no middleware change needed. start()has NO idempotency key (StartOptions=world/specVersion/deploymentIdonly) → the RedisSETNXdispatch claim in PR 6 is confirmed necessary. DefaultdeploymentIdpins the current deployment (skew protection).- Run API:
runId,readable,getReadable({ startIndex }),status,returnValue,cancel(),wakeUp(). For PR 8: client disconnect must NOTcancel()the run (durability is the point) — the route just stops piping; verify billing behavior on Preview. - Retries: step default is
maxRetries = 3— every step we ship sets it explicitly (fn.maxRetries = Non the step function).FatalError/RetryableError({ retryAfter }),getWritable,sleep,defineHookall import from'workflow'. - No concurrency-cap option found on the SDK surface → radar burst
mitigation stays jittered
sleep()+ app-level semaphore; measure queue behavior in the Preview burst probe. - Shipped in PR 0:
lib/flags/workflow-dispatch.ts(Redis override → non-prod default: workflow always ON for dev/preview incl. staging-branch deploys → PostHog → env default — the flag gates PRODUCTION only; fail-through (env-default resting state); bounded single-flight caches; 25 tests. Redis kill works in every environment as the QA escape hatch),lib/claude/workflow-input.ts(ChatWorkflowInputV1/RadarWorkflowInputV1, strict + JSON-structural + iterative depth guard (64) + 800 KB size guard, all falling back inline; 14 tests),workflows/smoke.ts+/api/internal/workflow-smoke(GETfor signed-in internal users — browser triggerable on previews;POSTBearer INTERNAL_API_TOKEN for automation;?stream=1exercises thegetWritable()→run.readabletransport chat will use). - Staging’s unbounded
"undici": ">=7.28.0"override breaks two things (it resolves undici 8.7.0, a major jump):@workflow/world-localpasses a v7 dispatcher → every local queue delivery failed (UND_ERR_INVALID_ARG, workflows enqueued but never executed in dev); and jsdom@28/@29 deep-importundici/lib/handler/wrap-handler.js(removed in v8) → the vitest jsdom environment collapsed (171 errors, 4/175 files running) — a live problem on staging HEAD. Fixed with scoped overrides ("@workflow/world-local": {"undici": "7.28.0"},"jsdom": {"undici": "^7.28.0"}), both satisfying the ≥7.28.0 security floor. - Webpack integration landmine (fixed): importing
workflow/apifrom a route crashes at module scope under webpack dev/build —@vercel/queue → @vercel/oidc → xdg-app-pathsthrowspath.parse(undefined)when bundled into the RSC vendor chunk.withWorkflow()does not externalize its own runtime chain. Fix: add'@vercel/queue'AND'@vercel/oidc'toserverExternalPackages(directive-free runtime deps; installingworkflowdedupes @vercel/oidc to 3.8.0 repo-wide — also under the pre-existing @vercel/functions — so the oidc external covers every import path, incl./api/ai-import’s chunk). - Local smoke results (dev server, filesystem world):
start()returnswrun_*ids; steps execute as separate HTTP invocations (POST /.well-known/workflow/v1/flow//step— all 200 through Clerk middleware, confirming finding 2 empirically); streamed step output arrives viarun.readable. Caveat — CONFIRMED IN BOTH WORLDS:run.readabledoes NOT close on workflow completion (local filesystem world AND Vercel preview — runwrun_01KXE84JKYR0GTREF6YJ68KXHR, 2026-07-13: content delivered in ~6s, thenVercel Runtime Timeout Error: Task timed out after 60 secondson the GET holdingrun.readableas its response body — i.e. an unclosed stream bills the consumer function’s FULL maxDuration). PR 8 MUST (a) have the producer step explicitlywriter.close()when the turn finishes (pattern now demonstrated inworkflows/smoke.ts), and (b) keep the SSEdoneevent as the client-side terminator (it already is). - Workflow-bundle purity (found in the W1 local e2e): the workflow
FUNCTION’s top-level import graph is compiled into the deterministic
replay context, where Node.js modules are FORBIDDEN — a top-level
import { helper } from '@/lib/radar-worker-core'pulls claude-service → bedrock-sdk/crypto/fs into the workflow bundle and the dev/build fails with 6workflow-node-module-errors. Pattern (demonstrated inworkflows/radar.ts): top-level imports are TYPE-ONLY; each'use step'body doesawait import('@/lib/...')— dynamic imports land in the step bundle (Node context) only. W3’s chat workflow MUST follow the same shape (lib/claude/chat-turn.tswill transitively import the Anthropic SDK, Supabase, etc.). Also verified live: stepfn.maxRetries = Nis enforced exactly (deliverResult failed withattempt: 4, retryCount: 3), and a step exhausting retries fails the run witherrorCode: USER_ERROR— for radar that’s acceptable because the FAILED-callback delivery itself exhausting means AWS is unreachable, and the existing stale-PROCESSING re-dispatch (handler.py:1121/2082) is the recovery of record.
/api/claude) and radar (/api/radar/run) — behind per-surface
feature flags so each migration ships as a canary with an instant, deploy-free kill switch.
1. Goals & non-goals
Goals- One shared Claude agentic-loop core used by every consumer (chat SSE, radar, simulation, proactive prompts). No more duplicated loops/retry/fallback logic.
- Workflows adoption on radar and chat, each behind its own flag, each independently canaried.
- Identical business logic on both sides of every flag fork — the flag only ever selects transport/durability, never analysis behavior. This is what makes the canary comparison honest.
- Zero client changes for the chat canary: the workflow path must emit the byte-identical SSE
protocol that
hooks/use-claude-chat.tsalready speaks. - Zero AWS changes for the radar canary: QStash keeps delivering to
/api/radar/run; the fork lives inside that route.
- Migrating chat to
@ai-sdk/workflow/WorkflowAgent. We keep the raw Anthropic SDK (citations beta, compaction, adaptive thinking effort, fast mode) and stream our own bytes. - Step-per-model-call decomposition of the chat loop (Phase 2 — see §8).
- Migrating the other ~15 QStash dispatcher/consumer pairs (follow-on program, same template).
2. Target architecture
runAgenticLoop() is pure library code — injected deps (Anthropic/Bedrock
clients, Supabase, tool context) and a transport-agnostic callback emitter. The same function runs
inside an SSE route handler and inside a 'use step' body without modification.
3. Patterns & conventions (the quality bar)
-
Transport-agnostic emitter. The loop never touches a
ReadableStreamcontroller or a workflow writable directly. It callsLoopCallbacks; adapters own transports: -
SSE event vocabulary is a versioned contract. Event names + payload shapes live in one
exported const (
lib/claude/events.ts). Both adapters serialize from it. Golden-transcript tests (§7) enforce it. -
Workflows in
workflows/<surface>.ts, one per file. Workflow functions contain only orchestration (call steps, branch on results). All I/O, randomness,Date.now(), env reads live in steps — the workflow body must be deterministic for replay. -
Steps are thin wrappers over
lib/functions. A'use step'function does: build deps → call library function → return serializable result. No business logic in the step body itself, so inline and workflow paths can never drift. -
Idempotency at every entry that can be delivered twice. Redis
SET NX EXclaim keyed by the natural id (radar:{radarId},chat:{run_id}) beforestart(). Settled by the PR-0 spike:start()has NO idempotency key (StartOptions=world/specVersion/deploymentIdonly), so the manual claim is required, not provisional. -
Path tagging everywhere.
dispatch_path: 'workflow' | 'inline'on: structured logs, the PostHogworkflow_dispatchevent, radar’s AWS callback payload, andai_chat_logsmetadata. Every canary readout is a group-by on this field. -
Flag failures fall through, and the chain’s resting default is inline. A PostHog
evaluation error, timeout, or missing flag does NOT hard-select inline — it falls through to
the env default (
WORKFLOW_DISPATCH_<SURFACE>), which is unset in prod and therefore resolves inline today. Setting that env var to a percentage changes what “PostHog unavailable” means — treat it as part of the dispatch contract, not an emergency lever (env changes require a redeploy; the Redis override is the emergency path). -
No silent behavior merges. Where the route loop and claude-service loop differ today
(retry taxonomy,
rate_limit_switch_to_bedrock, thinking-effort clamps, citations wrapping), the difference is reconciled explicitly in PR review with a documented decision — the route’s implementation is presumed canonical.
4. Flag design
Module:lib/flags/workflow-dispatch.ts
getDispatchDecision also returns the deciding source). Resolution
order — first authoritative answer wins:
- Redis override
workflow-dispatch:<surface>=off | 0..100— the RUNTIME kill switch, works in every environment, cached ≤5s per instance. This is the only instant, deploy-free kill path (Vercel env-var changes require a redeploy). - Non-prod default: dev + preview deploys (incl. the staging branch) always take the
workflow path — dogfood everywhere except prod. Detection:
VERCEL_ENV,NODE_ENVfallback (mirrorslib/landing/section-order-experiment.ts). - PostHog flags (PRODUCTION ONLY):
radar-workflow-dispatch,chat-workflow-transport. Percentage rollout, deterministic onclerkId; internal users targetable first. Cached 60s per (surface, clerkId), bounded (5k entries, sweep-then-evict-oldest), single-flight on concurrent misses. Definitive true/false is authoritative; missing flag/client/timeout falls through. - Env default (production, deploy-time only):
WORKFLOW_DISPATCH_RADAR|CHAT=off|0..100. NOT a kill path. Unset/invalid →'workflow'since W5a (was'inline').
'workflow' — an
unset/invalid env default, and an unexpected resolver error (source: 'resolution-error').
During the rollout these resolved to 'inline', the trusted path at the time; once workflow
carried all traffic that became backwards, and a PostHog timeout with the env var unset put
3 chat + 1 radar turns on inline in the 30 days before the flip. Only an explicit instruction
selects inline now: Redis off, PostHog false, or env off/percentage. The exception is a
missing clerkId, which stays 'inline' (source: 'fail-closed') because every layer and the
workflow envelope are keyed on it. To get inline behavior on staging (e.g. transport-diff
golden transcripts in PR 8): set the Redis key to off.
Start failures (W5b PR 1, CFO-9360, 2026-10-08): a start() failure no longer falls back to
inline on either surface. Over the 60 days before, start() failed on ~1 in 15,000 chat turns
(13 of 197,072), isolated, all fetch failed; and on 9 of 359,514 radar jobs, clustered at the
scheduled batch minutes.
- Chat retries
start()once after 500 ms with the same envelope, so the same canonicalrunId. If the first request secretly reached Vercel, the second run collapses on the(clerk_id, run_id)unique index asduplicate_runbefore any model call. A second failure returns a retryable 503 the client shows as-is (dispatch_path: 'failed',fallback_outcome: 'request_failed'). - Radar answers 503 for
start_failedandclaim_unavailable, so QStash redelivers with backoff (dispatch_path: 'redelivery'). The AWS stale-PROCESSING sweep is the backstop if QStash’s retries run out.
5. PR sequence
REORDERED 2026-07-14 (Felipe): workflows land on both surfaces FIRST (radar → chat), flags control the rollout, and the refactors move to the END — where, with both surfaces at 100% on workflows, they largely become deletions of the inline paths. Rationale: get testable value early; avoid refactoring code we intend to delete. One structural exception: chat’s loop lives inside its route handler and a workflow step cannot call into a route (Next route files may only export HTTP methods), so PR W3 includes the minimal mechanical move of the loop body into a callable function — a cut-and-paste with an injected emitter, NOT the full unification (that stays in the final refactor).
Known cost of the reorder (accepted): the loop-drift bugs stay live until W5 — chat keeps
missing the CFO-5042 ECONNRESET/
UND_ERR_* retries, claude-service keeps its weaker Bedrock
translation. Neither worsens: both fork sides of each surface share that surface’s existing
code, so canary comparisons remain apples-to-apples.
Phase 2 (post-rollout, separately flagged): see §8.
6. Canary protocol (both surfaces)
Stages: internal users → 5% → 25% → 100%, minimum 3 days per stage (radar: include at least one 13:00 UTC burst per stage). Baseline first (Clerk-v7 ritual): capture pre-flag metrics for one week before enabling anything. Shared gates (perdispatch_path):
- Completion rate (
COMPLETED/done) — no regression > 0.5pp - Error rate + retry counts — no regression
- End-to-end duration p50/p95
- Token usage + cache-read rate parity (loop logic is shared, so drift here = bug)
use-claude-chat client errors in Sentry
attributable to the path.
Kill switch: Redis workflow-dispatch:<surface>=off (≤5s propagation, works in every
environment) or PostHog flag → 0% (production). Env vars are deploy-time defaults, NOT a kill
path — Vercel env changes require a redeploy.
Rollback ritual: flip the Redis key, confirm inline traffic recovers, then diagnose offline.
Telemetry contract (implemented in lib/radar-run-telemetry.ts, 2026-07-14)
Lifecycle events (PostHog, both transports, canonical context = radar_id / schedule_id / execution_id / clerk distinct id / dispatch_path / dispatch_source / workflow_run_id):workflow_dispatch (every decision) · workflow_fallback (workflow-intended dispatch degraded,
with fallback_reason) · workflow_started (entry step, attempt 1, with queue_latency_ms) ·
workflow_step_failed (every failed attempt: step_name / attempt / max_retries / terminal /
duration_ms / error_class) · workflow_callback_failed (delivery attempts, with callback HTTP
status) · workflow_completed (terminal, BOTH paths: status / duration_ms / model /
token+cache usage / error_class). Sentry receives ACTIONABLE failures only, with searchable
tags (surface, dispatch_path, radar_id, workflow_run_id, error_class, stage/step_name):
terminal step exhaustion, permanent callback 4xx, workflow-start failures, claim-cleanup
failures. Sentry environment now reports VERCEL_ENV (preview/staging separable from prod).
Alerts to create before the staging bake ends (W2 entry gate)
7. Verification harness
- Golden SSE transcripts (built in PR 2, reused forever): fixed prompt set — plain answer, single tool, multi-tool loop, forced 529 fallback, compaction-length conversation, GenUI component answer — recorded as ordered event-name+shape sequences. Diffed on every PR touching the loop, and across inline vs workflow paths in PR 8.
- Radar parity run: silvia-simulation against a fixture portfolio; structured output
(
sendAlert,triggerMet,confidence, summary shape) diffed before/after PRs 3, 4, 6. - Load probe (pre-PR-7 graduation): synthetic 200-job radar burst against the workflow path in preview, watching queue latency + Anthropic 429 behavior.
8. Phase 2 (after both canaries graduate)
2026-07-21: chat’s Phase-2 slice PULLED FORWARD as W4.5 — the “only if 800s truncations are observed” condition fired on internal-canary turn #2 (video build, step 504 at 800s). Chat turns now slice at loop boundaries past 180s-per-invocation, with client re-attach for streams that outlive the /api/claude pump. See docs/plans/w3-chat-extraction-map.md → “W4.5 ADDENDUM”. Radar analyze (single-step, minutes-scale) stays as-is.
- 2a — Resumable chat streams: client reconnect via
run.getReadable({ startIndex }); retire the hand-rolled recovery (2s partial persistence + session rehydration). Own flag. - 2b — Step-per-model-call chat loop: decompose so deep-research turns escape any single
invocation’s
maxDuration. Only if 800s truncations are observed in practice. - 2c — QStash pair migration program: template from PR 6 applied to document queue first, then burst-prone schedulers.
- 2d — Rate limiting (from the audit; independent of workflows): per-user request +
concurrency caps on
/api/claude; concurrency semaphore on radar analysis if the PR-6 spike finds no native per-workflow concurrency control.