Skip to main content

Vercel Workflows Rollout — Chat + Radar (feature-flagged)

Status: PR 0 in review (PR #2663) · Owner: Felipe · Last updated: 2026-07-10
Reviewed plan supersedes the PR framing below. The approved implementation plan (post external review) introduced these amendments: a new PR 0 (SDK integration + control plane + input schemas + spikes); runtime kill switch via Redis override (workflow-dispatch:<surface>, ≤5s cache — Vercel env-var changes require a redeploy and are NOT a kill path); explicit per-step retries (chat turn maxRetries=0; radar analysis maxRetries=2 = parity with today’s QStash whole-job retries; FatalError for business outcomes); radar terminal-failure design (workflow posts FAILED through the existing idempotent completion path — no stranded PROCESSING rows); no chat delta coalescing in the compatibility canary (chunk boundaries are semantic: lib/needs-space-between.ts is shared by the client and the server accumulator + citation offsets); and PR 7 graduation is a processor code + CDK change, not an env flip (RADAR_WORKER_URL is embedded in the QStash publish URL, handler.py:1905).

PR 0 — spike findings (2026-07-09, workflow@4.6.0 pinned)

  1. Composition works: withBotId(withSentryConfig(withWorkflow(nextConfig))) in next.config.mjs (withWorkflow innermost).
  2. SDK internal routes: /.well-known/workflow/v1/* (manifest also emitted under public/.well-known/workflow/v1 at build). The Clerk middleware matcher already excludes any dotted path (.*\..*$, middleware.ts:391) → no middleware change needed.
  3. start() has NO idempotency key (StartOptions = world / specVersion / deploymentId only) → the Redis SETNX dispatch claim in PR 6 is confirmed necessary. Default deploymentId pins the current deployment (skew protection).
  4. Run API: runId, readable, getReadable({ startIndex }), status, returnValue, cancel(), wakeUp(). For PR 8: client disconnect must NOT cancel() the run (durability is the point) — the route just stops piping; verify billing behavior on Preview.
  5. Retries: step default is maxRetries = 3 — every step we ship sets it explicitly (fn.maxRetries = N on the step function). FatalError / RetryableError({ retryAfter }), getWritable, sleep, defineHook all import from 'workflow'.
  6. No concurrency-cap option found on the SDK surface → radar burst mitigation stays jittered sleep() + app-level semaphore; measure queue behavior in the Preview burst probe.
  7. Shipped in PR 0: lib/flags/workflow-dispatch.ts (Redis override → non-prod default: workflow always ON for dev/preview incl. staging-branch deploys → PostHog → env default — the flag gates PRODUCTION only; fail-through (env-default resting state); bounded single-flight caches; 25 tests. Redis kill works in every environment as the QA escape hatch), lib/claude/workflow-input.ts (ChatWorkflowInputV1/RadarWorkflowInputV1, strict + JSON-structural + iterative depth guard (64) + 800 KB size guard, all falling back inline; 14 tests), workflows/smoke.ts + /api/internal/workflow-smoke (GET for signed-in internal users — browser triggerable on previews; POST Bearer INTERNAL_API_TOKEN for automation; ?stream=1 exercises the getWritable() → run.readable transport chat will use).
  8. Staging’s unbounded "undici": ">=7.28.0" override breaks two things (it resolves undici 8.7.0, a major jump): @workflow/world-local passes a v7 dispatcher → every local queue delivery failed (UND_ERR_INVALID_ARG, workflows enqueued but never executed in dev); and jsdom@28/@29 deep-import undici/lib/handler/wrap-handler.js (removed in v8) → the vitest jsdom environment collapsed (171 errors, 4/175 files running) — a live problem on staging HEAD. Fixed with scoped overrides ("@workflow/world-local": {"undici": "7.28.0"}, "jsdom": {"undici": "^7.28.0"}), both satisfying the ≥7.28.0 security floor.
  9. Webpack integration landmine (fixed): importing workflow/api from a route crashes at module scope under webpack dev/build — @vercel/queue → @vercel/oidc → xdg-app-paths throws path.parse(undefined) when bundled into the RSC vendor chunk. withWorkflow() does not externalize its own runtime chain. Fix: add '@vercel/queue' AND '@vercel/oidc' to serverExternalPackages (directive-free runtime deps; installing workflow dedupes @vercel/oidc to 3.8.0 repo-wide — also under the pre-existing @vercel/functions — so the oidc external covers every import path, incl. /api/ai-import’s chunk).
  10. Local smoke results (dev server, filesystem world): start() returns wrun_* ids; steps execute as separate HTTP invocations (POST /.well-known/workflow/v1/flow / /step — all 200 through Clerk middleware, confirming finding 2 empirically); streamed step output arrives via run.readable. Caveat — CONFIRMED IN BOTH WORLDS: run.readable does NOT close on workflow completion (local filesystem world AND Vercel preview — run wrun_01KXE84JKYR0GTREF6YJ68KXHR, 2026-07-13: content delivered in ~6s, then Vercel Runtime Timeout Error: Task timed out after 60 seconds on the GET holding run.readable as its response body — i.e. an unclosed stream bills the consumer function’s FULL maxDuration). PR 8 MUST (a) have the producer step explicitly writer.close() when the turn finishes (pattern now demonstrated in workflows/smoke.ts), and (b) keep the SSE done event as the client-side terminator (it already is).
  11. Workflow-bundle purity (found in the W1 local e2e): the workflow FUNCTION’s top-level import graph is compiled into the deterministic replay context, where Node.js modules are FORBIDDEN — a top-level import { helper } from '@/lib/radar-worker-core' pulls claude-service → bedrock-sdk/crypto/fs into the workflow bundle and the dev/build fails with 6 workflow-node-module-errors. Pattern (demonstrated in workflows/radar.ts): top-level imports are TYPE-ONLY; each 'use step' body does await import('@/lib/...') — dynamic imports land in the step bundle (Node context) only. W3’s chat workflow MUST follow the same shape (lib/claude/chat-turn.ts will transitively import the Anthropic SDK, Supabase, etc.). Also verified live: step fn.maxRetries = N is enforced exactly (deliverResult failed with attempt: 4, retryCount: 3), and a step exhausting retries fails the run with errorCode: USER_ERROR — for radar that’s acceptable because the FAILED-callback delivery itself exhausting means AWS is unreachable, and the existing stale-PROCESSING re-dispatch (handler.py:1121/2082) is the recovery of record.
Adopt Vercel Workflows as the durability layer for our two AI surfaces — chat (/api/claude) and radar (/api/radar/run) — behind per-surface feature flags so each migration ships as a canary with an instant, deploy-free kill switch.

1. Goals & non-goals

Goals
  • One shared Claude agentic-loop core used by every consumer (chat SSE, radar, simulation, proactive prompts). No more duplicated loops/retry/fallback logic.
  • Workflows adoption on radar and chat, each behind its own flag, each independently canaried.
  • Identical business logic on both sides of every flag fork — the flag only ever selects transport/durability, never analysis behavior. This is what makes the canary comparison honest.
  • Zero client changes for the chat canary: the workflow path must emit the byte-identical SSE protocol that hooks/use-claude-chat.ts already speaks.
  • Zero AWS changes for the radar canary: QStash keeps delivering to /api/radar/run; the fork lives inside that route.
Non-goals (now)
  • Migrating chat to @ai-sdk/workflow / WorkflowAgent. We keep the raw Anthropic SDK (citations beta, compaction, adaptive thinking effort, fast mode) and stream our own bytes.
  • Step-per-model-call decomposition of the chat loop (Phase 2 — see §8).
  • Migrating the other ~15 QStash dispatcher/consumer pairs (follow-on program, same template).

2. Target architecture

Key invariant: runAgenticLoop() is pure library code — injected deps (Anthropic/Bedrock clients, Supabase, tool context) and a transport-agnostic callback emitter. The same function runs inside an SSE route handler and inside a 'use step' body without modification.

3. Patterns & conventions (the quality bar)

  1. Transport-agnostic emitter. The loop never touches a ReadableStream controller or a workflow writable directly. It calls LoopCallbacks; adapters own transports:
  2. SSE event vocabulary is a versioned contract. Event names + payload shapes live in one exported const (lib/claude/events.ts). Both adapters serialize from it. Golden-transcript tests (§7) enforce it.
  3. Workflows in workflows/<surface>.ts, one per file. Workflow functions contain only orchestration (call steps, branch on results). All I/O, randomness, Date.now(), env reads live in steps — the workflow body must be deterministic for replay.
  4. Steps are thin wrappers over lib/ functions. A 'use step' function does: build deps → call library function → return serializable result. No business logic in the step body itself, so inline and workflow paths can never drift.
  5. Idempotency at every entry that can be delivered twice. Redis SET NX EX claim keyed by the natural id (radar:{radarId}, chat:{run_id}) before start(). Settled by the PR-0 spike: start() has NO idempotency key (StartOptions = world/specVersion/ deploymentId only), so the manual claim is required, not provisional.
  6. Path tagging everywhere. dispatch_path: 'workflow' | 'inline' on: structured logs, the PostHog workflow_dispatch event, radar’s AWS callback payload, and ai_chat_logs metadata. Every canary readout is a group-by on this field.
  7. Flag failures fall through, and the chain’s resting default is inline. A PostHog evaluation error, timeout, or missing flag does NOT hard-select inline — it falls through to the env default (WORKFLOW_DISPATCH_<SURFACE>), which is unset in prod and therefore resolves inline today. Setting that env var to a percentage changes what “PostHog unavailable” means — treat it as part of the dispatch contract, not an emergency lever (env changes require a redeploy; the Redis override is the emergency path).
  8. No silent behavior merges. Where the route loop and claude-service loop differ today (retry taxonomy, rate_limit_switch_to_bedrock, thinking-effort clamps, citations wrapping), the difference is reconciled explicitly in PR review with a documented decision — the route’s implementation is presumed canonical.

4. Flag design

Module: lib/flags/workflow-dispatch.ts
Implemented in PR 0 (getDispatchDecision also returns the deciding source). Resolution order — first authoritative answer wins:
  1. Redis override workflow-dispatch:<surface> = off | 0..100 — the RUNTIME kill switch, works in every environment, cached ≤5s per instance. This is the only instant, deploy-free kill path (Vercel env-var changes require a redeploy).
  2. Non-prod default: dev + preview deploys (incl. the staging branch) always take the workflow path — dogfood everywhere except prod. Detection: VERCEL_ENV, NODE_ENV fallback (mirrors lib/landing/section-order-experiment.ts).
  3. PostHog flags (PRODUCTION ONLY): radar-workflow-dispatch, chat-workflow-transport. Percentage rollout, deterministic on clerkId; internal users targetable first. Cached 60s per (surface, clerkId), bounded (5k entries, sweep-then-evict-oldest), single-flight on concurrent misses. Definitive true/false is authoritative; missing flag/client/timeout falls through.
  4. Env default (production, deploy-time only): WORKFLOW_DISPATCH_RADAR|CHAT=off|0..100. NOT a kill path. Unset/invalid → 'workflow' since W5a (was 'inline').
Defaults (W5a, 2026-09-22): every “couldn’t decide” case now resolves to 'workflow' — an unset/invalid env default, and an unexpected resolver error (source: 'resolution-error'). During the rollout these resolved to 'inline', the trusted path at the time; once workflow carried all traffic that became backwards, and a PostHog timeout with the env var unset put 3 chat + 1 radar turns on inline in the 30 days before the flip. Only an explicit instruction selects inline now: Redis off, PostHog false, or env off/percentage. The exception is a missing clerkId, which stays 'inline' (source: 'fail-closed') because every layer and the workflow envelope are keyed on it. To get inline behavior on staging (e.g. transport-diff golden transcripts in PR 8): set the Redis key to off. Start failures (W5b PR 1, CFO-9360, 2026-10-08): a start() failure no longer falls back to inline on either surface. Over the 60 days before, start() failed on ~1 in 15,000 chat turns (13 of 197,072), isolated, all fetch failed; and on 9 of 359,514 radar jobs, clustered at the scheduled batch minutes.
  • Chat retries start() once after 500 ms with the same envelope, so the same canonical runId. If the first request secretly reached Vercel, the second run collapses on the (clerk_id, run_id) unique index as duplicate_run before any model call. A second failure returns a retryable 503 the client shows as-is (dispatch_path: 'failed', fallback_outcome: 'request_failed').
  • Radar answers 503 for start_failed and claim_unavailable, so QStash redelivers with backoff (dispatch_path: 'redelivery'). The AWS stale-PROCESSING sweep is the backstop if QStash’s retries run out.
Inline is still reachable through the explicit kill switches, an invalid envelope, and radar’s unconfirmed held claim, until PR 2 deletes the inline transport.

5. PR sequence

REORDERED 2026-07-14 (Felipe): workflows land on both surfaces FIRST (radar → chat), flags control the rollout, and the refactors move to the END — where, with both surfaces at 100% on workflows, they largely become deletions of the inline paths. Rationale: get testable value early; avoid refactoring code we intend to delete. One structural exception: chat’s loop lives inside its route handler and a workflow step cannot call into a route (Next route files may only export HTTP methods), so PR W3 includes the minimal mechanical move of the loop body into a callable function — a cut-and-paste with an injected emitter, NOT the full unification (that stays in the final refactor). Known cost of the reorder (accepted): the loop-drift bugs stay live until W5 — chat keeps missing the CFO-5042 ECONNRESET/UND_ERR_* retries, claude-service keeps its weaker Bedrock translation. Neither worsens: both fork sides of each surface share that surface’s existing code, so canary comparisons remain apples-to-apples. Phase 2 (post-rollout, separately flagged): see §8.

6. Canary protocol (both surfaces)

Stages: internal users → 5% → 25% → 100%, minimum 3 days per stage (radar: include at least one 13:00 UTC burst per stage). Baseline first (Clerk-v7 ritual): capture pre-flag metrics for one week before enabling anything. Shared gates (per dispatch_path):
  • Completion rate (COMPLETED/done) — no regression > 0.5pp
  • Error rate + retry counts — no regression
  • End-to-end duration p50/p95
  • Token usage + cache-read rate parity (loop logic is shared, so drift here = bug)
Radar-specific: callback delivery success; DynamoDB result-shape diff on sampled pairs; burst behavior at 13:00 UTC (queue absorption vs today’s stampede). Chat-specific: TTFT p50/p95 (the decision metric); recoverable-disconnect rate; $-per-turn including Workflows billing (events + data written — measured WITHOUT coalescing; the result decides whether PR 10 happens); no use-claude-chat client errors in Sentry attributable to the path. Kill switch: Redis workflow-dispatch:<surface>=off (≤5s propagation, works in every environment) or PostHog flag → 0% (production). Env vars are deploy-time defaults, NOT a kill path — Vercel env changes require a redeploy. Rollback ritual: flip the Redis key, confirm inline traffic recovers, then diagnose offline.

Telemetry contract (implemented in lib/radar-run-telemetry.ts, 2026-07-14)

Lifecycle events (PostHog, both transports, canonical context = radar_id / schedule_id / execution_id / clerk distinct id / dispatch_path / dispatch_source / workflow_run_id): workflow_dispatch (every decision) · workflow_fallback (workflow-intended dispatch degraded, with fallback_reason) · workflow_started (entry step, attempt 1, with queue_latency_ms) · workflow_step_failed (every failed attempt: step_name / attempt / max_retries / terminal / duration_ms / error_class) · workflow_callback_failed (delivery attempts, with callback HTTP status) · workflow_completed (terminal, BOTH paths: status / duration_ms / model / token+cache usage / error_class). Sentry receives ACTIONABLE failures only, with searchable tags (surface, dispatch_path, radar_id, workflow_run_id, error_class, stage/step_name): terminal step exhaustion, permanent callback 4xx, workflow-start failures, claim-cleanup failures. Sentry environment now reports VERCEL_ENV (preview/staging separable from prod).

Alerts to create before the staging bake ends (W2 entry gate)


7. Verification harness

  • Golden SSE transcripts (built in PR 2, reused forever): fixed prompt set — plain answer, single tool, multi-tool loop, forced 529 fallback, compaction-length conversation, GenUI component answer — recorded as ordered event-name+shape sequences. Diffed on every PR touching the loop, and across inline vs workflow paths in PR 8.
  • Radar parity run: silvia-simulation against a fixture portfolio; structured output (sendAlert, triggerMet, confidence, summary shape) diffed before/after PRs 3, 4, 6.
  • Load probe (pre-PR-7 graduation): synthetic 200-job radar burst against the workflow path in preview, watching queue latency + Anthropic 429 behavior.

8. Phase 2 (after both canaries graduate)

2026-07-21: chat’s Phase-2 slice PULLED FORWARD as W4.5 — the “only if 800s truncations are observed” condition fired on internal-canary turn #2 (video build, step 504 at 800s). Chat turns now slice at loop boundaries past 180s-per-invocation, with client re-attach for streams that outlive the /api/claude pump. See docs/plans/w3-chat-extraction-map.md → “W4.5 ADDENDUM”. Radar analyze (single-step, minutes-scale) stays as-is.
  • 2a — Resumable chat streams: client reconnect via run.getReadable({ startIndex }); retire the hand-rolled recovery (2s partial persistence + session rehydration). Own flag.
  • 2b — Step-per-model-call chat loop: decompose so deep-research turns escape any single invocation’s maxDuration. Only if 800s truncations are observed in practice.
  • 2c — QStash pair migration program: template from PR 6 applied to document queue first, then burst-prone schedulers.
  • 2d — Rate limiting (from the audit; independent of workflows): per-user request + concurrency caps on /api/claude; concurrency semaphore on radar analysis if the PR-6 spike finds no native per-workflow concurrency control.

9. Risks & open questions


10. File map (end state)