Skip to main content

has_created_radar sync invariant

Invariant

For every Silvia user, the following three surfaces must eventually converge on the same truth value for whether the user has scheduled at least one radar: Segment 16 is the global exit condition on the radar-activation drip (Campaign 12) and the inverse-target for every “no radar yet” broadcast. Drift between radar_schedules and CIO segment 16 means users who already activated will keep receiving activation emails, which is the exact bug that motivated this document.

Who writes each layer

radar_schedules is populated by the AWS-managed radar Lambda in response to a POST from one of three Silvia-Web surfaces:
  1. /api/radar/schedules (POST), the /radar page UI.
  2. app/api/chat/tools.ts:createRadarEvent, the Silvia chat tool.
  3. /api/incoming-call-webhook-tool/create-radar-event, the ElevenLabs voice webhook.
Each of these three routes performs the inline stamp for layers 2 and 3 after the Lambda accepts the create:
  • Layer 2 (all_users.has_created_radar) is a direct Supabase UPDATE with the service-role key. Idempotent, since setting true on an already-true row is a no-op.
  • Layer 3 (CIO has_created_radar) is identify(clerkId, { has_created_radar: true }) via lib/customerio.ts. Idempotent for the same reason.
Both stamps are wrapped in try/catch blocks whose exceptions are logged to Vercel logs but never propagated. This is by design, since a Customer.io outage must not fail a user-facing radar create.

What could go wrong (and did, before this doc)

The inline stamps silently fail whenever:
  • CUSTOMERIO_ENABLED=false at deploy time (isEnabled() short-circuits identify()),
  • the Supabase UPDATE returns a non-throw error (logged but ignored),
  • shouldSkipCustomerIo() returns true for a legitimate user (e.g. Supabase read fails, so the guard defaults to “skip”),
  • the request handler throws upstream of the stamp block,
  • the voice webhook (historically) never wrote to all_users at all.
Any single one of those turns a radar creator into a “still receives activation drip” ghost. The 2026-08-27 audit found ~4,200 such ghosts.

Convergence guarantee

/api/cron/reconcile-radar-flag runs every 15 minutes. It closes every gap using a CIO-first, DB-after per-user loop that is safe under both handler timeouts and Track-API failures. Detection runs three paginated scans in parallel and produces two candidate sets:
  • DB-drift candidates are clerk_ids in radar_schedules that do not appear in all_users.has_created_radar = true.
  • CIO retry window candidates are clerk_ids whose radar_schedules row was created in the past 24 hours (rolling backstop for stamps whose CIO leg silently failed while the DB leg succeeded).
Processing runs each candidate through the same sequential loop, with DB-drift users placed first so a time-budget bail always prioritizes them:
  1. Call identify(clerkId, { has_created_radar: true }) via setHasCreatedRadar.
  2. If (and only if) the CIO call succeeded AND the user was in the DB-drift set, UPDATE all_users.has_created_radar = true.
Test-user AND offboarded-account rows are filtered out up-front, since test fixtures never enter CIO and re-identify-ing a user whose account was soft-deleted by app/api/account/delete/route.ts would silently resurrect their CIO profile and opt them back into every drip they were removed from. Both filters read the same columns (all_users.is_test_user and all_users.account_deleted) that shouldSkipCustomerIo and the delete flow rely on, so the cron cannot diverge from the inline callers on eligibility. The bulk pre-filter is refreshed with a per-user recheck right before each Track-API call, which reduces (but does not eliminate) the delete-versus-identify race window to the ~50 ms between the recheck SELECT and the CIO PUT. A concurrent account delete that lands inside that ~50 ms window can still cause this cron to re-identify an offboarded profile with has_created_radar = true; that write is a leak of a single attribute, not of drip delivery, because app/api/account/delete/route.ts:softDelete sets unsubscribed = true in CIO and that flag blocks every broadcast even after subsequent attribute writes. Eliminating the residual window would require coordinating with the delete flow through shared state (advisory lock or “pending_deletion” marker read by both sides); we’ve accepted the CIO-side backstop as sufficient for now. Users flagged ineligible at recheck time are counted as raceSkipped in the summary. Ordering inside the CRON is load-bearing. The DB flag flips ONLY after the CIO stamp succeeds, so a Vercel maxDuration timeout or a Track-API outage leaves the DB in a state where the NEXT cron tick can re-detect the same clerk_id and retry. No user is left in the “DB flag true, CIO not stamped” purgatory by this route. Inline PRODUCERS (app/api/radar/schedules/route.ts, app/api/chat/tools.ts:createRadarEvent, and the voice webhook) write in the opposite order: DB first, then CIO. A CIO failure in a producer therefore does leave the “DB flag true, CIO not stamped” state, which is exactly the drift the 24 h CIO retry window is designed to catch. The cron-only ordering guarantee above is what keeps a runaway backlog draining without stranding rows; it is NOT a claim that no code path ever produces that state. Per-tick cap is 1500 users, a defense-in-depth stop that fits inside the 250 s hard time budget at ~134 ms/user with the recheck included (the actual limit is wall-clock, not the count). A DB-drift backlog larger than 1500 drains at 1500/tick = 6000/hr; the remainder is picked up on the next tick because those users still fail the drift check. In steady state (no backlog, no failures), worst-case drift is 15 min (cron interval) + ~2 min (CIO segment recompute), so every radar creator lands in segment 16 within ~20 min. When the cron is draining a backlog, convergence for the tail of the backlog scales as ceil(backlog_size / 1500) * 15 min. For example, the historical 4,200-user backfill would have taken three ticks (~45 min) via the cron alone, which is why the one-off backfill script exists for large historical drifts. The convergence guarantee is bounded by Customer.io availability. A Track-API outage lasting longer than the 24-hour CIO retry window can strand a user whose DB stamp succeeded inline but whose CIO stamp failed and could not be retried before the window expired. That user is neither in the DB-drift set (DB flag is already true) nor in the recent-radar set (radar older than 24 h), so the cron will not detect them. Outages that long are already paged and require running the one-off backfill script from the operator playbook below.

Failure surfaces the cron does NOT cover

  • Radars deleted after creation. has_created_radar intentionally stays true once set (per the CIO attribute doc comment in lib/customerio.ts). If we ever need “actively has a radar” as a distinct concept, add a new attribute; do not derive it from this one.
  • Users deleted from CIO for reasons OTHER than a Silvia-side account delete (e.g. manually removed from the CIO dashboard). The cron re-identifys the user and resurrects a minimal profile. Silvia-side deletes are handled by the account_deleted = true filter described above, so the resurrection only happens for out-of-band deletions.
  • Customer.io outages longer than 24 hours. See the caveat at the end of the Convergence guarantee section.

One-off backfill script

scripts/backfill-customerio-has-created-radar.ts performs a full-fleet sync of all_users.has_created_radar = true up to CIO for every user where is_test_user IS NOT TRUE AND account_deleted IS NOT TRUE. Used once on 2026-08-27 to close the historical 4,200-user gap. Safe to re-run; every write is idempotent, and each worker re-reads eligibility right before issuing the Track-API PUT so a concurrent account delete during the run cannot slip through (the workers log a SKIP audit line for those and count them in the final summary). Dry-run supported via --dry-run.

Operator playbook

Symptom: users report receiving radar-activation emails after they created a radar.
  1. Confirm the drift is real:
  2. Check the CIO segment 16 count vs. the DB flag count; they should be within a few hundred of each other. Larger gaps indicate the cron isn’t running or the CIO write leg is broken.
  3. Check Vercel logs for [cron/reconcile-radar-flag]; every tick logs a summary with dbDriftDetected, identifiesFired, identifiesFailed, dbFlipped, raceSkipped, deferred, filterChunksFailed, elapsedMs. deferred > 0 means the tick hit the hard time budget and the remaining candidates roll to the next tick, which is normal during a backlog drain but concerning if it stays non-zero indefinitely. filterChunksFailed > 0 means PostgREST rejected one or more eligibility-filter chunks (typically URL-length or network); the tick still processed whatever chunks succeeded but ran degraded. raceSkipped > 0 is expected non-zero any time an account delete lands mid-tick.
  4. If the cron is unhealthy, re-run the backfill script: