has_created_radar sync invariant
Invariant
For every Silvia user, the following three surfaces must eventually converge on the same truth value for whether the user has scheduled at least one radar:
Segment 16 is the global exit condition on the radar-activation
drip (Campaign 12) and the inverse-target for every “no radar yet”
broadcast. Drift between
radar_schedules and CIO segment 16 means
users who already activated will keep receiving activation
emails, which is the exact bug that motivated this document.
Who writes each layer
radar_schedules is populated by the AWS-managed radar Lambda in
response to a POST from one of three Silvia-Web surfaces:
/api/radar/schedules(POST), the/radarpage UI.app/api/chat/tools.ts:createRadarEvent, the Silvia chat tool./api/incoming-call-webhook-tool/create-radar-event, the ElevenLabs voice webhook.
- Layer 2 (
all_users.has_created_radar) is a direct Supabase UPDATE with the service-role key. Idempotent, since settingtrueon an already-true row is a no-op. - Layer 3 (CIO
has_created_radar) isidentify(clerkId, { has_created_radar: true })vialib/customerio.ts. Idempotent for the same reason.
try/catch blocks whose exceptions are
logged to Vercel logs but never propagated. This is by design,
since a Customer.io outage must not fail a user-facing radar create.
What could go wrong (and did, before this doc)
The inline stamps silently fail whenever:CUSTOMERIO_ENABLED=falseat deploy time (isEnabled()short-circuitsidentify()),- the Supabase UPDATE returns a non-throw error (logged but ignored),
shouldSkipCustomerIo()returnstruefor a legitimate user (e.g. Supabase read fails, so the guard defaults to “skip”),- the request handler throws upstream of the stamp block,
- the voice webhook (historically) never wrote to
all_usersat all.
Convergence guarantee
/api/cron/reconcile-radar-flag runs every 15 minutes. It closes
every gap using a CIO-first, DB-after per-user loop that is safe
under both handler timeouts and Track-API failures.
Detection runs three paginated scans in parallel and produces two
candidate sets:
- DB-drift candidates are
clerk_ids inradar_schedulesthat do not appear inall_users.has_created_radar = true. - CIO retry window candidates are
clerk_ids whoseradar_schedulesrow was created in the past 24 hours (rolling backstop for stamps whose CIO leg silently failed while the DB leg succeeded).
- Call
identify(clerkId, { has_created_radar: true })viasetHasCreatedRadar. - If (and only if) the CIO call succeeded AND the user was in
the DB-drift set, UPDATE
all_users.has_created_radar = true.
identify-ing a user
whose account was soft-deleted by app/api/account/delete/route.ts
would silently resurrect their CIO profile and opt them back into
every drip they were removed from. Both filters read the same
columns (all_users.is_test_user and all_users.account_deleted)
that shouldSkipCustomerIo and the delete flow rely on, so the
cron cannot diverge from the inline callers on eligibility.
The bulk pre-filter is refreshed with a per-user recheck right
before each Track-API call, which reduces (but does not eliminate)
the delete-versus-identify race window to the ~50 ms between the
recheck SELECT and the CIO PUT. A concurrent account delete that
lands inside that ~50 ms window can still cause this cron to
re-identify an offboarded profile with has_created_radar = true; that write is a leak of a single attribute, not of drip
delivery, because
app/api/account/delete/route.ts:softDelete sets unsubscribed = true in CIO and that flag blocks every broadcast even after
subsequent attribute writes. Eliminating the residual window
would require coordinating with the delete flow through shared
state (advisory lock or “pending_deletion” marker read by both
sides); we’ve accepted the CIO-side backstop as sufficient for
now. Users flagged ineligible at recheck time are counted as
raceSkipped in the summary.
Ordering inside the CRON is load-bearing. The DB flag flips
ONLY after the CIO stamp succeeds, so a Vercel maxDuration
timeout or a Track-API outage leaves the DB in a state where the
NEXT cron tick can re-detect the same clerk_id and retry. No
user is left in the “DB flag true, CIO not stamped” purgatory by
this route.
Inline PRODUCERS (app/api/radar/schedules/route.ts,
app/api/chat/tools.ts:createRadarEvent, and the voice webhook)
write in the opposite order: DB first, then CIO. A CIO failure
in a producer therefore does leave the “DB flag true, CIO not
stamped” state, which is exactly the drift the 24 h CIO retry
window is designed to catch. The cron-only ordering guarantee
above is what keeps a runaway backlog draining without stranding
rows; it is NOT a claim that no code path ever produces that
state.
Per-tick cap is 1500 users, a defense-in-depth stop that fits
inside the 250 s hard time budget at ~134 ms/user with the
recheck included (the actual limit is wall-clock, not the count).
A DB-drift backlog larger than 1500 drains at 1500/tick =
6000/hr; the remainder is picked up on the next tick because
those users still fail the drift check.
In steady state (no backlog, no failures), worst-case drift is
15 min (cron interval) + ~2 min (CIO segment recompute), so
every radar creator lands in segment 16 within ~20 min. When
the cron is draining a backlog, convergence for the tail of the
backlog scales as ceil(backlog_size / 1500) * 15 min. For
example, the historical 4,200-user backfill would have taken
three ticks (~45 min) via the cron alone, which is why the
one-off backfill script exists for large historical drifts.
The convergence guarantee is bounded by Customer.io availability.
A Track-API outage lasting longer than the 24-hour CIO retry
window can strand a user whose DB stamp succeeded inline but
whose CIO stamp failed and could not be retried before the window
expired. That user is neither in the DB-drift set (DB flag is
already true) nor in the recent-radar set (radar older than 24 h),
so the cron will not detect them. Outages that long are already
paged and require running the one-off backfill script from the
operator playbook below.
Failure surfaces the cron does NOT cover
- Radars deleted after creation.
has_created_radarintentionally staystrueonce set (per the CIO attribute doc comment inlib/customerio.ts). If we ever need “actively has a radar” as a distinct concept, add a new attribute; do not derive it from this one. - Users deleted from CIO for reasons OTHER than a Silvia-side
account delete (e.g. manually removed from the CIO dashboard).
The cron re-
identifys the user and resurrects a minimal profile. Silvia-side deletes are handled by theaccount_deleted = truefilter described above, so the resurrection only happens for out-of-band deletions. - Customer.io outages longer than 24 hours. See the caveat at the end of the Convergence guarantee section.
One-off backfill script
scripts/backfill-customerio-has-created-radar.ts performs a
full-fleet sync of all_users.has_created_radar = true up to CIO
for every user where is_test_user IS NOT TRUE AND
account_deleted IS NOT TRUE. Used once on 2026-08-27 to close
the historical 4,200-user gap. Safe to re-run; every write is
idempotent, and each worker re-reads eligibility right before
issuing the Track-API PUT so a concurrent account delete during
the run cannot slip through (the workers log a SKIP audit line
for those and count them in the final summary). Dry-run supported
via --dry-run.
Operator playbook
Symptom: users report receiving radar-activation emails after they created a radar.- Confirm the drift is real:
- Check the CIO segment 16 count vs. the DB flag count; they should be within a few hundred of each other. Larger gaps indicate the cron isn’t running or the CIO write leg is broken.
- Check Vercel logs for
[cron/reconcile-radar-flag]; every tick logs a summary withdbDriftDetected,identifiesFired,identifiesFailed,dbFlipped,raceSkipped,deferred,filterChunksFailed,elapsedMs.deferred > 0means the tick hit the hard time budget and the remaining candidates roll to the next tick, which is normal during a backlog drain but concerning if it stays non-zero indefinitely.filterChunksFailed > 0means PostgREST rejected one or more eligibility-filter chunks (typically URL-length or network); the tick still processed whatever chunks succeeded but ran degraded.raceSkipped > 0is expected non-zero any time an account delete lands mid-tick. - If the cron is unhealthy, re-run the backfill script: