Skip to main content

​
A/B experiments

How to add a PostHog experiment to Silvia-Web with the shared primitive. Required reading before any experiment PR, human or agent.

​
The contract

Two files carry every experiment:
  • lib/experiments/registry.ts declares it: flag key, variants (control first), launch time, audience, surface, goal event, owner.
  • hooks/use-experiment.ts resolves it: useExperiment(EXPERIMENTS.name) returns { variant, isReady, isEligible }.
The hook owns exposure. It fires $feature_flag_called exactly once per user per browser session, before the state update that lets the treatment mount, so impressions and goal events always land after exposure. Do not call getFeatureFlag for the experiment anywhere else. Control is the default for every failure mode: flag missing, flag not yet started, PostHog absent, user outside the audience, flags not loaded. A merged experiment PR is inert until someone starts the experiment in PostHog.

​
Adding one

  1. Add the registry entry. Key convention is <surface>-<change>-vNN. launchedAt is the planned start time in UTC and is the eligibility cutoff for new-users audiences.
  2. At exactly one surface, read the hook and branch on variant. Gate any treatment-only rendering on isReady so users never flash control.
  3. Fire the goal event where the user converts, if it does not exist yet. Prefer an event the funnel already emits (see pricing_intent_*, onboarding_*, upgrade_cta_clicked).
  4. Add a unit test for the surface: control renders the old UI, test renders the new one, neither renders before isReady.
  5. Create the experiment in PostHog in draft with the same key and variants. The growth lane in llm-platform does this through the API. Starting it is a human action after the code is in production.
QA on previews: append ?<key_with_underscores>_variant=test to force a variant. Production ignores the override.

​
Retiring one

When the readout picks a winner: flip the flag to 100 percent on the winner in PostHog, then open a PR that hardcodes the winner, deletes the registry entry, and removes the branch. Delete the flag after that PR is in production.

​
Rules for agent-authored experiment PRs

  • One experiment per PR, one surface per experiment, control-first.
  • Copy, layout, ordering, timing, and visibility of existing elements only. No changes to Stripe or checkout API logic, auth, account connections, or anything that sends a message to users.
  • Keep the diff small. Around 150 lines excluding tests is the ceiling.
  • Never touch app/themes.css, app/globals.css, or components/ui/**. Variant-only styling goes through data-theme-variant per docs/THEMING.md.
  • Check docs/SILVIA-SKILLS-SYNC.md. A variant that renames or moves something the assistant explains to users needs a skills PR too.

​
History

Older experiments each carry their own hook (use-sidebar-cta-location-experiment, use-account-connection-security-experiment, use-cost-basis-line-flag). They keep working and are the source of the rules above. New experiments use the shared hook. Migrating the old ones is optional cleanup.