​

A/B Test Variant Tracking Bug Fix

Historical record — status as of 2026-09-23. This describes a fix to an earlier landing-page test. The two landing components it names, components/landing-page-client.tsx and components/landing-page-variant2.tsx, were deleted in cleanup PR 3 (docs/MODULUS-CLEANUP-PLAN.md); both had been unreferenced for months. components/variant-persistence.tsx still exists but nothing mounts it. The live landing experiment today runs through components/home-landing-router.tsx and hooks/use-landing-conversion-experiment.ts on a different PostHog flag — read those, not this, for how it works now.

​

Problem Summary

1. User visits landing page → sees control variant
2. User clicks sign in → enters email/code
3. During redirect to onboarding, page re-renders
4. New variant (test) briefly flashes on screen (< 1 second)
5. PostHog captures pageview with new variant instead of control
6. Session recording shows user saw control, but PostHog attributes them to test variant

​

Root Causes

​

1. Variant Assignment Instability

• getBootstrapData() called on every server render
• Variant could change between renders (especially after authentication)
• No persistence mechanism to lock variant assignment

​

2. Pageview Tracking During Redirects

• PostHog pageview tracking fired immediately on page load
• Captured brief variant flashes during authentication redirects
• No delay to filter out redirect flashes

​

3. Missing Experiment Exposure Tracking

• Only pageviews tracked, not explicit experiment exposure
• No session-based deduplication of exposure events

​

Solution Implemented

​

1. Variant Persistence (Server + Client)

• Check for persisted variant cookie before calling PostHog
• If variant exists and is fresh (< 24 hours), use it instead of PostHog assignment
• This prevents variant switching during authentication

• New component that persists variant in cookie on first page load
• Sets 24-hour expiration
• Includes timestamp for freshness check

​

2. Delayed Pageview Tracking

• Added 500ms delay for landing page (/) pageviews
• If page unmounts before 500ms (e.g., during redirect), pageview not tracked
• Prevents tracking brief flashes during authentication

​

3. Explicit Experiment Exposure Tracking

• Added useEffect to track $feature_flag_called event on mount
• Uses sessionStorage to prevent duplicate tracking
• Tracks which variant user actually saw, not just pageviews

​

Files Modified

1. /app/utils/getBootstrapData.js
   • Added variant cookie reading logic
   • Override PostHog variant with persisted variant if available
2. /components/variant-persistence.tsx (NEW)
   • Client component to persist variant in cookie
3. /app/(home)/page.tsx
   • Added <VariantPersistence> component to both variant branches
4. /components/landing-page-variant2.tsx
   • Added experiment exposure tracking on mount
   • SessionStorage-based deduplication
5. /components/landing-page-client.tsx
   • Added experiment exposure tracking on mount
   • SessionStorage-based deduplication
6. /app/_providers/posthog-provider.tsx
   • Added 500ms delay for landing page pageview tracking

​

How It Works

​

First Visit (No Cookie)

1. User visits / → getBootstrapData() called
2. No persisted variant found → PostHog assigns variant (e.g., “control”)
3. Page renders control variant
4. <VariantPersistence> component sets cookie: lp_experiment_variant={"variant":"control","timestamp":...}
5. Control variant tracks experiment exposure via $feature_flag_called
6. After 500ms (if still on page), pageview tracked

​

Sign-In Flow (Cookie Exists)

1. User clicks sign in → Clerk modal opens
2. User enters email/code → Clerk redirects
3. During redirect, page re-renders → getBootstrapData() called again
4. Cookie found with “control” variant → Uses persisted variant, not new PostHog assignment
5. Control variant renders (same as before)
6. Page unmounts before 500ms → No pageview tracked (prevents flash tracking)
7. User lands on /onboarding

​

Result

• User consistently sees control variant throughout session
• PostHog correctly tracks user in control group
• No variant switching during authentication
• No false tracking from redirect flashes

​

Testing Instructions

​

Test 1: Variant Persistence

1. Open incognito window
2. Visit landing page
3. Open DevTools → Application → Cookies
4. Verify lp_experiment_variant cookie is set
5. Note the variant value (e.g., “control”)
6. Refresh page multiple times
7. Expected: Same variant renders each time

​

Test 2: Sign-In Flow

1. Clear cookies and start fresh
2. Visit landing page → note variant shown (e.g., control)
3. Click “Sign In” button
4. Enter credentials in Clerk modal
5. Watch carefully during redirect
6. Expected: If brief flash occurs, it should be same variant (control)
7. Check PostHog session recording
8. Expected: PostHog shows user assigned to correct variant

​

Test 3: Experiment Exposure Tracking

1. Clear cookies and start fresh
2. Open DevTools → Console
3. Visit landing page
4. Look for log: 📊 Experiment exposure tracked: {experimentKey: "landing-page-conversion", variant: "control"}
5. Refresh page
6. Expected: Log appears only once per session (sessionStorage prevents duplicates)

​

Test 4: Pageview Delay

1. Visit landing page
2. Immediately navigate away (< 500ms)
3. Check PostHog events
4. Expected: No $pageview event for landing page
5. Visit landing page again
6. Stay for > 500ms
7. Expected: $pageview event appears in PostHog

​

PostHog Events to Monitor

​

Correct Tracking (After Fix)

1. $feature_flag_called
   - $feature_flag: "landing-page-conversion"  
   - $feature_flag_response: "control"
   - Timestamp: [When user first sees page]

2. signup_button_clicked (if user clicks CTA)
   - source: "landing_page" or "landing_page_variant2"
   - variant: "control"
   - experiment: "landing-page-conversion"

3. $pageview (only if user stays > 500ms)
   - $current_url: "https://mycfosilvia.com/"

​

Incorrect Tracking (Before Fix - Bug)

1. $pageview (during brief flash)
   - $current_url: "https://mycfosilvia.com/"
   - Variant: "new-test" (wrong - user saw control!)

2. signup_button_clicked
   - variant: "control" (correct - but inconsistent with pageview)

​

Cookie Schema

​

lp_experiment_variant

{
  "variant": "control",  // or "new-test"
  "timestamp": 1735945200000  // Unix timestamp
}

• Path: /
• Expiration: 24 hours
• SameSite: Lax
• Secure: Inherited from domain settings

​

Session Storage Schema

​

experiment_exposure_landing-page-conversion

• Value: "true"
• Purpose: Prevent duplicate exposure tracking within same session
• Cleared: When browser tab/window closes

​

Rollback Instructions

1. Remove variant persistence:
   • Remove lines 36-55 from /app/utils/getBootstrapData.js
   • Remove lines 69-72 from same file
2. Remove pageview delay:
   • Revert /app/_providers/posthog-provider.tsx to always track immediately
3. Remove exposure tracking:
   • Remove useEffect blocks from both landing page components
4. Remove cookie component:
   • Remove <VariantPersistence> from /app/(home)/page.tsx
   • Delete /components/variant-persistence.tsx

​

Future Improvements

1. Server-side variant persistence in database for logged-in users
2. Middleware-based variant assignment to prevent client-side delays
3. A/B test framework to handle multiple experiments consistently
4. Automated testing for variant stability across authentication flows

​

Related Issues

• PostHog A/B test inconsistency
• Variant switching during authentication
• Session recording vs analytics mismatch
• Feature flag evaluation timing

​

References

• PostHog Feature Flags: https://posthog.com/docs/feature-flags
• PostHog Experiments: https://posthog.com/docs/experiments
• Next.js Server Components: https://nextjs.org/docs/app/building-your-application/rendering/server-components