A/B Testing for Developers: Feature Flags and Implementation

A/B testing compares two variants (A = control, B = treatment) by randomly assigning users to each and measuring whether a target metric differs significantly between groups. Statistical significance (p-value < 0.05) means: if there were truly no difference between variants, you would see a difference this large or larger by chance less than 5% of the time. It does not mean you are 95% confident the result is correct — it means you have controlled for random variation at a chosen threshold. Confidence intervals (e.g., 95% CI: +2.1% to +8.4% conversion lift) are more informative than p-values alone because they show the range of plausible true effects. Power (typically 80%) is the probability of detecting a real effect if one exists — underpowered tests produce false negatives. Run a power calculation before starting every test, not after.

The most common A/B testing mistake is stopping a test when it looks significant. To avoid this, calculate the required sample size before starting. The formula depends on: your baseline conversion rate (e.g., 4.2% checkout completion), your minimum detectable effect (MDE — the smallest improvement worth acting on, e.g., 10% relative lift to 4.62%), your desired statistical power (0.80 is standard), and your significance threshold (0.05 is standard). Use an online calculator (Evan Miller's A/B test calculator or PostHog's built-in calculator) rather than implementing the formula manually. Example: baseline 4.2%, MDE 10% relative lift, power 0.80, alpha 0.05 → approximately 14,000 users per variant. If your traffic cannot reach that in 2–4 weeks, the MDE is too small to detect reliably — either increase the MDE target or accept that the test will be underpowered.

A minimal A/B assignment system requires: deterministic user assignment (same user always gets same variant), random distribution across variants, and tracking of which variant each user saw. A hash-based assignment is deterministic without storing assignments in a database:

1// Minimal A/B test assignment using FNV-1a hash
2// Deterministic: same userId + experimentId always returns same variant
3
4function hashString(str: string): number {
5  let hash = 2166136261; // FNV offset basis
6  for (let i = 0; i < str.length; i++) {
7    hash ^= str.charCodeAt(i);
8    hash = (hash * 16777619) >>> 0; // FNV prime, keep as 32-bit unsigned
9  }
10  return hash;
11}
12
13type Variant = 'control' | 'treatment';
14
15function assignVariant(
16  userId: string,
17  experimentId: string,
18  trafficPercent = 100 // 0-100: percentage of users in the experiment
19): Variant | null {
20  const hash = hashString(`${experimentId}:${userId}`);
21  const bucket = hash % 10000; // 0-9999
22
23  // Check if user is in the experiment at all
24  if (bucket >= trafficPercent * 100) return null;
25
26  // Split remaining traffic 50/50
27  const variantBucket = hashString(`${experimentId}:variant:${userId}`) % 2;
28  return variantBucket === 0 ? 'control' : 'treatment';
29}
30
31// Usage in a Next.js API route or React component
32export async function getExperimentVariant(
33  userId: string
34): Promise<{ checkoutButtonColor: Variant | null }> {
35  return {
36    checkoutButtonColor: assignVariant(userId, 'checkout-button-color-v2', 50),
37  };
38}
39
40// Track assignment alongside page view
41async function trackExperimentAssignment(
42  userId: string,
43  experimentId: string,
44  variant: Variant
45): Promise<void> {
46  await fetch('/api/analytics/experiment', {
47    method: 'POST',
48    headers: { 'Content-Type': 'application/json' },
49    body: JSON.stringify({
50      userId,
51      experimentId,
52      variant,
53      timestamp: Date.now(),
54      sessionId: getSessionId(),
55    }),
56  });
57}

For production use, replace this with a dedicated feature flag service: PostHog Feature Flags (open-source, self-hostable), GrowthBook (open-source), or LaunchDarkly (commercial). These add a UI for managing experiments, automatic exposure logging, and built-in statistical analysis.

Peeking and stopping early: checking results daily and stopping when p < 0.05 inflates your false positive rate from 5% to 26%+. Fix: commit to the pre-calculated sample size before starting and do not stop early. Multiple comparisons (the multiple testing problem): testing 10 metrics simultaneously means one will appear significant by chance at p < 0.05 even if nothing changed. Fix: pre-register one primary metric; treat secondary metrics as exploratory. Novelty effect: users click a new design element because it is new, not because it is better — the effect disappears after a week. Fix: run tests for at least two weeks to capture steady-state behaviour. Sample ratio mismatch: your variant assignment is not 50/50 (e.g., 52/48 split) indicating a bug in the assignment or tracking code. Fix: always check the actual split before analysing results. Interaction effects: running multiple experiments simultaneously where users can be in both — variant A of test 1 may interact with variant B of test 2. Fix: use a mutex layer in your flag system to prevent overlap on the same UI surface.

PostHog provides feature flags, experiment assignment, and automatic statistical analysis in one open-source package. Basic integration:

1// PostHog A/B test integration (Next.js)
2import posthog from 'posthog-js';
3
4// Initialize once in _app.tsx or layout.tsx
5posthog.init(process.env['NEXT_PUBLIC_POSTHOG_KEY']!, {
6  api_host: process.env['NEXT_PUBLIC_POSTHOG_HOST'] ?? 'https://app.posthog.com',
7  loaded: (ph) => {
8    if (process.env.NODE_ENV === 'development') ph.opt_out_capturing();
9  },
10});
11
12// Get feature flag variant (returns string or boolean)
13const checkoutVariant = posthog.getFeatureFlag('checkout-cta-experiment');
14// Returns: 'control' | 'bold-cta' | undefined
15
16// PostHog automatically logs $feature_flag_called events
17// Track your conversion metric explicitly:
18function trackCheckoutStarted(variant: string): void {
19  posthog.capture('checkout_started', {
20    experiment_variant: variant,
21    // Include other relevant properties for segmentation
22  });
23}
24
25// React hook for clean variant consumption
26function useExperimentVariant(flagKey: string): string {
27  const [variant, setVariant] = React.useState('control');
28  React.useEffect(() => {
29    const v = posthog.getFeatureFlag(flagKey);
30    if (typeof v === 'string') setVariant(v);
31  }, [flagKey]);
32  return variant;
33}

Define the experiment in the PostHog UI with your primary metric (e.g., conversion rate on 'checkout_started'), set the target sample size from your power calculation, and PostHog will run Bayesian or frequentist analysis automatically and notify you when significance is reached.