A/B Testing Framework

A/B Testing Framework generates statistically rigorous experimentation infrastructure to avoid the common mistakes that invalidate most A/B tests.

Pre-Experiment Design

Sample size calculator with inputs: baseline conversion rate, minimum detectable effect (MDE), statistical power (80% default), and significance threshold (α=0.05). Runtime estimator based on current traffic volume. Multiple comparison correction (Bonferroni) for multi-variant tests.

Assignment

Deterministic user bucketing via MurmurHash3 on `user_id + experiment_id`. Ensures users see the same variant on every visit. Traffic allocation by percentage. Holdout groups for long-term effect measurement. Exclusion rules to prevent experiment interference.

Feature Flags

Integrates with LaunchDarkly, Unleash, or a self-hosted flag service. Server-side flag evaluation prevents flickering. SDK wrappers for React (useFlag hook), Python, and Go.

Analysis

Frequentist — Z-test for proportions, t-test for continuous metrics, chi-square for multi-category. Confidence intervals. p-value with multiple testing correction.

Bayesian — Beta-Binomial conjugate model for conversion rates. Probability to be best, expected loss, credible intervals. Thompson Sampling for multi-armed bandit scenarios.

Common Pitfalls Detection

Sample Ratio Mismatch (SRM) detection, novelty effect warnings for long-run test drift, and network effect warnings for social products.