Power & Sample Size Calculator
Why power comes before launch
Underpowered experiments waste traffic and return inconclusive results that only look like failed features. Power is the probability your test detects an effect that truly exists, and it depends on alpha, effect size, and sample size. A/B testers and research teams size studies before launch so the audience matches the smallest lift worth detecting.
The same math runs in reverse. Fix your target power, say 80 percent, and the tool solves for the per-group sample size the experiment needs before anyone hits publish.
The math, with a worked example
For a two-sample comparison of means, n per group = 2 * (z_a + z_b)^2 / d^2, with d the effect in standard deviations, z_a = 1.96 for two-sided alpha 0.05, and z_b = 0.84 for 80 percent power. Detecting d = 0.5 needs 2 * (2.8)^2 / 0.25 = 62.7, so roughly 63 users per group.
Halve the effect to d = 0.25 and the need grows past 250 per group, because required N scales with the inverse of the squared effect. That inverse-square rule is why saving traffic and saving certainty fight each other.
Sizing your study in the tool
Pick what to Calculate: Sample Size or Power. Then describe the setup with the Experiment for option, choosing Single sample or Difference between samples, and whether the Differences between involve Means (continuous) or Proportions (binomial).
Match The data is setting to your metric, enter alpha, target power, and the effect you care about, then press Calculate. If the requirement exceeds your traffic, raise the minimum detectable effect or plan a longer run rather than launching underpowered. Sizing first beats restarting an experiment mid-way every time.
Frequently Asked Questions
What does 80 percent power actually mean?
If the true effect equals your planning assumption, the test rejects the null 80 percent of the time. The remaining 20 percent is the risk of missing a real lift.
How does alpha influence required sample size?
Stricter alpha raises the bar for significance and demands more data. Moving alpha from 0.05 to 0.01 lifts the z multiplier from 1.96 to 2.58, increasing required sample size by about half in a typical two-group design.
Why do smaller effects need larger samples?
Required N is inversely proportional to the squared effect size, so a lift half as large forces about four times the sample per group. Subtle changes simply hide deeper inside sampling noise.
Sample size for proportions or for means?
Both are supported. Continuous outcomes like revenue use means and a standard deviation, while binomial outcomes like conversion rates use proportions, so choose the option matching your metric.
Should I run a single-sample study instead?
Choose Single sample when comparing one group against a known benchmark, such as a historical rate. That design needs fewer observations than a two-sample difference because only one estimate carries sampling error.