Severity Calculator
What severity adds to significance
After your experiment returns a significant result, a natural question follows: which discrepancies does the evidence actually rule out, and how well? Severity answers that. Formally, SEV is the post-data power-style function from the severe testing literature, measuring how capable your test was of detecting discrepancies, judged in light of the result you actually saw.
The tool computes severity for differences between proportions and for differences between means. Graduate students defending a study, A/B testers auditing experiments, and analysts reporting results all use it to state what a p-value leaves unsaid.
The computation, with a worked example
Severity works like power but is evaluated at your observed statistic, in standard error units. Suppose an observed z of 2.1 from a one-sided Z-test at alpha 0.05. The severity for discrepancies up to 0.4 SE above the null is Phi(2.1 – 0.4) = Phi(1.7), about 0.955, showing the test was highly capable of catching a shift that size.
Push the discrepancy to 2 SE and severity collapses to Phi(2.1 – 2) = Phi(0.1), about 0.54. A significant p-value alone never reveals that contrast; the severity curve does.
Running the calculator
First state where the Severity for differences lies: between proportions or between means. Next pick the Statistic to use, Z-test or T-test, and set Alpha, typically 0.05.
Then supply the Input as raw data or summary data and press Calculate SEV. Read the severity values across difference sizes: low figures warn that your significant result barely separates larger effects from noise, while high figures confirm the test would very likely have flagged them.
Frequently Asked Questions
What does SEV stand for here?
SEV is severity, the post-data power-style function from the severe testing literature. It reports the probability your test would have flagged a given discrepancy, judged against the result actually observed.
How is severity different from power?
Power is fixed in advance at chosen alternatives, while severity is computed after the data arrive at the observed statistic. Same mathematical family, different reference point: the actual result rather than a hypothetical effect size.
Can a significant result have low severity?
Yes. With a tiny sample, an observed z near 1.65 can pass a 0.05 cutoff while severity for even modest discrepancies stays low, meaning the evidence barely separates that effect from background noise.
Which statistic should I choose, Z or T?
Use the Z-test when variance is known or the sample is large, and the T-test for small samples with estimated variance. The tool then applies the matching distribution’s tail behavior.
Does severity work with proportions too?
Yes. Choose the between proportions option, supply counts or rates as summary data, and the tool evaluates severity on the proportion difference scale using your stated alpha.