The Convexity of Improbability: How Rare are K-Sigma Effects?
In my experience, people seldom appreciate just how much more compelling a 5-sigma effect is than a 2-sigma effect. I suspect part of the problem is that p-values don’t invoke the visceral sense of magnitude that statements of the form, “this would happen 1 in K times”, would invoke.
To that end, I wrote a short Julia script to show how often a K-sigma effect would occur if the null hypothesis were true. A table of examples for K between 1 and 12 is shown below.
| A K Sigma Effect | Occurs 1 in N Times under the Null |
|---|---|
| 1 | 1 in 3 times |
| 2 | 1 in 22 times |
| 3 | 1 in 370 times |
| 4 | 1 in 15,787 times |
| 5 | 1 in 1,744,278 times |
| 6 | 1 in 506,797,346 times |
| 7 | 1 in 390,682,215,445 times |
| 8 | 1 in 803,734,397,655,352 times |
| 9 | 1 in 4430,313,100,526,877,940 times |
| 10 | 1 in 65,618,063,552,490,270,597,321 times |
| 11 | 1 in 2,616,897,361,902,875,526,344,715,013 times |
| 12 | 1 in 281,455,127,862,356,011,929,402,519,726,336 times |
One thing I like about showing things in this format: it’s much more obvious that increasing K by one additional unit has an ever-increasing effect on strengthening your claims. This is what I’m jokingly calling the “convexity of improbability” in the title: each additional sigma makes your claim much stronger than the previous one.