The Convexity of Improbability: How Rare are K-Sigma Effects?

In my experience, people seldom appreciate just how much more compelling a 5-sigma effect is than a 2-sigma effect. I suspect part of the problem is that p-values don’t invoke the visceral sense of magnitude that statements of the form, “this would happen 1 in K times”, would invoke.

To that end, I wrote a short Julia script to show how often a K-sigma effect would occur if the null hypothesis were true. A table of examples for K between 1 and 12 is shown below.

A K Sigma Effect Occurs 1 in N Times under the Null
1 1 in 3 times
2 1 in 22 times
3 1 in 370 times
4 1 in 15,787 times
5 1 in 1,744,278 times
6 1 in 506,797,346 times
7 1 in 390,682,215,445 times
8 1 in 803,734,397,655,352 times
9 1 in 4430,313,100,526,877,940 times
10 1 in 65,618,063,552,490,270,597,321 times
11 1 in 2,616,897,361,902,875,526,344,715,013 times
12 1 in 281,455,127,862,356,011,929,402,519,726,336 times

One thing I like about showing things in this format: it’s much more obvious that increasing K by one additional unit has an ever-increasing effect on strengthening your claims. This is what I’m jokingly calling the “convexity of improbability” in the title: each additional sigma makes your claim much stronger than the previous one.