In my experience, people seldom appreciate just how much more compelling a 5-sigma effect is than a 2-sigma effect. I suspect part of the problem is that p-values don’t invoke the visceral sense of magnitude that statements of the form, “this would happen 1 in K times”, would invoke.
To that end, I wrote a short Julia script to show how often a K-sigma effect would occur if the null hypothesis were true. A table of examples for K between 1 and 12 is shown below.
|A K Sigma Effect||Occurs 1 in N Times under the Null|
|1||1 in 3 times|
|2||1 in 22 times|
|3||1 in 370 times|
|4||1 in 15,787 times|
|5||1 in 1,744,278 times|
|6||1 in 506,797,346 times|
|7||1 in 390,682,215,445 times|
|8||1 in 803,734,397,655,352 times|
|9||1 in 4430,313,100,526,877,940 times|
|10||1 in 65,618,063,552,490,270,597,321 times|
|11||1 in 2,616,897,361,902,875,526,344,715,013 times|
|12||1 in 281,455,127,862,356,011,929,402,519,726,336 times|
One thing I like about showing things in this format: it’s much more obvious that increasing K by one additional unit has an ever-increasing effect on strengthening your claims. This is what I’m jokingly calling the “convexity of improbability” in the title: each additional sigma makes your claim much stronger than the previous one.