In my experience, people seldom appreciate just how much more compelling a 5-sigma effect is than a 2-sigma effect. I suspect part of the problem is that p-values don’t invoke the visceral sense of magnitude that statements of the form, “this would happen 1 in K times”, would invoke.

To that end, I wrote a short Julia script to show how often a K-sigma effect would occur if the null hypothesis were true. A table of examples for K between 1 and 12 is shown below.

A K Sigma Effect | Occurs 1 in N Times under the Null |
---|---|

1 | 1 in 3 times |

2 | 1 in 22 times |

3 | 1 in 370 times |

4 | 1 in 15,787 times |

5 | 1 in 1,744,278 times |

6 | 1 in 506,797,346 times |

7 | 1 in 390,682,215,445 times |

8 | 1 in 803,734,397,655,352 times |

9 | 1 in 4430,313,100,526,877,940 times |

10 | 1 in 65,618,063,552,490,270,597,321 times |

11 | 1 in 2,616,897,361,902,875,526,344,715,013 times |

12 | 1 in 281,455,127,862,356,011,929,402,519,726,336 times |

One thing I like about showing things in this format: it’s much more obvious that increasing K by one additional unit has an ever-increasing effect on strengthening your claims. This is what I’m jokingly calling the “convexity of improbability” in the title: each additional sigma makes your claim much stronger than the previous one.