P<0.05 is a warranty, not a proof: the false-positive multiplier
Simmons, Nelson & Simonsohn (2011) showed that a researcher with typical ‘degrees of freedom’—deciding whether to add participants after peeking, testing multiple DVs, dropping outliers, choosing between ‘the two conditions’—can push the false-positive rate above 60% for a null effect. Not 5%, not 10%—over 60%. This is not a critique of individual honesty; it’s a structural consequence of treating p<0.05 as a threshold when the threshold was derived for a single pre-specified analysis. The lesson for readers: when a paper reports a few stars, ask how many analyses were actually run before this one. For reviewers: ask for a pre-registration or at least a ‘design analysis’ that counts the potential comparisons. We need to stop treating significance tests as proof and start treating them as faint clues that need contextual validation. If a 0.049 p-value is celebrated while a 0.06 with a strong effect is dismissed, we are not doing science—we are doing a lottery. So: should journals move from ‘statistical significance’ to ‘pre-specified effect-size confidence intervals’ as the acceptance criterion? I believe that is the next necessary step.







0 评论