On the significance of statistically significant results

Andrew Gelman notes that

there are suprisingly many papers with results that are just barely
statistically significant (t=1.96 to 2.06) and surprisingly few that
are just barely not significant (t=1.85 to 1.95).

in a selection of empirical studies in political science; Mark Thoma wonders if this extends to economics as well. I’m pretty sure it does, although perhaps not to the same extent.

There is a case for excluding superfluous explanatory variables: if a right-hand-side variable really does have a ‘true’ coefficient of zero, there are efficiency gains to be had by imposing a correct restriction. But careful researchers are also aware of the perils of interpreting the results of ‘pre-test estimators’ that first test for significance and then re-estimate using only the RHS variables that pass a significance test. The standard errors for this estimator are not what the regression package might report.

As a Bayesian, I’m not ashamed to report results in which the posterior mean is (say) one posterior standard deviation greater than zero. Although a frequentist might think that the coefficient was not significantly greater than zero, I’d note that there’s about an 85% posterior probability that the coefficient is positive.

And in any case, a good model isn’t one with tightly-estimated parameters; any model with a sufficiently large sample size will give you arbitrarily large t-statistics. The true test of a model is out-of-sample prediction.

4 comments

  1. Paul Mason's avatar

    An interesting study would investigate the dodginess of published results with the proximity of the date for renewal of the researcher’s funding.

  2. EclectEcon's avatar

    Is it possible that articles without statistically significant results tend to be rejected, whereas those with significant results do get published?

  3. optimuscrime's avatar

    Great timing for this post — our graduate methods class just read Cohen’s ‘The World Is Round (p<.05)’ and had a great discussion about the problem of these useful-but-artificial distinctions between significance and insignificance. Thanks for the links.

  4. optimuscrime's avatar

    Here’s the ref. for anyone interested:
    Cohen, J. (1994). “The world is round (p < .05),” American Psychologist, 49, 997–1003.