And Science's Reproducibility Crisis Marches On

So the roots of science's ever-growing reproducibility crisis probably lie in what's called the "file drawer problem" or "publication bias." As Wikipedia describes,

Publication bias is a type of bias that occurs in published academic research. It occurs when the outcome of an experiment or research study influences the decision whether to publish or otherwise distribute it. Publication bias matters because literature reviews regarding support for a hypothesis can be biased if the original literature is contaminated by publication bias. Publishing only results that show a significant finding disturbs the balance of findings.

Studies with significant results can be of the same standard as studies with a null result with respect to quality of execution and design. However, statistically significant results are three times more likely to be published than papers with null results.

And just how bad is the problem? Well according to a new study from The Royal Society; really, reall bad.

In this paper, we show how Bayes' theorem can be used to better understand the implications of the 36% reproducibility rate of published psychological findings reported by the Open Science Collaboration. We demonstrate a method to assess publication bias and show that the observed reproducibility rate was not consistent with an unbiased literature. We estimate a plausible range for the prior probability of this body of research, suggesting expected statistical power in the original studies of 48–75%, producing (positive) findings that were expected to be true 41–62% of the time. Publication bias was large, assuming a literature with 90% positive findings, indicating that negative evidence was expected to have been observed 55–98 times before one negative result was published. These findings imply that even when studied associations are truly NULL, we expect the literature to be dominated by statistically significant findings.

The underlined part is mine, of course. And I'm just going to repeat that again. "Publication bias was large, assuming a literature with 90% positive findings, indicating that negative evidence was expected to have been observed 55-98 times before one negative result was published."

Ouch! How long until we can conclude that every new science study--at least those in sociology and psychology--is completely bogus?