The Blog of Andrew Syrios
  • Home
  • About
  • Writings and Interviews
  • Contact

The Blog of ANDREW sYRIOs

And Science's Reproducibility Crisis Marches On

9/20/2018

Comments

 
So the roots of science's ever-growing reproducibility crisis probably lie in what's called the "file drawer problem" or "publication bias." As Wikipedia describes,
Publication bias is a type of bias that occurs in published academic research. It occurs when the outcome of an experiment or research study influences the decision whether to publish or otherwise distribute it. Publication bias matters because literature reviews regarding support for a hypothesis can be biased if the original literature is contaminated by publication bias. Publishing only results that show a significant finding disturbs the balance of findings.

Studies with significant results can be of the same standard as studies with a null result with respect to quality of execution and design. However, statistically significant results are three times more likely to be published than papers with null results.
And just how bad is the problem? Well according to a new study from The Royal Society; really, reall bad.
In this paper, we show how Bayes' theorem can be used to better understand the implications of the 36% reproducibility rate of published psychological findings reported by the Open Science Collaboration. We demonstrate a method to assess publication bias and show that the observed reproducibility rate was not consistent with an unbiased literature. We estimate a plausible range for the prior probability of this body of research, suggesting expected statistical power in the original studies of 48–75%, producing (positive) findings that were expected to be true 41–62% of the time. Publication bias was large, assuming a literature with 90% positive findings, indicating that negative evidence was expected to have been observed 55–98 times before one negative result was published. These findings imply that even when studied associations are truly NULL, we expect the literature to be dominated by statistically significant findings.
The underlined part is mine, of course. And I'm just going to repeat that again. "Publication bias was large, assuming a literature with 90% positive findings, indicating that negative evidence was expected to have been observed 55-98 times before one negative result was published."

Ouch! How long until we can conclude that every new science study--at least those in sociology and psychology--is completely bogus?
Comments
comments powered by Disqus

    Andrew Syrios

    "Every day is a new life to the wise man."

    RSS Feed

    Archives

    November 2022
    September 2022
    August 2022
    July 2022
    June 2022
    May 2022
    April 2022
    March 2022
    February 2022
    January 2022
    December 2021
    November 2021
    September 2021
    August 2021
    July 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    June 2020
    May 2020
    April 2020
    March 2020
    February 2020
    January 2020
    December 2019
    November 2019
    October 2019
    September 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    February 2018
    January 2018
    December 2017
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    May 2017
    April 2017
    March 2017
    February 2017

    Picture
    Business Websites

    Stewardship Properties
    333 Rent
    Blog Roll

    The Real Estate Brothers
    The Good Stewards


    Bigger Pockets
    REI Club
    Meet Kevin
    Tim Ferris
    Joe Rogan
    Adam Carolla
    MAREI
    1500 Days
    Worcester Investments
    Just Ask Ben Why
    Entrepreneur
    Inc.
    KC Source Link
    The Righteous Mind
    Star Slate Codex
    Mises Institute
    Tom Woods
    Michael Tracey

    Consulting by RPM
    The Scott Horton Show
    Swift Economics
    The Critical Drinker
    Red Letter Media

    Categories

    All

    View my profile on LinkedIn
Powered by Create your own unique website with customizable templates.
  • Home
  • About
  • Writings and Interviews
  • Contact