Overview

Teaching: 5 min
Exercises: 0 min

Questions

• Can you cite some examples where issures related to reproducibility have been documented to be a problem?

Objectives

• Show some of the evidence that reproducibility is causing problems in neuroscience.

A number of studies have brought the reproducibility of science into question.

Prinz, et al. Nature Reviews Drug Discovery 10, 712 (September 2011) Bayer Healthcare

• link to paper

Image from Prinz:

Insert here some words of description related to the Prinz figure…

But just what is ‘reproducibility’ anyway?

• Re-executability (publication-level replication). The exact same data, operated on by the exact same analysis should yield the exact same result. This is currently a problem since publications, in order to maintain readability, do not typically provide complete specification of the analysis method or access to the exact data.

• Generalizability: We can divide generalizability into three variations:

  • Generalization Variation 1: Exact Same Data + Nominally ‘Similar’ Analyses should yield a ‘Similar’ Result (i.e. FreeSurfer subcortical volumes compared to FSL FIRST)

  • Generalization Variation 2: Nominally ‘Similar’ Data + Exact Same Analysis should yield a ‘Similar’ Result (i.e. my kids with autism compared to your kids with autism)

  • Generalized Reproducibility: Nominally ‘Similar’ Data + Nominally ‘Similar’ Analyses should yield a ‘Similar’ Result

Because we do not really characterize data, analysis and results very precisely in the current literature, ‘Similar’ has lots of wiggle room for interpretation (both to enhance similarity and discount differences).

Key Points

• Ir-reproducibility costs society dearly

---