Overview

Teaching: 5 min
Exercises: 0 min

Questions

• What issues affect reproducibility?

Objectives

• To identify issues that affect reproducibility so that they can be addressed in the future

The issues that can affect reproducibility include:

• Low power
• Mistakes
• Ineffective Data Sharing
• Methodological Variance
• Sampling
• P – hacking (P-phishing)
  • Definition…
• Publication Bias
  • Only reporting potentially significant results, and not the rest

Some related publications:

• P-Curve: A Key to the File Drawer. Simonsohn, et al. Journal of Experimental Psychology: 2014, Vol. 143, No. 2, 534–547 link
• Power Failure: Why small sample size undermines the reliability of neuroscience. Button, et al. Nat Rev Neurosci. 2013 May;14(5):365-76. link

Emerging Solutions

• Low power –> Better Data Sharing
  • Special Issue on Neuroimaging Databases (Eickoff, et al, NeuroImage, 2016)
• Mistakes –> Better Methods/Results ‘Publication’
  • Workflow Specifications & Results Databases (i.e. NeuroVault)
• Methodological Variance –> Better Methods Description/Methods Standardization
  • Workflow Specifications & Standard Execution Environments
• P – hacking –> Analysis Pre-Registration
  • For Example: Open Science Foundation
• Publication Bias –> Results Databases
  • Results Databases (i.e. NeuroVault)

Key Points

• We want the user to be cognizant of the important issues that relate to reproducibility.

---