Casper Albers: what is the replication crisis and how can it be solved?

What is the replication crisis and how can it be solved?

In the past decade, many within the field of social sciences have realized there are flaws in the scientific process. Many scientific results turn out to be too good to be true. A large-scale replication study from 2015 showed that only about a third of psychological results stand up to scrutiny: two thirds are noise. The problem here is that, without additional research, it is impossible to determine which of every three psychological studies is reliably executed.

The cause of this crisis comes down to the fact that it is too easy to – consciously, but primarily unconsciously – cheat a little bit. When you have put a lot of time and energy into a study and see that you have not found a significant result, it is quite tempting to gather just a few more bits of data to obtain that significance. The outcome is that the results are incorrect.

A conscious demonstration of such an approach by Simmons, Nelson and Simonsohn from 2011 shows that this allows you to obtain almost any result. In an apparently correct way, they calculate that you can literally get younger if you listen to the Beatles, and then explain what statistical errors they have made and obscured.

The department of Psychometrics & Statistics is working to improve scientific practice. We aim to explain the rules of the game better, and promote open data and open science. Another pillar is pre-registration: a researcher is required to record all of their choices before the data is analysed. This results in a much higher level of replicability.