Advice and Information on Computational Reproductions: Students May Find Them Messier Than Expected
Nate Breznau, University of Bremen — October 13, 2022
Session 4 of the Webinar series: Replicating empirical studies in economics - an opportunity for students
Watch the pre-recorded video presentation prior to the session: here!
Registration for this session was possible [here!](https://ysi.ineteconomics.org/event/s4-advice-and-information-on-computational-reproductions/)
Use the discussion page to comment or ask questions in written form.
Slides of the talk available on Nate Breznau’s “Teaching Resources” OSF page (https://osf.io/qvmnb). The recording of this session can be watched here and listened to here.
Nate Breznau (University of Bremen), October 13, 2022, 18h Brussels/Paris/Berlin time.
Abstract
Drawing on experience teaching and conducting replications this talk focuses on the most basic form of replication: the computational reproduction. This means simply trying to reproduce the numerical results of a previous study using the same data and statistical routines from that study. As it turns out, there are many potential pitfalls that come from both the original study and the replicator, that could lead to different results. Different computing environments or data versioning and formatting are also culprits. This talk reviews these pitfalls using examples from published literature and several attempts across disciplines to computationally reproduce findings published in journals. It also reports results from a crowdsourced replication, where eighty-five independent teams attempted a computational replication of results reported in an original study that is central to economics and many other disciplines on the links between policy preferences and immigration. Although when teams had the original data and code they were able to achieve a high rate of similar results (95.7%), a random half of teams did not get the original code and struggled (89.3%). What was more surprising was that exact numerical reproductions to the second decimal place were far less common (76.9% and 48.1%). This has wider implications for science obviously, but it should serve as a lesson for students using replication that things are not a clean cut as one might hope. (full paper on the crowdsourced study) Watch the pre-recorded video presentation prior to the session: here!
Register for this session!
Bio
Postdoctoral Fellow at the University of Bremen. Researcher at the Comparative Research Center “The Global Dynamics of Social Policy”. Principal Investigator of the research project, “The Reciprocal Relationship of Public Opinion and Social Policy”. Principal Investigator of “The Crowdsourced Replication Initiative”. Open Science advocate. Open Science Fellow at Wikimedia with the project, “Giving the Results back to the Crowd”. User of preprints. Crowdsourcing and Mertonian-norm advocate. Author of the open science and crowdsourcing blog “Crowdid”.
This page mirrors the archived ReplicationWiki session page. The source text is preserved in full; only headings, link formatting, typography, and page structure have been adapted for this site.