Replication webinar series

The opening session presented the YSI Philosophy of Economics Working Group and introduced replication as a continuum: from rerunning an original study’s data and code, to testing robustness or generalizability with new data and methods. It covered how to identify studies suitable for replication, share results, detect errors, contribute to scientific progress, and navigate resistance from authors and editors.

Gustavo A. Castillo Alvarez conceived the series through the YSI working group. Jan H. Höffler founded ReplicationWiki in 2013 and has researched and taught replication internationally. Diana Soeiro is a scientific writer and policy analyst whose work connects philosophy, public policy, governance, and sustainable development.

This session distinguished ex post replication—independently reproducing results from a previous publication—from ex ante documentation: assembling and publicly sharing the data, code, and supporting information needed to reproduce a new study. It introduced Project TIER and showed why learning either practice helps students perform the other.

Richard Ball, Professor of Economics, and Norm Medeiros, Associate Librarian, are the co-founders and co-directors of Project TIER at Haverford College.

Macroeconomics combines established public datasets with substantial, often study-specific computation. The session examined the field’s still-developing replication practices: what successful and failed replications mean, how to choose and conduct a replication, when to contact original authors, how to make new work easier to reproduce, and where the field needs to improve.

Robert Kirkby is a macroeconomist at Victoria University of Wellington. His research develops theoretical and computational tools for macroeconomic policy analysis.

Computational reproduction means trying to regenerate a study’s numerical results using its original data and statistical routines. Drawing on published work and a crowdsourced project involving 85 independent teams, this talk showed how missing code, computing environments, data versions, formatting, and decisions by both authors and replicators can prevent exact reproduction even when broad conclusions agree.

Nate Breznau researches social policy, public opinion, crowdsourced replication, and open science at the University of Bremen.

This student project replicated Auerbach and Gorodnichenko’s 2016 article, “Effects of Fiscal Shocks in a Globalized World.” It recovered the direction of the original effect but not the exact results, tested an extended sample period, and evaluated potential sources of the remaining differences.

Franziska Strunz was a PhD student and research assistant at Universität der Bundeswehr Munich, working in macroeconomics.

The talk examined questionable research practices and the open-science reforms intended to limit false or misleading results. It also asked what the public thinks of these practices, what happens when they enter courtrooms and influence legal decisions, and what economists can learn from that experience.

Jason Chin is an interdisciplinary legal scholar with training in social psychology and law. His work studies evidence, open science, and practical responses to the challenges faced by lawyers and courts.

The authors replicated Section IV.A of Chetty et al. (2020), which analyzes how consumer spending, business reopenings, and employment responded to state reopening policies. Because the original replication materials were not public, they implemented the methods independently in Stata. Their estimates did not reproduce the originals exactly, but supported broadly similar conclusions. The project also proposed extending the analysis to additional reopening policies.

Original study: Chetty et al., “The Economic Impacts of COVID-19: Evidence from a New Public Database Built Using Private Sector Data,” NBER Working Paper 27431. The presenters were undergraduate students at Haverford College.

This presentation analyzed open-science practices and publishing requirements in leading political science journals, including preregistration, and surveyed their implementation in Latin America. It addressed both the promise of reproducibility for theory-building and the epistemological and methodological debates around making these practices disciplinary standards.

Related article: “Documentation Requirements in Political Science Journals: Moving Towards Open Access Practices.” Carolina Curvale is a political scientist specializing in institutions and political economy in Latin America. Gustavo Pérez-Arrobo researches the political economy of development, research methods, and Latin American politics.

This project reproduced Oskooii’s analysis of perceived discrimination and minority voting behavior, then extended it with European Social Survey data for the United Kingdom and Western Europe. The extension found that group discrimination lowered the propensity to vote, in contrast with prominent findings in the literature.

Andrej Cvetić was a PhD student and research assistant in political science at Trinity College Dublin, studying political psychology and immigrant integration in Western Europe.

This session discussed interim results and next steps for replicating Orlando J. Sotomayor’s “Can the minimum wage reduce poverty and inequality in the developing world? Evidence from Brazil” (2021).

Oscar Jaramillo is an economist from the Central University of Ecuador who pursued a master’s degree in economics at Universidad Torcuato Di Tella in Argentina.

The session reviewed large replication projects in behavioral economics and psychology, including why larger samples are often needed when original effect sizes are exaggerated. It also covered prediction markets and forecasting surveys for replication outcomes, along with reforms such as Registered Reports that can reduce publication bias and improve reliability.

Anna Dreber is the Johan Björkman Professor of Economics at the Stockholm School of Economics. Her research focuses on meta-science, behavioral economics, replication, and forecasting research outcomes.

This session considered replication from the perspective of an academic publisher and journal editor. Jack Reardon is the founding editor of the International Journal of Pluralism and Economics Education and has written extensively on pluralist economics education, sustainability, and the history of economic thought.

This study examined the reproducibility of Krause, Krause, and Bränfors (2018), which found that women signing peace agreements significantly improves the durability of peace. The principal findings were reproduced despite minor inconsistencies, while additional analysis identified low statistical power and underspecified theoretical and empirical estimands as important limitations.

Lenka Olejníková completed a PhD at UNSW on the limitations of statistical modeling in research on gender and later worked at the intersection of child sexual abuse and domestic violence.

The presentation discussed the scarcity of reproducible work and replication studies in agricultural economics and policy, along with the causes and consequences. Drawing on an Applied Economic Perspectives and Policy special issue containing 11 replication studies, it proposed ways for researchers, journals, and policymakers to strengthen replicability and make better use of replication evidence.

Robert Finger is Professor of Agricultural Economics and Policy at ETH Zürich and Chair of its World Food System Center. His research connects agricultural science and economics to more resilient and sustainable food systems.

The session focused on an often-overlooked part of open scholarship: teaching students how to do transparent and reproducible research. It introduced FORRT’s open educational resources and discussed curricular reform, epistemological pluralism, inclusive teaching, academic incentives, and practical ways to incorporate open scholarship into teaching and mentoring.

FORRT is an international, interdisciplinary community working to integrate open-scholarship principles into higher education. Flavio Azevedo researches political psychology and anti-scientific attitudes; Helena Hartmann studies social, cognitive, and affective neuroscience; and Bethan Iley researches conspiracy beliefs and resistance to social change.