Webinar Series Introduction: Why replication? How is it done? Where to find replication material?

← Back to the webinar series

Session 1 of the Webinar series: Replicating empirical studies in economics - an opportunity for students

Registration was possible here for the live question & answer session on September 8, 2022, 18h Brussels/Paris/Berlin time. Watch the pre-recorded video here. Use the discussion page to comment or ask questions in written form. Recordings of the session are available to watch or listen.

Gustavo A. Castillo Alvarez (Universidad de los Andes), Jan H. Höffler (ReplicationWiki), Diana Soeiro (ISCTE-University Institute of Lisbon), September 8, 2022, 18h CEST

Abstract

We present the Institute for New Economic Thinking Young Scholars working group Philosophy of Economics.

We then introduce the concept of replication in the narrow sense of just rerunning original studies’ data and code to check results and learn how they are produced to updating or using entirely new data or methods to check for robustness and/or generalizability. We give advice how to identify studies suitable for replication and how to share results. With examples from the literature we show how replications help to question published findings, detect errors, contribute to scientific progress and incentivize discussion but also how they face resistance from authors of original studies and editors.

Apart from the Young Scholars Initiative working group Philosophy of Economics and the ReplicationWiki that we will introduce in this session, the webinar series was prepared with the Project Teaching Integrity in Empirical Research (TIER) that will be described in the next session.

Outline¹

In this introduction to the webinar series, we will first present our working group on Philosophy of Economics, then our personal motivation to engage in this activity, we continue with why in general replication is important in science, then with how it is done, we give some focus on where to find material for replication, we continue with technical instructions for this webinar series, and in the end you find the literature that we cite during the presentation.

Working group Philosophy of Economics

The Institute for New Economic Thinking (INET) was founded after the global financial crisis. Apart from sponsoring research and organizing events it also set up the Young Scholars Initiative (YSI) because the next generation of researchers was seen as pivotal for not only coming up with ideas for new economic thinking but especially for sustainably putting them into practice. It is structured into working groups with regional or thematic focus like for South or East Asia, Africa, Europe, [Northern] and Latin America, Economic Development, History of Economic Thought or Sustainability, Inequality, or Finance, Law and Economics. Anyone can freely choose to join these groups and have a say on what they should focus on. Our group for Philosophy of Economics has organized ongoing groups for discussion and presentation of our research, or research we are interested in, for writing, for peer review, on feminist epistemology, we had workshops, and participated in events like the Turin Festival of Economics.

Personal motivations

We entitled this webinar series “Replicate, Replicate - How You Can Take Part” because we want to activate the next generation of researchers. Motivating why replications are important in general in the social sciences is not difficult. But there are many things that are important in some way, and we all need to make choices about what we care about enough to take action. To convince individual students why exactly they should engage, we thought it could help to identify if we also present our personal motivation why we got involved.

Gustavo

In the midst of my econometric courses in my undergraduate program we started to see how some of the methods that we had previously studied weren’t necessarily the best methods for empirical work. In more advanced undergrad courses we were taught ways of addressing these weaknesses that came with the different methods we had been taught. This meant that there were (or are) many ways to approach an empirical problem, which in turn means that there is no one-size-fits-all method or approach, not even if the same topic is addressed or even if the same data is used. In fact, as I kept learning econometrics I realized that most of the times there is no “clean”, optimal, and perfect method with which to approach a particular analysis. At some point in time there was a need for the researchers to be able to argument and convince themselves that the particular empirical approach was indeed appropriate. The takeaway in the end always was a call to being critical, curious and most importantly, responsible on the approach one takes. This left space for doubt, for questioning, learning and discussion, which is what I later learned is at the heart of replicating empirical studies.

In the first exercises with data during my undergrad program we used simulated data. We were solicited interpretations and sometimes even recommendations. At the same time they stressed the importance of ethical implications of our conclusions. But how is one expected to truly understand the practical implications when working with simulated data? The need to use real data in undergrad courses seemed important to me.

The first (and so far, my only) laboratory experiment I ran was on public goods. Our results were not in accordance with the literature, nor with our hypothesis. Only we had the original data and fiddling with the data to get what we were “supposed” to get did go through our minds, because in the end we would still be graded. In my first courses we were obsessed with statistical significance, and even though now some top journals prohibit these indicators, it seems like the tradition or “stargazing” obsession is still there, and as long as it is we require a way to discourage the search for these asterisks, and replication certainly serves as a solution for these well established incentives.

A professor’s (Andrés Moya, Uniandes) presentation on a failure to replicate a study on dishonesty (Dan Ariely et al’s 2012 PNAS study) during the same class where I ran my first experiment pointed me to the irony and misfortune of dishonesty in an article regarding dishonesty. The question bounces back to us, how to assess dishonesty, but not in a insurance policy context, but in research contexts?

Knowing about DataColada’s work was surprising. Their collaborative work was online, for free, voluntary and most importantly constructive. It is still surprising to me that this was published on a website and not in a journal, regarding the impact that this PNAS study had.

Jan

Gustavo’s experience reminds me I once participated in an experiment by psychologists in Germany and one of them said “We already did this once but we didn’t get the results that we wanted”…

It was my first PhD supervisor who pointed me to the topic of replication. We gave a first seminar on replications, which turned out a bit of a disaster with only two students finishing their papers and one of them at the last moment after months of trying to write complicated code that the original authors would have had to provide to the journal archive. After the exchange of many emails that were often nice but didn’t get him what he needed he finally called them, and over the phone they just said frankly: We have worked on that for five years with four people, we will not share the code. My supervisor was very focused on publishing his own research, never seemed to care much about the students, and showed disappointment that they could not easily be converted into labor force for a project on replicability. So I realized to make such courses work they need to be designed well and they need good instructors.
I saw a former professor of mine had been editor for a pivotal article on replication in economics in the Canadian Journal of Economics.² He had been one of the best instructors I had ever had. He actually seemed to care both for what he was teaching AND for the students, which is a good but not super common combination, especially among the highly skilled who are often skilled at doing what they do but not at teaching others about it. So I contacted him, raised funding to travel to Canada and joined him for a seminar. He was super well organized, deep thinking, and that passes on to students, high quality teaching leads to high quality results.
But before I came to that point I was working on first research that did not seem replicable even to me. My PhD supervisor had asked me to convert a part of his own thesis that he had written some years earlier into a book chapter, just updating it a bit so he’d have one more publication on his list. His interpretations of results to me seemed like made up explanations of what might be reasons why the results occurred, without actual tests that these were really the reasons for the results. He said he did not have the code or data any longer, and our university had a rule that data needed to be kept for ten years. Some of the sources cited came to results contradicting those of others, and identifying the reasons for that and thus drawing own conclusions was impossible as the underlying replication materials at the time were generally not available. It frustrated me, I kept procrastinating about it, and my advice to all students is: Don’t waste time on research that you do not really want to do. You will never be good at it, and if even you do not like it, it is unethical to try and convince others to like or even publish and read it.
I read about Institute for New Economic Thinking’s grants, and new thinking - that made me think about what I could contribute and what mattered to me.
I had been editing on Wikipedia since the 9/11 attacks when it turned out a very good source of information condensing many pieces of information into relevant text at a high speed. As a public good it is often undervalued by economists. It is my impression that Wikipedia articles on economics topics are often of much lower quality than articles from other fields. At the university you are always told Wikipedia is not a scientific source, and of course it isn’t, and it’s true anyone can edit it and enter low quality information, but the volunteers working on it very often provide high quality sources. I wrote my grant proposal to INET based on some of the sources on lack of replicability and even fraud in research and explained how crowdsourcing works on Wikipedia and how I thought a wiki offering teaching material and collecting replications as a grassroots activity could make a contribution.
Then I found out my PhD supervisor had his student assistants edit Wikipedia articles to promote his research and he did not even admit that this breaks rules on conflicts of interest³ and paid editing⁴ and obviously any common sense principle of how an encyclopedia should be written. So I had to search for a new supervisor - but that’s a different story.
The INET grant allowed me to set up the ReplicationWiki, to travel to many conferences internationally, that’s how I got in touch with the Young Scholars Initiative and Richard and Norm of Project TIER, and to teach replication at different universities on three continents. I found that rewarding and that’s why I continue doing it now.

Diana

My academic background is Philosophy-Economics-Public Policy and I work in the realm of Cities and Spatial Planning. In this context, since 2017 I’ve been working with the United Nations Global Compact Network aiming at harmonizing management and governance strategies. This implies dwelling on SDG 16⁵ which is about promoting “peace, justice and strong institutions“. At stake are matters of transparency, access to information (and its accessibility) and combat to corruption.

The role of replication in Economics, and in any other discipline, is key to assure that the standard of science remains high making its results reliable and trustworthy. Additionally, scientific data is often used as the foundation to inform and support a specific public policy, or to influence broader agenda-setting strategies. In complex decision-making, it is not possible to be knowledgeable in every scientific field that is relevant to find the best possible solution. Therefore, conducting a replication is crucial to support and validate scientific, public and general confidence bringing added legitimacy and trust to results. This is why it is important to acknowledge that replication, not only is important in theory but also in practice having the ability to, sooner or later, significantly impact our lives.

Key relevant questions that experts can contribute to, and that will be approached in this Webinar, encouraging students to put their skills to the test are: How to choose an article or a study to replicate, what is the best strategy to replicate, which approaches can you take to replicate, which protocol to follow to assure you replicate well, what obstacles can you expect to find and how to overcome them, when should you abandon a replication and know it won’t work, what is a successful replication, how to publish a replication?

Why replication?

We haven’t yet defined what we are actually talking about, what replication exactly is, and as we will see later that is not trivial, but at the moment let’s just say it is looking at an empirical study again, checking if you can get the original results again and maybe also see what happens if you make certain changes.

A reason why we should do this is it
1) helps learning about new methods or data sources. A lot of empirical work lies in data cleaning, how to treat missing values, outliers, the decision what should actually be seen as outliers. You will see that there are many decisions that need to be taken and it is often not straightforward which ones are better than others. This is described as the “researcher degrees of freedom”. Realizing this hands on while doing a replication
2) supports critical thinking. Just because something is peer reviewed and published doesn’t mean it is perfect and the final word has been spoken, science makes continuous progress in often very small steps. Replication can be seen as a tool for empowerment, we can contribute to this scientific progress. You can
3) make an own contribution: build on published findings, question them, detect errors, test robustness and generalizability

If you wonder what these last two terms mean, hold on for a second, that will be explained further down in this text.

A student replication that made a change

With this example we want to show that already as a student one can make an important contribution. Thomas Herndon was a PhD student taking a replication seminar when he tried to replicate an article on the relationship between countries’ debt and their economic growth. It had made the news and was cited even by politicians as it presented an easily understandable stylized fact that seemed to have direct policy implications: Countries that are indebted by more than 90% of the GDP usually experience significantly less or even negative growth. Thomas could not replicate the original results with the data available to him, asked the authors of the original study, who were both professors at Harvard University, for the data they had used, and when he finally received them he found that some of the data had erroneously not been used for the analysis that had been done with a simple Excel sheet. Left out data in an Excel sheet is an error even a clever high school student should have been able to detect. Thomas did a deeper analysis than just finding that error but given that it was on the very hot topic relevant for politicians around the world in the time after the financial crisis when many countries followed austerity measures directly affecting peoples’ lives, two aspects already led to a debate around the world: It seemed that economists don’t have the right safeguard measures in place so that such errors are detected in a systematic way. And how come economic studies can be based on such simple descriptive statistics? To be fair the original authors had already presented a book with deeper analysis themselves but their reaction to the detection of the error did not convince many, and the journal editors even refused to publish a correction as they argued it was just a conference issue, which to me is a lame excuse because if they could publish a flawed conference contribution why would they not be able to publish at least a correction online? So the replication was published in the Cambridge Journal of Economics.⁶ Thomas joined us for a replication workshop we had in San Francisco in 2016 while he was on the job market. He became assistant professor and is now at the John Jay College Graduate Program in Economics at the City University of New York.⁷

Some people regard replications as a threat to the reputation of researchers. In this case it apparently has not had a dramatic effect as one of the original authors, Carmen Reinhart, has become World Bank Chief economist (on leave from Harvard Kennedy School).^{8 9}

As we saw, replication can

4) incentivize discussion.

It can also help to

5) introduce a more diverse perspectives to economics,

which as Daniel Hamermesh pointed out in his 2007 article, is very much dominated by researchers based in the US working with US data, so there is a lot of room for replications with data from other countries to improve the understanding of the world economy.

It is a horror of many researchers that their research could be called irreplicable, which sounds as if associating them with malpractice or even fraud.
We have to be careful to consider what it really means when a study’s result cannot be replicated. When we deviate from the original study like just described using different data, but also when using different methodology, it should be no surprise that outcomes can be different, and that doesn’t necessarily invalidate the original research.

How to replicate?

Replication in the narrow sense

In the narrow sense, replicating would mean to just rerun original studies’ data and code: check results and learn how they are produced.¹⁰

Of this most basic form you’d say it should be taken for granted that it usually works: If researchers put their data and code into some journal archive or on their website they should at least check that it produces the results they publish.

This most basic form of replication should always be the first step for a student replication project if possible because only then if you get different results with your own analysis you can detect whether the deviations result from the adjustments you made or if already with the original material that you had there was a problem. And then you can disentangle what role each factor played.

We initially thought when we will have done these easy checks then we can go on and do what Daniel Hamermesh suggested and test if results still hold if for example data from different countries is used. But we usually did not even make it to that step. And that for the articles for which replication material is available, which is still a small minority in the social sciences, even though in economics by now most of the very top journals have introduced mandatory archives for replication material - but there are still many articles for which you cannot easily access the data as they are proprietary or sensitive, and there are articles for which software or even hardware was used that is not available to many researchers.

Even if a replication in the narrow sense isn’t possible because the original data and/or code or soft- or hardware are not available, a replication can still make sense if reasonably similar data and/or code can be produced. Like in Thomas Herndon’s case who first replicated the study with the data he could find and then when he showed it to the original authors they shared their data and the reason of the deviation could be found.

When asking original authors for material it is always helpful if you show that you have really prepared and don’t just cause them work of which you could have done important parts without their help.

Replication in a wider sense

In a wider sense, a replication can also entail updating or using entirely new data or methods to check for robustness and/or generalizability.¹¹ Here replications can make an important contribution beyond just checking correctness of published results. They can help to overcome the incentive to preferably publish statistically significant results. Replication can help to overcome the incentive to preferably publish “significant” results. This is called the publication bias: editors often prefer publishing studies that present results that pass certain statistical thresholds, usually p-values, that are supposed to indicate that the results are to a certain degree unlikely to have been produced by chance, because such studies are usually more read and cited. Who wants to read a study that says “I had an interesting hypothesis but it turned out I was wrong.”? Just as in the case of the laboratory experiment that I participated in, some researchers find unscientific ways to avoid such outcomes. Authors know that it is difficult to publish research that doesn’t pass the p-value thresholds and out of anticipatory obedience try to meet them, so they dump research that doesn’t meet them. That is called the file-drawer effect. Or they may engage in questionable research practices, fiddling around with their regressions until they get the p-values that meet their hypotheses, that is called p-hacking, or if they find significant results where they had not expected them they just make up hypotheses that fit, that is called HARKing, hypothesizing after results are known. There is an ongoing debate about whether p-values help us at all, for that we refer to another webinar series.¹² Replications can help to overcome the problems as while there are many legitimate decisions how empirical research should be done, replicators may have different views from those of the original researchers about which decisions are the best. If this leads to differences in results, it can be discussed and ideally contribute to a consensus. Robustness checks would mean checking if reasonable deviations from the original work lead to different results, to check if just the one specification is presented that gives significant results or if different legitimate choices do not have strong effects, for example including different control variables or using different estimation strategies. Of course any result will break at some point if you try hard enough, so you have to carefully consider which choices are convincing and what deviations of replication results mean for the interpretation of the original study. It should not be the aim of a replication to kind of add another publication bias by searching for specifications until something significantly different from the original results shows up.

When generalizability is concerned, so if by using different data we check if original results generalize to other time periods or countries or regions, as already mentioned, an original result is of course not invalidated if it doesn’t generalize. What can be invalidated are overconfident conclusions that can often be encountered in the literature where results based on data from a certain time period and place are frequently described as if they held everywhere. So authors write: “We show that X has an effect on Y” instead of “We show that X had an effect on Y in the time period and the country, region, city or firm or firms we observed.”¹³

Keep in mind that in the end any study can be seen as a replication as there won’t be any study that doesn’t build on previous ones in terms of methodology, research question and data. It should be seen the more as a replication the more it focuses on checking results of previous studies or comparing them to the ones presented.

Many terminologies have been suggested for how to call these different types of what we all call replication here. Some researchers use the term reproduction but etymologically it is not clear what the difference between replication and reproduction would be. Some authors use the terms interchangeably, some use replication as the umbrella term as we do, some do it the other way round and think replication is just one form of reproduction. Some use descriptive terms like “direct replication” but there you need to be careful again: To a behavioral economist it may seem obvious that a direct replication is if an experiment is rerun. Economists who work with data from for example natural experiments that they cannot even rerun would probably think that a direct replication is one where the original data was used again. Even in the first abstracts of our presentations for this webinar series we saw that our different presenters use different terminology for some types of replication. All inherently made sense but it can be confusing if in the context of the same webinar series different terms are used. So be careful to clearly understand what type of replication is meant when you hear different terms, and if you are unsure just ask.

Why students can make a contribution: They are less under pressure to publish, so their time is not lost if they just learn from a publication but do not find anything that they could write up for a journal. Students around the globe are a huge labor force that as Hamermesh already pointed out can be a valuable resource. With them replication can become a mass grass roots activity that can contribute to a change in the field: the narrow sense of replication is already a worthwhile activity for undergraduates because it is essentially following the steps authors take to produce the results seen in the study, and in following those steps a whole lot is learned. More advanced students can go beyond.

After what we presented so far there are still a lot of open questions on how to exactly proceed in which case but we hope many of them can be clarified in the course of this webinar series.

Prepare a preanalysis plan!

What definitely helps: Make a plan. Instead of poking around in the original study until you find something that could be publishable, carefully consider what you want to investigate and how, what your hypotheses are and how you want to test them. What are your hypotheses and how do you want to test them?

In other contexts, like field experiments, preanalysis plans are used to show how researchers planned ahead and are perceived as a quality sign. That works where data collection is a huge effort and one can trust that the plan was actually there before the data. With replications where the data have already been collected, a preanalysis plan will not help much to convince anyone that you planned ahead as you could just have produced it after your analysis. It however helps yourself to make a clear plan! Even if you don’t invest weeks or months to collect data, clearly planning ahead can safe you valuable time compared to endless trials and errors of what could be interesting to do.

Replication examples

Here we list some replications that we consider particularly worth reading. In the first one from 2013 it was shown replicating many studies that even using different versions of the same data set, here the Penn World Table, can lead to significantly different results in published studies.¹⁴ For many empirical studies one can think of alternative data sources that could be used to check results, and doing so can help to understand how solid the conclusions are that we derive from empirical research.

More recently, a study in the Journal of the European Economic Association used data to empirically test the conclusions of a theoretical paper,¹⁵ which is another form of work in the context of replication that deserves attention. Many times one of the authors of this article, Jan, as founder of the ReplicationWiki has been asked if there should not also be a ReplicationWiki for theoretical studies as there are many errors known in the literature that have never been corrected anywhere publicly. We totally agree that this would be a valuable contribution and hope we can broaden this initiative.

In a third case, the authors replicated several dozen studies on related topics, all published in top journals and just investigated if they could get the same results qualitatively and what where the reasons if it didn’t work out.¹⁶ We had Andrew Chang with us for a YSI congress in Budapest. He had a particularly revealing side story to tell about the American Economic Review editor in chief during a panel discussion at the 2016 Annual Meeting of the American Economic Association that his and Philipp’s work was just flawed because in contrast to what they wrote all research published in the American Economic Review can easily be replicated. Andrew and Philipp had clearly described in which cases which problems had occurred. I still hope to convince Andrew that he shares some of their insights with us for this webinar series, too. Their study received a lot of media attention but they could not publish it in a top journal, and the publication took seven years, which brings us to the next point:

Resistance from authors of original studies & editors

Editors of the Journal of Political Economy repeatedly blocked the publication of replications even when studies contained clear errors,¹⁷ also Econometrica refused to publish a correction of a study that contained errors.¹⁸ We intentionally do not cite the original authors but comments that were not published in academic journals as we regard it important to depersonalize debates. A lot of the aggressiveness in debates on replications results from original authors’ fear that their name will be identified with a conflict rather than with their contribution, so we think it is important not to speak of for example the “Reinhart and Rogoff scandal” or something like that but of the debate on the relationship between economic growth of countries and their debt - even if that doesn’t sound equally catchy.

The next one is a drastic example of aggressive language in a replication debate: Daron Acemoglu et al 2012 entitled the working paper for a response to a replication of their work in the American Economic Review: “Hither thou shalt come but no further”.¹⁹

More recently, an editor of the Economic Journal even replied to a correction of a study that had been positively reviewed that after checking with others it turned out they have the general policy not to publish replications²⁰ - even though you can see in the ReplicationWiki that they have published some in the past.²¹ Usually they were on studies that had been published in other journals. Like that journals avoid conflicts with their own editors, reviewers and authors who may think that corrections question the quality of their work. Well, refusing to publish corrections is telling about the quality of the editing.

Where to find replication material?

Data and code availability is often an issue. It is difficult to detect reasons of deviations from original results if original ones cannot be obtained. But that can be an opportunity as we saw in the case of Thomas Herndon!

ReplicationWiki

Apart from listing more than 760 replications that can inspire you for your own work, it offers a database also of some corrections and retractions, and of more than 5,600 empirical studies informing about the availability of replication material. It also covers studies published in less prestigious journals or even books or blogs. Some are from way before the journals introduced mandatory data or code archives. You can search by keywords, JEL codes, data sources, data types, methods and software used. It does not cover all the most recent studies, but it has already a number of studies from neighboring social science disciplines, and we are working on an expansion.

The website Find Economic Articles with Data…

… was set up by Sebastian Kranz. It covers only studies for which some replication material is available in the archives of a number of economics journals. It covers a bit more studies than the ReplicationWiki, and one can also search in their abstracts.

You can also search in the

data and code archives of journals

directly. The ReplicationWiki offers a list of journals that keep such archives. It is great if you can find additional journals whose archives should be listed. It is a wiki, so just go ahead and add them. Another potential place where you can find replication material is

author websites

Many authors put links to replication material for their research on their websites, some even do so for articles published in journals that have no mandatory data and/or code archives. If they indicate it there, they may get more attention to their research, and less emails asking them for the material. Before you ask authors for their material, always check if the availability of their data and code is indicated on their website.

Most important for your replication: Search for a topic that you are really interested in! Interest assures motivation, and if you have a motivation on top of getting a good mark or in the ideal case a journal publication, this may help you to be less tempted to resort to unscientific practices to improve the publication chances of your replication work.²²

Technical instructions for the webinar series

Check the schedule on the webinar series page and choose which presentations you’d like to participate in.

Create accounts on the ReplicationWiki and on the YSI platform.

Watch the pre-recorded video lecture before the live sessions.

If you want to present your own replication in the webinar series you can prepare a video for a live session - but if you prefer to only present live that is also fine. In general for all the discussion sessions if there are participants who prefer not to have them recorded that is fine, we can always put a written summary online that we need anyway for our special journal issue.

Present your replication preparing a video for the live sessions! Those presenting replications can also start a sub-page under the webinar site on the ReplicationWiki with your session’s information. Gustavo prepared instructions for this.²³

To prepare your session, link to your replication materials (if you wish to share publicly) or just summarize on the page what you have done.

Participate in live Q & A sessions of your peers (if available) and contribute to discussion pages of your peers at the time of your convenience!

The community of YSI has members from all over the world, thus we have adopted this flipped classroom format so that it is possible to participate even without the hassle to schedule or work with several time zones with the help of the our platforms, both on YSI and the ReplicationWiki. Thus the importance of having your accounts.

Wiki pages on the ReplicationWiki with tutorials and guides to create pages, creating and following discussions in a wiki page, and whatever else is necessary. If there is something that isn’t clear or that you deem requires a tutorial either feel free to create it yourself, or at least share the need with us so that we can create the corresponding tutorial or help page.

In brief, actively engage during the webinar to make the most of it! We are looking forward to the discussions both live and on the ReplicationWiki discussion pages!

Presenter bios

Gustavo A. Castillo Alvarez has been working in a project to fight disinformation in Colombia (Detox Information Project) since 2021, and studies political science and economics at Universidad de los Andes where he participated in the Complexity Economics Research group. He had the idea to start this webinar series in the YSI working group Philosophy of Economics.

Jan H. Höffler founded the ReplicationWiki in 2013 during his PhD in economics at the University of Göttingen after studies at Mannheim University, Humboldt-University of Berlin and exchange programs at University of Toronto and Graduate Institute, Geneva. He has presented about replication at numerous conferences and as invited speaker internationally, published about the topic in the American Economic Review Papers and Proceedings and D-Lib Magazine and has been teaching replication at various German universities, University of Toronto, Graduate Institute, Geneva, Nanjing University, and United Nations University in Maastricht since 2008.

Diana Soeiro, scientific writer and policy analyst, currently hosted at ISCTE-Lisbon University Institute. PhD in Philosophy and Urban Studies (2011); Post-Grad in Economics and Public Policy (2018). My most recent book Cities, Health and Wellbeing: Global governance and intersectoral policies (Palgrave 2022) emphasizes the role of urban design, innovative governance strategies and intersectoral policies. I am co-coordinator of the ‘Philosophy of Economics’ Working Group at YSI-Institute for New Economic Thinking (New York, USA). In 2017, I was appointed Honorary Ambassador for United Nations 2030 Agenda for Sustainable Development Goals (UN Global Compact Network, Portugal).

References

Category:Webinar series

This page mirrors the archived ReplicationWiki session page. The source text is preserved in full; only headings, link formatting, typography, and page structure have been adapted for this site.