… the actions of economists today bear on the life chances of the world’s population far more substantially than do the actions of the members of most other professions.

George DeMartino

Replication is an activity that doesn’t attract enough attention, enough credit, or enough effort in the social sciences. But it is an activity that is getting a lot of attention at this precise moment. This has come courtesy of the exposure of both flaws and contestable methodological choices in Reinhart and Rogoff’s landmark study of public debt and economic growth.

The economic blogosphere has exploded with debate over the issue. But, just in case you’re not following it, here are the key points. Reinhart and Rogoff followed up their major historical work looking at debt and economic growth This time is different with a paper called Growth in a time of debt published in the American Economic Review in 2010. Their key result is that levels of public debt in excess of 90% of GDP are associated with lower rates of economic growth. Indeed, the mean annual growth rate they report, once debt crosses the 90% threshold, is negative.

This body of work is highly influential.

A quick search on Google Scholar will tell you that the NBER version of Growth in a time of debt has been cited 450 times, while This time is different has been cited over 2000 times since 2009. That is a lot of citations for social science publications: you’re doing pretty well once citations for a piece get into double figures within four years.

In a paper published this week, Thomas Herndon, Michael Ash, and Robert Pollin (HAP) of the University of Massachusetts, Amherst identify problems with the Reinhart and Rogoff result.

There are three main issues:

First, Reinhart and Rogoff selectively exclude years of high debt and average growth. Second, they use a debatable method to weight the countries. Third, there also appears to be a coding error that excludes high-debt and average-growth countries. All three bias in favor of their result, and without them you don’t get their controversial result.

Since the HAP paper has received publicity there has been a huge amount of debate. Reinhart and Rogoff have issued two responses. The second is rather more conciliatory than the first, inasmuch as it concedes the point that there was a data processing error. But broadly-speaking their responses defend their existing position and argue that even if you correct the calculations their key finding still stands.

Some commentators have stated, over the last day or so, that they never found the Reinhart and Rogoff result very convincing in the first place, particularly the idea that there was an identifiable threshold when debt hits 90% of GDP. If you graph the data one of the most striking things is that the data points are all over the place and therefore any association between debt and growth is decidedly sketchy. Also, whatever association can be found is driven by a few countries with particular characteristics and circumstances.

If we’ve all been convinced all along that these results were so flaky, it is curious that they have achieved such prominence, rather than being rubbished or ignored. But that is by the by.

The reason all this is important, rather than just being a bit of a spat over academic methodology, is that the Reinhart and Rogoff result has moved beyond academia to be hugely influential in policy terms. Reinhart and Rogoff are both at Harvard and are well-placed within academic economics, but they have also worked at the IMF so are plugged in to economic policy debates at global level. What they say has influence.

Their result has been absolutely central in buttressing arguments in favour of austerity. Azizonomics yesterday provided a selection of quotes from senior political figures invoking the Reinhart and Rogoff result as offering scientific justification for the benefits of fiscal consolidation.

There is rather more debate over whether Reinhart and Rogoff themselves see their results as justifying austerity policy. This in part turns on the question of causation. Reinhart and Rogoff have defended their work with the argument that they were only pointing out an association between debt and economic growth, not suggesting that high levels of debt causes low economic growth. Indeed, it is highly likely that causation also runs the other way – hitting a period of low growth leads to ballooning public debt.

That might well formally be true of the position Reinhart and Rogoff adopt, but their statements on the matter have not been a model of clarity or caveat. Commentators have pointed to various of their statements over the years which might be construed without too much difficulty as suggesting that Reinhart and Rogoff thought not only that the relationship was causal but also that austerity was the way to go, even if they didn’t quite set out their position in those terms.

Whether it is possible to track down clear statements made by Reinhart and Rogoff that implicate them in supporting austerity isn’t the main issue. There are at least two points that are more important.

First, if we accept their formal position at face value and conclude that all they have done is identified associations between variables, without making any sort of causal claim, then really you’ve only just started on the analysis. In fact, unless you are going to claim – or imply – causation the finding isn’t all that illuminating. As Noah Smith observes:

I think that books like This Time Is Different are merely jumping-off points for an investigation of that hypothesis; they do not constitute any kind of proof. Naturalism is where understanding of the world begins, but not where it ends.

It may be that similar looking patterns and associations between variables in different countries at different points in history are driven by entirely different causal processes. Until you’ve grappled with causation then you’ve not got very far. You can say very little.

The second point about this episode is that it takes me back to some of the arguments advanced by George DeMartino in The Economist’s Oath, on the need for a professional ethics for economics. DeMartino’s position is that economists’ traditional resistance to the conscious cultivation of a professional ethic is unjustified and unacceptable. As noted in the epigraph, he argues that:

… the actions of economists today bear on the life chances of the world’s population far more substantially than do the actions of members of most other professions. (p98)

The Reinhart and Rogoff incident would seem to represent a paradigmatic example.

Now, the conventional response to this would most probably be that Reinhart and Rogoff were just producing objective analysis in their role as economic scientists. They cannot be held responsible for the way in which that analysis was taken up, used and abused to further particular political agendas.

DeMartino’s point is that this stance is inadequate. It is an abdication of responsibility. It is not acceptable to claim that one can only be held responsible for the intended consequences of one’s actions when “[u]nintended consequences are sometimes predictable, probable, and/or significant” (p92).

Reinhart and Rogoff may wish to style their work as objective analysis that is tentative and to be treated with caution. But it was entirely predictable and probable that producing such work in 2009 and 2010 was going to give political actors further ammunition with which to justify the austerity drive that was already under way. And the consequences of this austerity drive have been hugely significant for populations all over the world. Many people have been reduced to destitution as a result. People have died. The institutional structures of a number of countries have been deeply, possibly fatally, undermined.

Are Reinhart and Rogoff responsible for all this? Clearly not. Did they have any responsibility to think through how their work might have been put to use? The conventional answer to that is no. The more responsible answer would be yes. It may have led the work to be presented in different ways. It may have led to conclusions and caveats to be reframed in ways that reduced the possibility of the work being abused in pursue of a political agenda.

And if Reinhart and Rogoff do not wish to align themselves with the austerity agenda, and feel that the invocation of their work in this context is misplaced, then should they not have responded before now? A comment from Colander and colleagues is entirely apposite:

Researchers have an ethical responsibility to point out to the public when the tool that they developed is misused. It is the responsibility of the researcher to make clear from the outset the limitations and underlying assumptions of his model and warn of the dangers of their mechanical application.

It would be interesting to reflect on the extent to which Reinhart and Rogoff had been actively seeking to highlight the misuse of their findings as a justification for austerity in the policy process, before everything kicked off this week.

Few people outside of the world of economics would contest that economists and economic thinking have significant impact upon the lives and livelihoods of the world’s population. This would tend to suggest that deep ethical deliberation is necessary in order to ensure that such power is wielded with great care. Yet, economics has so far been pretty resolute in its rejection of the relevance of ethical reflection.

Perhaps the Reinhart and Rogoff incident will rekindle the debate. Perhaps it will convince some of the doubters that simply denying an intimate connection between academic economics and practical action in no way means the connection ceases to exist. And therefore attention to ethics is a key professional responsibility.

Image: Cover of Reinhart and Rogoff (2009), from the author’s collection.