Objective Data on World-Leading Research

Following the conclusions of the recent Research Assessment Exercise, Andrew Oswald uses citations to judge the impact of UK economic research. He finds that the UK produced 10 per cent of the influential research in economics over the RAE period and some of this came from departments not normally thought of as amongst the elite.

“4* – Quality that is world-leading in terms of originality, significance and rigour…comparable to the best work in the field or sub-field whether conducted in the UK or elsewhere. Such work … has become, or is likely to become, a primary point of reference in its field or sub-field.”
                                   -  Research Assessment Exercise: RAE 2008 www.rae.ac.uk

Our discipline was ranked in the latest Research Assessment Exercise as the United Kingdom’s best subject. However, journalists and others have asked me if the result is believable. This article is concerned with that issue.

I believe that as academics we suffer from -- and here I include myself -- the following biases:

(i) we tend to overestimate the importance of our own department;
(ii) we tend to overestimate the importance of our own sub-field of economics;
(iii) we tend to be badly informed about how influential the articles can be from a range of journals (it has become common, in my 30-year professional lifetime, to hear people focus on a tiny number of journal ‘labels’ per se, and even sometimes to speak as though a publication in a place like the American Economic Review is an end in itself rather than mattering for its content or its contribution to human welfare);
(iv) we tend to be poorly informed about the latest flow of research and excessively influenced by out-of-date stock variables (such as, for example, the long-standing reputation of another department, or of a particular economist).

I therefore collected data on the world’s most-cited articles over the period of the RAE, namely, from 2001-2008. My aim is to try to make operational the kind of 4* concept encapsulated in the quote at the beginning.

The method
I use data from the ISI Web of Science. Citations are taken here as a proxy for the objective quality of an article -- measured with the benefit of hindsight. See for instance Goodall (2009) and Hudson (2007).

This country has produced some of the most-cited articles in the world in a number of important field journals. It has also been the source of some of the most influential articles in the AER, Review of Economic Studies, and Econometrica. But it did not do well in certain other journals, especially the Quarterly Journal of Economics and Journal of Political Economy.

I take the journals listed by the Helpman Committee in the recent ESRC Benchmarking Report on Economics in the United Kingdom. There is little dispute that these are important journals. There are 22 of them. They are divided into 9 general journals and 13 field journals. I adopt this list simply because it is the one chosen by Helpman and thus for this particular study helps avoid suggestions that I was consciously or unconsciously biased in my selection.

For each journal, I searched on articles published between January 2001 and December 2008. I searched as carefully as possible by hand through the articles for the UK-based ones.1 The problem with not doing this by hand is that any mechanical search on England will throw up articles that are not truly from England – such as those authored by Americans with an honorary affiliation to the CEPR in London.2

It is perhaps worth emphasising that there is evidence that early citations numbers to an article are a good indicator of long-run citations numbers. This is shown, for example, in Adams (2005). In other words, if an article acquires few citations early on it is extremely unusual -- of course there are occasional exceptions -- for it ever to acquire a high number.

The key data are set out in the table below. It tells us the influential recent articles from UK economics and, in particular, where they lie in a world-ranking of influence.

To try to adjust for the fact that some journals attract a particularly large quantity of good articles, I allow different journals to have different numbers of articles in the key table -- 50 for the American Economic Review, 10 for the Economic Journal, and so on. These cut-offs were chosen to try to be fair to the different journals. I attempted approximately to equalize the numbers of citations of the marginal excluded article.

All this is designed to get at the principle of “4*… a primary reference in its field”.

To read the table, the procedure is the following. Take the numbers in the top row as an example. These tell us that if we look at the 50 most-cited articles published by all countries in the American Economic Review over the 2001-2008 period then the UK was the source of four of these out of the fifty. The UK ones were the 12th most-cited article, the 32nd, 35th, and 38th. Moreover, these four articles came, respectively, from Warwick and LSE on the first, LSE on the second, Cambridge on the third, and LSE on the fourth. It can be seen from the table that the UK attained the top slot in the Rand Journal, the International Economic Review, Journal of Econometrics, and Journal of Public Economics; 2nd in the Journal of Health Economics; 3rd in the Journal of Development Economics; and 4th in the Journal of International Economics, the Journal of Monetary Economics, and the Journal of Urban Economics.

This is a substantial achievement for the United Kingdom. Nevertheless, although I do not report the full data, the UK numbers are far behind those for the (obviously much larger) United States.

It should be noted that my measure here assigns full weight to a jointly authored article, so that a tri-authored Article Y by economists from Universities A, B, C would see each of those universities credited above.

In the full version, Oswald (2009), I try to present various checks on the calculations, and some results for other European countries. I also attempt to do a simple cross-check against the quality of other disciplines.

The United Kingdom comes out reasonably well on my criterion. Over the RAE period, it produced 10% of the influential research in economics.

As Table 1 reveals, one quarter of these objectively important UK articles emanate wholly or partly from departments of economics not normally considered to be in the top half-dozen in the country. Therefore, outstanding work -- a set of genuinely world-leading articles -- comes from a wide range of sources (literally speaking, 21 different universities in Table 1) and thus by implication that it might be a mistake to concentrate funding narrowly3 on a tiny number of universities.

Perhaps economists, whose discipline has long espoused competition and opposed monopolies, should not be surprised at the idea that pluralism can encourage iconoclasm. Perhaps economists should take a lead in UK academia in arguing against a growing modern concern with ‘top’ departments, ‘top’ journals, and other monopoly-creating devices.

Notes and references:

1. It is possible that some errors of counting remain, but I hope they are slight enough not to lead to incorrect conclusions.

2. It has come to light recently that there are problems with the data from Evidence Ltd in their work for the Helpman Report (2008). Tables 1 and 2 of Helpman (2008) appear to overstate the true numbers of articles by UK authors. The main source of the error is that CEPR-affiliated Americans are wrongly classified as UK economists. Evidence Ltd has not released to me their data or work-sheets. Hence I cannot be sure about the size of the errors. But for the Journal of Monetary Economics, for example, the Helpman Report states that between 2002 and 2006 approximately 12% of articles in JME came from UK authors. My own examination of the raw data, in so far as I can check their calculations independently, suggests that the true number for the JME is probably closer to 6%.

3. Any economist, however, would argue that it is the marginal productivity of research funding that matters. Public discussion, by contrast, is almost always about average productivities.

Adams, J. 2005. Early citation counts correlate with accumulated impact. Scientometrics, 63 (3), 567-581.
Goodall, Amanda H. 2009. Socrates in the boardroom: Why research universities should be led by top scholars. Princeton University Press. Forthcoming.
Helpman, E et al. 2008. ESRC International benchmarking study of UK economics. ESRC. Swindon.
Hudson, J. 2007. Be known by the company you keep: Citations - quality or chance? Scientometrics 71(2), 231-238.
Oswald, A J,. 2009. World-leading research and its measurement. Warwick Economic Research Papers, #887.

Page Options