Publish (and be saved?)
20 Jan 2021
Publications have long been one important method of evaluating academic performance. The pros and cons of this, as well as strengths and weaknesses in the procedure, have often been discussed in these pages.1 A recently published eBook2 examines these issues with a series of short essays focusing on how economists publish their research and measure academic success. While there is much to be proud of about the state of the economics profession, the chapters in the eBook suggest there is still work to be done to make economics more open and inclusive and the publication process fairer and more efficient.
The eBook is divided into six sections covering: measuring success, citation patterns, publication lags, social ties and co-authorship, the race problem in economics (with a specific focus on US academia), and how the Covid-19 pandemic has impacted economic research. The last chapter presents a short review of topics which were not covered in the volume and describes a number of papers which complement some of the other chapter of the eBook.
The eBook starts with a chapter by Daniel Hamermesh, who presents a critical evaluation of how economists measure success. The central message of the chapter is that we should not rank individual scholars’ achievements by summary measures, such as where the research is published or the institution with which a scholar is affiliated. This contrasts with the profession’s tendency to heavily discount papers published in less prestigious outlets. Research by Nattavudh Powdthavee, Yohanes Riyanto, and Jack Knetsch shows that this discounting can be so steep as to give negative value to publications in lower-rated journals. James Heckman and Sidharth Moktan study the tyranny of the ‘Top Five’ economics journals and suggest that reliance on highly ranked journals as a screening device raises serious concerns both because of its weak empirical support (if judged by its ability to produce impactful papers) and because of the risk of clientele effects surrounding these journals and their editors. They suggest that the profession should start a conversation on alternatives for judging the quality of research, ideally with a shift from the current publications-based system of deciding tenure to a system that emphasises departmental peer-review of a candidate’s work.
The second section of the eBook focuses on citation patterns. Maria Victoria Anauati, Sebastian Galiani and Ramiro Gálvez study citation patterns of more than 6,000 economics research articles published in different types of journals and find that there is a strong overlap in the distribution of received citations across tiers and that the influence (in term of citations) of Top Five articles is overestimated. The same authors use a dataset consisting of more than 5 million citations to nearly 60,000 articles spanning 12 disciplines, from astronomy to statistics, to study patterns of citation ageing, showing that there are large differences in citation ageing across disciplines but also across fields within economics. They conclude that although citation counts can be a valuable tool for assessing the impact of academic research, there are caveats with ‘one-size-fits-all’ yardsticks and that citation counts should be adjusted by field- and discipline-specific factors.
Another possible issue with using citation counts as a measure of academic productivity is that citation patterns may be driven by strategic considerations. Joshua Aizenman and Kenneth Kletzer study the potential importance of strategic citations by focusing on premature deaths by highly cited economists. Their findings support the view that citations are not a pure measure of scientific impact and may be affected by strategic considerations. The section concludes with a chapter that studies whether certain journals are particularly important for policy institutions. Raphael Auer and Christian Zimmerman focus on central bank publications and show that different journals have different audiences and that economists should not be evaluated on the basis of one-size-fits-all rankings.
In Section 3 of the book John Conley, Mario Crucini, Robert Driskill, and Ali Sina Önder provide evidence of a decrease in publications by young scholars which is especially marked for those who are not at the top of their cohort. They conclude that the profession should be careful when evaluating people for tenure and promotions as the rules of the game have changed. In the next chapter, Daniel Hamermesh shows that through the 1990s about half of the papers published in top journals were by authors under the age of 35 and almost nobody over the age of 50 published a paper in these outlets. Things have since changed, however. In 2011, authors under-35 accounted for only one-third of papers published in top journals and over-50 authors accounted for nearly 20 per cent.
The next two chapters of this section use confidential data to evaluate possible strategies to speed up the publication process. Ivan Cherkashin, Svetlana Demidova, Susumu Imai, and Kala Krishna study the handling of more than 3,000 papers submitted to the Journal of International Economics (JIE) between 1995 and 2004. They note that during that period, JIE had high ‘type 2 errors’ (7 per cent of published papers have no citations at all) and low ‘type 1 errors’ (very few papers rejected by the JIE were accepted at better-ranked journals) and that being well-connected increased the likelihood of acceptance. Raj Chetty, Emmanuel Saez, and László Sándor discuss the results of an experiment with 1,500 referees at the Journal of Public Economics. They conclude that a combination of shorter deadlines, cash incentives, and social incentives could play an important role in improving the refereeing process.
Social ties and co-authorship patterns
Section 4 of the eBook focuses on social ties and co-authorship patterns. John O’Hagan and Lukas Kuld show that, between 1996 and 2014, the share of solo-authored published economics papers dropped from 50 per cent to 25 per cent, with most of the rise in multi-authored papers accounted for by papers with more than two authors. The authors stress the need for a discussion on how tenure and promotion committees should evaluate contributions to co-authored papers. This is also the focus of the chapter by Stan Liebowitz, who suggests that the system adopted by most economics departments promotes false authorship and may penalise honest researchers. Tommaso Colussi studies the role of connections in the publication process and shows that there are important benefits associated with being connected to an editor. By studying how social ties affect the number of citations, he finds that papers authored by an editor’s former PhD students increase the number of citations but that this positive effect does not apply to articles authored by other types of connected scholars. The last chapter focuses on the geography of published economics research. Jishnu Das and Quy-Toan Do show that low-income countries are heavily under-represented in economic research and that there is a US bias in top economics journals — the probability of publication is much larger for papers on the US relative to other countries, and this pattern persists even after controlling for author ‘quality’. The authors conclude that the correlation between publication patterns and GDP per capita supports Bardhan’s (2003) concerns about a possible misallocation of talent across research institutions with limited incentives to focus on small and poor countries.
The race problem in economics
Recent events in the US have sparked an intense debate on the race problem in economics. Section 5 of the eBook discusses this issue with a focus on US academia. Trevon Logan and Samuel Myers point out that most modern economists are not trained to address questions of ‘structural racism’ or ‘systemic racism’ and that this has led to the structural determinants of current racial inequalities being ignored. They conclude with a set of suggestions aimed at addressing the problem of marginalisation of race in economics. In the next chapter, Gregory Price and Rhonda Sharpe focus on the lack of African American economic professors in US universities. They show that while the number of African Americans who receive a PhD in economics has increased over time, there has been no comparable increase of black economic professors in US universities. The authors suggest that a penalty system for institutions that do not broaden participation could help promote initiatives that foster an inclusive profession.
Economic research in the time of Covid-19
The last section of the eBook discusses economic research in the time of Covid-19. Noriko Amano-Patiño, Elisa Faraglia, Chryssi Giannitsarou, and Zeina Hasna ask who is doing research at the time of Covid-19. The authors find that while there has been a surge of papers on the Covid-19 shock by senior male economists, the productivity of female and, more generally, mid-career research economists have been negatively affected by the lockdown measures. In the next chapter, John Cochrane asks who will publish all the papers on Covid-19 that are being produced right now. He points to the inefficiency of the current publications system and states that ‘economists believe in markets, but not for papers’. He suggests that a possible solution to address some of the issues that plague the publication process in economics is to allow for simultaneous submissions and thus create a market which will allow papers to be better matched to journals. As a response to long publication lags in economics, at the end of March, CEPR issued a call for papers for Covid Economics, Vetted and Real-Time Papers. In the last chapter of this section, Charles Wyplosz described his experience as editor of Covid Economics.
Our reading of the evidence is that while there is much to be proud of about the state of the economics profession, there is still work to be done to make economics more open and inclusive and the publication process fairer and more efficient. Promoting stronger competition among journals could help in dealing with many, though not all, of the issues highlighted in the eBook. One way to achieve this is to coordinate a higher weight in the assessment of economists’ career success to high quality, non-Top Five, journals.
Bardhan, P (2003), "Journal Publication in Economics: A View from the Periphery", Economic Journal 113(488): 332-7.
Lundberg, S (2020), Women in Economics, a CEPR Press eBook.
1. Most recently Erin Hengel and Eunyoung Moon, ‘Gender and Quality at Top Economics journals’, Newsletter no. 190, July 2020 but see also remarks by James Heckman in ‘2019 Annual Conference Repor’ and ‘Do Economists Deserve Better?’, Newsletter no. 186, July 2019.
2. Publishing and evaluating success in economics: A new eBook, eds. Sebastian Galiani and Ugo Panizza. The book can be downloaded from https://voxeu.org/content/publishing-and-measuring-success-economics. This article draws on the summary of the book published at https://voxeu.org/article/publishing-and-evaluating-success-economics-new-ebook (accessed 10.11.20).