Another look at the refereeing process

Christian Seidl (University of Kiel), Ulrich Schmidt (University of Hanover) and Peter Grösche (RWE Essen) report some rather disturbing results from their survey of economists’ experience of the journal refereeing process.

At the end of 2001 and at the beginning of 2002 we addressed some 4,500 economists and some 15,000 related persons (professionals, managers) by e-mail asking them for their online responses to seven questions concerning their experience with the refereeing process of economics journals. 630 respondents embarked on the first question, 551 of which participated in our survey through to the seventh question.

We used two lists of journals, firstly 76 journals which we proposed (the journals of the Diamond list enlarged by 49 journals taken off the A and B categories of the list of the Association of the Universities in the Netherlands), and secondly journals suggested by the respondents. We report results only for journals with at least five responses, which yielded a list of 106 journals for the results of our survey. The design of our survey and the detailed results can be downloaded from the homepage:

http://www.wiso.uni-kiel.de/vwlinstitute/ifs/
chair/peerreview.php

A comprehensive report will be published in Estudios de Economía Aplicada, Vol. 23/3 (2005).

The following information serves only to whet the reader’s appetite and to encourage a look at this homepage and the more comprehensive article.

Although there are only a few studies which investigate refereeing processes in economics journals, much work has been done scattered around the various scientific disciplines at large. Among this work, the most spectacular experiment was conducted by Peters and Ceci for psychology journals [Behavioral and Brain Sciences 5, 1982]. They resubmitted twelve already published papers under fictitious names to prestigious psychology journals. Only three of them were detected, eight of the rest were rejected, and only one was accepted. As to the rejected papers, no referee cited a lack of novelty of the respective findings. This and related instances are surveyed in the more comprehensive paper.

Question 1 runs thus: ‘After submission of your paper, how long did it take on average to get a reply other than just a confirmation that your paper had been received?’

The shortest spell is reported for the Quarterly Journal of Economics with 0.6 weeks, followed by Economic Systems with 8.7 weeks. The longest spell is reported for Public Finance with 61.5 weeks. Most journals figure between 15 and 25 weeks, the more prestigious generally taking longer than the less prestigious. This contrasts markedly with a science like physics, where the length of the refereeing processes is between 4 and 8 weeks [cf., e.g., the letter by Ormerod in No. 117 of the RES Newsletter]. The Quarterly Journal of Economics is obviously an outlier. As the speediest referee process takes hardly less than two weeks (allowing for the time needed to forward the letters with the manuscripts), and given the high rejection rate of this journal, this indicates that the managing editor(s) of the Quarterly Journal of Economics reject(s) most of the submitted manuscripts without ever having consulted a single referee as to rejection or acceptance of a paper.

Question 2 asks: ‘How many referee reports did you receive on average?’

Some 22 per cent of the journals provide at least two referee reports on average, some 78 per cent provide at least 1.5 referee reports on average, and only three journals supply less than one referee report on average, among them Economics Letters as the only journal from the Diamond list. This means that this journal rejects some papers without any referee report at all after a mean spell of 14.8 weeks. Even the Quarterly Journal of Economics supports 1.25 referee reports on average [thus, the editor(s) quick rejection is taken as a referee report by respondents]. Only two journals from the Diamond list provide more than two referee reports on average, namely, Econometrica and Economic Inquiry.


Question 3 runs thus: ‘How many papers did you submit to this journal and how many papers were accepted?’

The list of journals is led by Empirica with a mean subjective acceptance rate of 100 per cent. The last journal in this list is the Quarterly Journal of Economics with a mean subjective acceptance rate of 7 per cent. Only three journals of the Diamond list have a mean subjective acceptance rate higher than 50 per cent, the Journal of Mathematical Economics, Economic Inquiry, and the Journal of Econometrics. When juxtaposing the subjective acceptance rates with some acceptance rates as reported by the editors of some journals, we observed that our data are biased in the direction of higher subjective acceptance rates. This may be due to a trend effect [true acceptance rates fell with the lapse of time, while respondents amalgamate past with present experience leading to their memory of higher acceptance rates], a selection effect [more successful scholars felt more attracted by our survey than less successful ones], or to a positive sensation effect [the sensation of a paper acceptance looms larger in a respondent’s memory than a paper rejection which fades away more quickly]. Unfortunately, we cannot control for these effects, but notice that they are at work. On the other hand, a survey such as ours is in danger of attracting primarily frustrated respondents who wish to deal a blow to those journals which they consider to have treated them unfairly. The responses to this question show that this is certainly not the case.

The rest of the questionnaire deals with evaluative questions. To deal with them we used seven-point Likert scales and coded the worst possible response by a 0, and the best by a 6.

Question 4 asks: ‘Were the referee reports competent?’

Surprisingly enough, it seems that the competence of referee reports is not positively correlated with the reputation of a journal. Both the journals of the Diamond list and of the journals proposed by us neither bunch at the upper nor at the lower end. They appear to be rather evenly distributed among the ranks. For instance, considering the mean rank of 44, 14 journals of the Diamond list rank ahead of, and 12 rank behind the mean rank. Moreover, prima facie there seems to be no significant correlation with the rejection rates of the respective journals. However, more detailed investigations have to be deferred to subsequent analyses of the data.

Question 5 runs thus: ‘Did the decision of the editor match the referee report?’

Generally it seems that most editors comply with the recommendations of the referee reports. As regards the journals’ rankings, some prestigious journals with, as it seems, high backlogs of manuscripts, such as the American Economic Review, the Journal of Political Economy, the Review of Economics and Statistics, the Journal of Public Economics, the Economic Journal, the Quarterly Journal of Economics, and some others, rank in the rear, possibly because of the editors’ zeal to curb the growth of the queues of papers waiting to be published. However, a good match of managing editors’ decisions with the recommendations of the referees need not be a proxy for fair refereeing because referee hostility or incompetence may be but insufficiently monitored by the editor. Notwithstanding that referees’ recommendations may be highly contradictory, editors tend to reject papers if only one referee recommends rejection irrespective of other referees’ judgments.

Question 6 asks: ‘Were the referee reports carefully done?’

The responses to this question are not flattering, in particular not for the most prestigious journals. As many as 66 out of 106 journals have scores less than 4. Out of the 40 journals which score 4 or better than 4, only 6 belong to the 26 remaining journals of the Diamond list, viz. the Journal of Mathematical Economics, Econometrica, the Journal of Economic Literature, the Journal of Econometrics, the Review of Economic Studies, and Economic Inquiry. Many reputed journals are among the 20 last ranks of carefully done referee reports, viz. the European Economic Review, the Oxford Economic Papers, Economics Letters, the Quarterly Journal of Economics, Economica, the Journal of Political Economy, and the Journal of Financial Economics.

Question 7 prompts respondents to summarize: ‘How was your overall satisfaction with the procedure of paper submission to the respective journal?’

The responses to this question are disappointing for the prestigious journals. Out of 106 journals, only 36 score 4 or better, among them only five journals from the Diamond list, namely, the Journal of Economic Literature, the Journal of Mathematical Economics, Economic Inquiry, the Journal of Econometrics, and the Journal of Law and Economics. Out of the 106 journals, 23 score worse than 3, among them eight journals from the Diamond list, to wit, the European Economic Review, Economica, the Quarterly Journal of Economics, the Journal of Labor Economics, the Economic Journal, the Journal of Monetary Economics, the Journal of Financial Economics, and the Journal of Political Economy.

Thus, we can concisely summarize our findings in the following items:

• Some journals which pretend to be refereed journals are in fact only semi-refereed journals.

• The most prestigious journals (e.g., those from the Diamond list) are not the leading ones in the evaluation questions 4-7.

• The mean spell of reply is about 20 weeks, which is four times longer than for leading physics journals. The most prestigious journals (20 out of the remaining 26 of the Diamond list) need more than 20 weeks to make a decision.

• Concerning acceptance rates, our data are biased by a trend effect, a selection effect, and a positive sensation effect, which may account for an upward bias in the responses to questions 4-7.

• Prestigious journals have higher rejection rates.

• Competence of referee reports is largely evenly spread.

• Compliance of editors' decisions and referees' recommendations is, in general, quite high. This may imply that editors have insufficient control of referee hostility or incompetence.

• Only few prestigious journals score 4 or more when it comes to carefulness of referee reports.

• The overall satisfaction with the referee process leaves the prestigious journals of the Diamond list lagging behind. Only 5 (among 36) score 4 or better, whereas 8 (among 23) score worse than 3.

Page Options