Reliable referees: a rare commodity

By Catarina Dutilh Novaes

In recent weeks, there has been much discussion on journal editorial practices at a number of philosophy blogs. Daily Nous ran an interesting post where different journal editors described (with varying degrees of detail) their editorial practices; many agree that the triple-anonymous system has a number of advantages and, when possible, should be adopted.* (And please, let us just stop calling it ‘triple-blind’ or ‘double-blind’, given that there is a perfectly suitable alternative!) Jonathan Ichikawa, however, pointed out (based on his experience with Phil Studies) that we must not take it for granted that a journal’s stated editorial policies are always de facto implemented. Jonathan (correctly, to my mind) defends the view that it is not desirable for a journal editor to act as a (let alone the sole) referee for a submission.

With this post, I want to bring up for discussion what I think is one of the main issues with the peer-reviewing system (I’ve expressed other reservations before: here, here, and here), namely the extreme difficulties journal editors encounter at finding competent referees willing to take up new assignments. Until two years ago, my experience with the peer-review system was restricted to the role of author (and I, as everybody else, got very frustrated with the months and months it often took journals to handle my submissions) and the role of referee (and I, as so many others, got very frustrated with the constant outpour of referee requests reaching my inbox). Two years ago I became one of the editors of the Review of Symbolic Logic, and thus acquired a third perspective, that of the journal editor. I can confirm that it is one of the most thankless jobs I’ve ever had.

I am now caught between authors who are (rightly) eager for good referee reports produced in a timely fashion (some of whom are junior colleagues who desperately need the publications), and over-worked referees who simply cannot say ‘yes’ to each referee request they receive. It is not uncommon that I have to contact at least 10 people before securing two referee reports for a paper. As I see it, my main mission as an editor is to ensure that authors get high-quality feedback, such that they will have gotten something out of it even if the paper is rejected. But reliable, thorough referees are a very rare, precious commodity, which must not be overused. Good referees are usually very competent people who already have a lot on their plates. One thing I do to circumvent the issue is to contact more junior but very capable people (say, at post-doc level) to act as referees, the thought being that they will be slightly less swamped with other obligations than more senior colleagues. But often what one needs is an experienced pair of eyes, and then we are back to the over-stretched senior colleagues…

The most frustrating aspect of the whole process are the people who do not even bother to reply to a referee request. Ok, this is one way of saying ‘no’, but besides being rude and uncollegial (after all, we all need the peer-reviewing system to work minimally well!), it holds things back substantially, as I usually wait at least a week or two before moving on to the next possible referee. So, my plea to all of you reading this: if you receive a referee request, please reply within a day or two; if you cannot accept the assignment, please suggest alternative names.

However, getting people to accept a referee request is only the beginning of the story; it remains to be seen whether the referee will then indeed produce a report (many quit without notice), whether the report will be produced timely, and whether it will be a useful, informative report. At times, when it’s been months since the initial submission and there are still no clear prospects of at least two reliable reports being produced, it may well happen that the editor will choose to produce a report herself; it is not ideal, but not uncommon. (Of course, this is something very different from Jonathan Ichikawa’s experience.)

So the point of this post is to reveal yet another aspect in which the peer-review system is very fragile. My impression is that authors, especially the more junior ones, find it hard to understand why it often takes so long before they hear back from editors on their submissions. Although it is definitely the case that some journals simply do not handle matters in an efficient, professional way, hardworking, serious editors face the huge challenge of getting referees to produce helpful reports in a timely fashion. In comments, I welcome suggestions from (more seasoned) journal editors on how to deal with this challenge, including the emotional stress involved in being stuck between eager authors and over-worked referees. (Seriously, sometimes this keeps me awake at night…)

----------------------------------------------------

* Notice however that in many disciplines, for example mathematics and physics, anonymous refereeing is not the norm. Unfortunately, the RSL does not have a policy of mandatory anonymous submissions, so it is not even double-anonymous by default (the author chooses whether to submit anonymously or not).