The state and prospects of academic peer review

(A PDF version of this post is available on HAL.)

Peer review plays a crucial role in the communication and management of research. How well is it working? How could it be improved? The answers depend on whom you ask. Here they are debated by seven characters with different points of view.

1. Osman: Now that everyone is here, let us start this debate on academic peer review: should it be strengthened, overhauled, or abandoned? When I arrived, Clementine was already in the room, reviewing a paper and complaining about it. This leads to my first question: is peer review taking too much time and effort? 

2. Clementine: I was indeed complaining, but like most colleagues I realize that peer review is an essential part of our work: this is how we validate each other’s work, this is how we assess its interest and likely impact, and this is how we collectively improve how papers are written. Now, the problem is that so many papers get submitted nowadays, it becomes hard to do a proper job on all of them. Even after declining many invitations to review, I have to be pragmatic and focus on the main question: is the submitted paper worth publishing in that journal? If the paper is obviously good work by well-known people, it can be waved through with minimal scrutiny. If it is mediocre work, the rejection should be short and final – I do not want to get dragged into a technical debate with the authors. If however I am unsure whether the paper should be accepted after a quick first reading, then I need to scrutinize it, and to find a good reason for either rejecting it, or accepting it after some improvements. 

3. Deirdre: So you spend more time reviewing second-rate works than really interesting papers? I am glad I stopped reviewing years ago. 

Reforming research assessment: nice declarations, little action?

There seems to be a consensus among universities and research funders that research assessment should not be based on crude quantitative metrics, such as: numbers of articles, numbers of citations, journal impact factors, the h-index, etc. The 2012 San Francisco Declaration on Research Assessment (DORA) formulates principles which could greatly improve research assessment if they were applied, although I would argue that the DORA is misguided in its recommendations to authors. The DORA has been signed by thousands of organizations: just for France this includes the Academy of Sciences, CNRS and HCERES. More recently, the European Commission has issued a report called Towards a reform of the research assessment system, which deals with the same issues and promotes similar principles.

Since the same principles have to be reiterated 9 years later, you may think that little has changed in all that time. And you would be largely right. Significant reforms of research assessment in individual organizations are so rare that they are newsworthy. And some universities are denounced for taking actions that directly contradict the principles they have officially endorsed.