The state and prospects of academic peer review

(A PDF version of this post is available on HAL.)

Peer review plays a crucial role in the communication and management of research. How well is it working? How could it be improved? The answers depend on whom you ask. Here they are debated by seven characters with different points of view.

1. Osman: Now that everyone is here, let us start this debate on academic peer review: should it be strengthened, overhauled, or abandoned? When I arrived, Clementine was already in the room, reviewing a paper and complaining about it. This leads to my first question: is peer review taking too much time and effort? 

2. Clementine: I was indeed complaining, but like most colleagues I realize that peer review is an essential part of our work: this is how we validate each other’s work, this is how we assess its interest and likely impact, and this is how we collectively improve how papers are written. Now, the problem is that so many papers get submitted nowadays, it becomes hard to do a proper job on all of them. Even after declining many invitations to review, I have to be pragmatic and focus on the main question: is the submitted paper worth publishing in that journal? If the paper is obviously good work by well-known people, it can be waved through with minimal scrutiny. If it is mediocre work, the rejection should be short and final – I do not want to get dragged into a technical debate with the authors. If however I am unsure whether the paper should be accepted after a quick first reading, then I need to scrutinize it, and to find a good reason for either rejecting it, or accepting it after some improvements. 

3. Deirdre: So you spend more time reviewing second-rate works than really interesting papers? I am glad I stopped reviewing years ago. 

Reforming research assessment: nice declarations, little action?

There seems to be a consensus among universities and research funders that research assessment should not be based on crude quantitative metrics, such as: numbers of articles, numbers of citations, journal impact factors, the h-index, etc. The 2012 San Francisco Declaration on Research Assessment (DORA) formulates principles which could greatly improve research assessment if they were applied, although I would argue that the DORA is misguided in its recommendations to authors. The DORA has been signed by thousands of organizations: just for France this includes the Academy of Sciences, CNRS and HCERES. More recently, the European Commission has issued a report called Towards a reform of the research assessment system, which deals with the same issues and promotes similar principles.

Since the same principles have to be reiterated 9 years later, you may think that little has changed in all that time. And you would be largely right. Significant reforms of research assessment in individual organizations are so rare that they are newsworthy. And some universities are denounced for taking actions that directly contradict the principles they have officially endorsed.

Perverse bibliometrics: the case of patents

Bibliometrics, the counting of publications and citations, is being used for evaluating researchers, research institutions, and academic journals. But simple bibliometric indicators can be gamed, and complex indicators lack transparency. No known indicator avoids these two problems, while some indicators (such as the journal impact factor) manage to have both. As a result, the misuse of bibliometrics has been widely denounced.

In spite of these problems with bibliometrics, someone had the idea to do bibliometrics with patents, in order to rank research institutions. The result is Reuters' list of the world's most innovative research institutions, which is topped by the Alternative Energies and Atomic Energy Commission (CEA). The methodology for establishing the list is not known in detail, but we do know that it is involves 10 different criterions, and is mainly based on the numbers of patents and citations thereof. 

Rating scientific articles: why and how?

Assuming we need to assess the quality of individual scientific articles, there are two broad approaches:

• the quantitative approach of counting citations and/or computing alternative metrics,
• the qualitative approach of conducting peer review, and giving quality tokens to articles. A quality token can be publication in a journal, or a grade.

The quantitative approach has the fundamental flaw of being an indirect way of assessing quality. And it has undesirable side effects, such as the misuse of citations for assigning credit, rather than for helping readers. In this post I will deal with the qualitative approach.

Giving grades, implicit or explicit.

Assuming some form of peer review is conducted, how can the results be concisely summarized?

Academic publishing reform 3. Institutional evolutions

Institutions such as universities and funding bodies play a major role in research policy, and therefore potentially in scientific publishing. There are many things institutions could do to improve the publishing system, such as:

• renouncing the use of impact factors and citation counts,
• evaluating researcher's works on their merits, and not on whether or where they are published,
• lauching and supporting innovative journals,
• stopping paying Big Deal subscriptions to publishers, and facilitating the buying, hoarding and sharing of individual articles by researchers,
• ordering the researchers they employ not to work as referees or editors for the worst publishers.

But institutions seldom if ever do these things, and since I want to discuss only observed behaviour, the discussion will be rather brief. Actually, institutions are sometimes more inclined to embrace pernicious innovations, such as the h-index or the publishers' Big Deals, than genuine progresses, especially when the pernicious innovations favour the bureaucracy's natural tendency to increase its own size and power.

For example, a common misconception is that large institutions are better than small ones at dealing with big publishers. Actually, given the all-or-nothing nature of the proposed Big Deals, the only available negociating tool is the threat to not subscribe at all. The smaller the institution, the more credible the threat. People who are specifically mandated to negociate with a publisher on behalf of a large institution or consortium cannot make such a threat, and must agree to the dictated price. (The publisher may pretend to demand an even more outrageous price at the beginning, and then retreat, in order to hide the fact that negotiations are meaningless.)

A common and significant way institutions try to improve the publishing system is by adopting open access policies. Clear and detailed explanations on the subject of open access to scientific publications are provided in a Unesco document. However, not all open access policies are effective. An effective policy should:

• ask researchers to deposit their articles in an institutional repository, in order to take control back from the publishers,
• be mandatory, and count only articles in the repository for managing careers,
• allow for publishers' embargos, but moot them by providing a "request article" button for semi-automatically distributing embargoed articles,
• avoid funding Gold open access publishing,
• provide some advantages, not only obligations, to researchers. These advantages, beyond the increase in readership which automatically comes with open access, can be a facilitation of administrative formalities, by the automatic compilation of publication lists.

Such a policy was adopted by the University of Liège under Bernard Rentier.

Academic publishing reform 2. New publishing models

The organization of traditional academic journals arose at a time when their most important task was to disseminate information, and when the printed word was expensive and therefore scarce. The electronic word is no longer expensive, but readers' time still is, and there remains some justification for imposing conciseness in articles. But disseminating articles can now be done for free, and subscription-based journals now hinder the flow of information using paywalls and legal threats, rather than helping it. Not to mention the very useful information which they never disclose -- the reviewers' reports.

There are many ideas on how to improve scientific communication. Articles themselves might be replaced by wikis or something else. Keeping articles as the primary means of communication, the natural idea is to build, as Gowers puts it,
"a cross between the arXiv, a social networking site, Amazon book reviews, and Mathoverflow". However, it is difficult to start a new system from scratch.
Any new system must be able to compete with the established journals, which not only are entrenched in the minds of academics, but also hold sway over their careers. Being able to compete limits the radicality of possible innovations: the new system must be officially recognized and indexed as a journal, and must play the game of bibliometrics. Of course, it is possible to solve each problem in turn, and to start winning over academics with services such as social networking, before trying to be recognized as a journal.

So, which innovations have been or are being tested?

The simplest idea is to start a traditional journal using modern tools, thereby reducing costs to a minimum. One can use free journal-management software. One can even rely on the arXiv for hosting and distributing the articles, and do only the selecting and reviewing: this is the idea of the overlay journals.

A more ambitious proposition is to start journals with advanced features such as open peer review or a new economic model. This was done by eLife, and PeerJ. These examples show that, when given the choice, most authors will make reviewers' reports public. Moreover, PeerJ has introduced a new economic model where authors pay of the order of 100 euros for a lifetime membership -- negligible costs when compared with the thousands of euros per article of typical Gold open access publishers.

Finally, the more radical idea of allowing the public to comment on articles, instead of picking one or more reviewers, is being tested by the Selected Papers Network and Publons. For the time being, these systems are not supposed to replace journals, but they can be used to publicly discuss articles. The community would certainly benefit if email exchanges between readers and authors, as well as reviewers' reports, were thus made public.

Write for humans, not for robots

The recent San Francisco Declaration On Research Assessment (DORA) aims at improving how scientific research is evaluated. To do this, the declaration wants not only to improve the evaluation process itself, but also to modify how research is reported, in particular by having authors of scientific articles cite original articles in preference to review articles. But this would be bad scientific practice: researchers should not have to worry about the reliability of bibliometrics when they do research or write articles.

As a method of evaluating research, bibliometrics has many well-known shortcomings. The DORA denounces some of these shortcomings, and proposes remedies. Some of these remedies are common-sense ideas, such as stopping using journal impact factors, and evaluating the contents of articles (and other research outputs) rather than relying on bibliometrics.
Some other proposals aim at improving bibliometrics. It is not obvious that this would be good, because bibliometrics will always encourage bad practices, such as trading citations and authorships, and splitting results into small articles. Eliminating some of the flaws at the expense of making metrics more complicated may not be worth the trouble.

And the proposals 10 and 16, which want authors of scientific articles to cite  original articles in preference to review articles, are downright pernicious. The problem is no longer to improve bibliometrics itself, but to modify how articles are written, with the aim of making bibliometrics more reliable. So the DORA claims that bibliometrics is so flawed that its use should be much reduced, but that it is so important that great efforts should be made to improve it -- efforts not limited to modifying the metrics themselves. In this sense, the DORA is self-contradictory. 

This might not matter too much, if citing original articles was good scientific practice. But it is not. The original literature is a tangled mess of more or less reliable and understandable texts. Review articles play the vital roles of making sense of it, and of promoting common, generally accepted terminology and ideas. Writing review articles is in some cases more useful than doing original research, and should be encouraged. Moreover, researchers often do not learn of existing results from original articles, but from other texts, which may be clearer or more accessible than the original articles. Citing the original articles would often mean citing articles without having read them, which is obviously bad practice.

Remember why articles cite earlier works in the first place: to avoid repeating material which is available elsewhere, and to help readers find the origins and proofs of results which are built upon. To do this efficiently, one should cite as few articles as necessary, and select them based on clarity, reliability and ease of access. (Open access should surely be favoured.) There is no reason to favour original articles. Favouring original articles serves another purpose: as the DORA puts it, to "give credit where credit is due". But the purpose of research articles is not to give credit. An article's contribution to the history of its subject is only a byproduct, and should not take precedence over its primary purpose of reporting scientific ideas.

Bibliometrics was supposed to help with the task of giving credit, but now we are told that it will not work unless we think more about bibliometrics than about readers when we write articles. This should be resisted: write for humans, not for robots.