No more rejections: eLife reinvents the academic journal

Academic journals as gatekeepers

Nowadays, the role of academic journals is not so much disseminating information, as choosing which submitted articles to accept or to reject for publication. Gatekeeping was surely important when journals were actually printed, in order to save paper and shelf space. Nowadays, it serves as a token of quality, which can make or break the careers of the articles’ authors.

The journal’s publishing decisions are based on the peer review process, which involves the work of reviewers and editors. The process can be fair — or biased, it can be rigorous — or cursory, it can be constructive — or confrontational. How would we know, when the result is only one bit of publicly available information? Not even one bit actually, since when an article is rejected, we do not even know that it was submitted.

In good cases, the process can in fact yield more useful information: because it results in the article being improved, and/or because the reviewers’ reports are made public. However, this generally does not occur when the article is rejected, even in best-practices-in-their-field journals such as SciPost.

eLife’s new model

To avoid wasting the reviewers’ work, and to make it impossible to judge articles by the journals that publishes them, the obvious solution is to eliminate accept/reject decisions. This is the basis of eLife’s new model, announced here. As with most radical progress, the challenge is to succeed in spite of the system’s corruption and its perverse incentives. In eLife’s case, this requires that the resulting articles count as publications for whoever manages the authors’ careers. To this end, eLife is assigning DOIs, and the authors can designate an official Version of Record.

Will eLife will thrive or perish? Will its new model be emulated? Will much-needed systemic change follow? Hard to tell. What I can do for the moment is to discuss the model’s logic in more detail, and point its strengths and weaknesses.

The editor is back

A striking feature of the new model is the crucial role of editors. In old times, editors used to run their journals without help from reviewers — Einstein was outraged when presented with a report from an anonymous reviewer in 1936, for the first time in his career. Nowadays, editors delegate much work and responsibility to reviewers, who are often asked to provide explicit recommendations on whether to accept or reject submitted articles.

In eLife’s new model, reviewers still write reports, but they can focus on giving their scientific opinion without worrying about journal’s publication criteria. Editors now make important decisions:

• Performing an initial screening to decide whether the submitted article deserves consideration, or should be desk-rejected. Given the large numbers of articles that are written nowadays, some sort of screening is unavoidable.

• After peer review has been done, rating the article for significance and strength of support.

The purpose of the rating system is to help readers quickly decide whether the article is worthy of trust and attention. Crucially, ratings are not optional and delegated to reviewers as in SciPost: they are done systematically, and involve the editors. This will hopefully allow the ratings to be done consistently and meaningfully, by people who are well-placed to compare many articles. To evaluate the quality and interest of an article, explicit ratings are potentially much better than a crude accept/reject decision.

This system requires a lot of work and commitment from editors: this could well be a weakness. However, not rejecting articles and doing all the review work in public can certainly save a lot of the editors’ and reviewers’ time, if not at eLife in particular, at least at a systemic level.

More details on the rating system

The rating system or “eLife assessment” system is described here:

• 5 ratings for significance of findings, from useful to landmark.

• 6 ratings for strength of support, from inadequate to exceptional.

These ratings are supposed to be embedded in a short text that summarizes the editors and reviewers’ opinion of the article. Having standard ratings promises to allow easy comparison between articles, and should be very valuable to readers. At the same time, it makes it possible to do statistics and build metrics, with all their potential for misuse. These metrics would however be directly based on an explicit evaluation of the article, and could therefore be more meaningful than citation numbers, let alone journal impact factors.

This rating system is not far from what I have been proposing in general and in particular. Possible improvements to the eLife system could include letting authors suggest ratings for their own articles, and adding a rating for clarity i.e. quality of writing and presentation.

Reactions

• Thoughtful blog post by Sophien Kamoun.

• The usual criticism that researchers have to play by the rules of the system, and would commit suicide by participating in eLife’s new model. By this logic, eLife should have been content to perform good quality open peer review: attempts at further improvement are doomed.

• HHMI’s enthusiastic support. After Plan S, another case of funders pushing researchers towards good practices.

With open peer review, open is just the beginning

Abstract

Open peer review does not just mean publishing existing reviewer reports, but should also lead to writing reports primarily for the public. We make a specific proposal for structured reviewer reports, based on the three criteria validity, interest and clarity.
This post is partly based on a joint proposal with Anton Akhmerov for improving the structure of reviewer reports at SciPost. Feedback from Jean-Sébastien Caux on that proposal is gratefully acknowledged.

 

Benefits of open peer review: the obvious and the less obvious

In the traditional model of academic peer review, reviewers’ reports on the submitted article are kept confidential, and this is a big source of inefficiency and waste. If the article is published, the readers can neither assess how rigorous the process was, nor benefit directly from the reviewers’ insights. If it is rejected, the work has to start all over again at another journal.

Open peer review, defined here as making the reports public, could help journals remedy the penury of reviewers: if applied to rejected articles, by avoiding duplicating effort, and if coupled with naming reviewers, by giving them better incentives to do the work. However, the consequences of open peer review may be more far-reaching. Published reports can indeed be used for evaluating the article’s interest and quality. In aggregate, they could be used for evaluating journals and researchers. For these purposes, they would certainly be better than citation counts.

The Im-flip condition in the two-dimensional Potts model

I have been using this blog for publishing the reviewer reports that I write for journals, since the journals typically do not publish the reports. However, the new journal SciPost Physics does publish the reports for accepted articles. I have recently reviewed an article by Gorbenko, Rychkov and Zan for SciPost Physics, and written about the experience: it would seem that I need not blog about that article, since my report is already online.

However, not everything that I have to say on the article made it into the report. I will now write on two calculations that I did: the first one is a test of one of the article’s main predictions in more general cases, the second one is a direct derivation and generalization of a technical result that they obtain in a roundabout way.

How strong are the objections to Plan S?

After a coalition of European science funding agencies announced their Plan S initiative for open access, a number of researchers wrote an open letter criticizing the move, under the title “Reaction of Researchers to Plan S: Too Far, Too Risky”. To summarize, they fear that Plan S would increase costs, lower quality, and restrict academic freedom. In order to evaluate how seriously these fears should be taken, let me start with a 5-point analysis of the issues, before discussing the open letter’s specific concerns.

Point 1: Traditional journals are overpriced by an order of magnitude.

 

Overall, publishers earn about 3800-5000 euros per article they publish. The true costs of organizing peer review are much lower: a few hundred euros per article at SciPost or PeerJ. The difference is therefore not mainly due to commercial publishers’ profits, but rather to large inefficiencies at all traditional publishers. (Have a look at the salaries of the American Chemical Society’s executives.)

Some publishers also use the money for subsidizing other activities, such as conferences (learned socities) or science journalism (Nature, Science). While these activities are valuable, they are not the main purpose of scientific publishing, and should not determine its future.

SciPost two years on

There are two reasons why we should care about the journal SciPost Physics: it practices open peer review, by publishing the exchanges between authors and reviewers, and it is free to authors and readers, at a time when academia is struggling to escape the stranglehold of predatory publishers such as Elsevier.

Two years after it was launched, SciPost Physics is alive on well, with 155 published articles at the time of this writing. The journal has managed to attract articles of high quality, some of them by well-known physicists such as Cardy, Verlinde, Seiberg, or Rychkov. The main challenges are now to attract many more authors, and sustainable funding. The journal is funded by institutional subsidies, an economic model that works for arXiv but that is rare for journals. If the finances worked out, the journal could ultimately hope to become a megajournal, and publish thousands of articles a year like PLOS One or PeerJ. For the moment, the stated aim of the “Expansion and Sustainability Drive” is to publish 500 articles a year at a cost of 200.000 euros. So each published article should cost 400 euros, a very reasonable price for a lean electronic journal: an order of magnitude more than arXiv, an order of magnitude less than Elsevier. (Unlike Elsevier, SciPost does not pay executives, lobbyists, shareholders, salespersons, lawyers, etc.)

In this post I will comment on SciPost’s workflow and platform, updating my earlier assessment in light of two years’ worth of experience. The aims are to help authors decide whether this journal suits them, to determine which features of SciPost should be emulated (or avoided) by other journals, and to provide feedback to SciPost itself.

Publishing in SciPost: a must?

Now that I have published my first article in SciPost, let me comment on that experience.

 

Open peer review!

 

The main reason I was attracted to SciPost in the first place is that it practises open peer review, which means that the referee reports are publicly viewable. (The referees can choose to remain anonymous.) If one wants to improve the communication of research results, publishing referee reports is the obvious first step, as it requires no extra work, and has potentially large benefits on the quality of the process. Actually, publishing reports on a rejected article can even save some work if the article is later submitted elsewhere. (SciPost however erases reports on rejected articles.)