A portrait of the blogger as a peer-reviewer

In a publish-or-perish system of scientific communication, large numbers of articles are written and submitted to journals. Editors face the challenge of finding peer reviewers for all these articles. Scientists who accept invitations to review, usually for free, play a vital role in the system. This gives them a lot of freedom to choose which invitations to accept, and to perform peer review as they wish. 

This blog post states my own current “code of conduct” as a peer reviewer: my answers to questions that arise whenever a researcher receives an invitation to review, or has an article to referee.

Should I review this article?

1. Only review for journals I would publish in. This means the publisher should not be Elsevier, the journal should be open access, should not have obnoxious formatting constraints, should not cost too much to the community, and should publish reviewer reports.

2. See the article’s full text before deciding. And the article should already be available on arXiv as a preprint.

The state and prospects of academic peer review

(A PDF version of this post is available on HAL.)

Peer review plays a crucial role in the communication and management of research. How well is it working? How could it be improved? The answers depend on whom you ask. Here they are debated by seven characters with different points of view.

1. Osman: Now that everyone is here, let us start this debate on academic peer review: should it be strengthened, overhauled, or abandoned? When I arrived, Clementine was already in the room, reviewing a paper and complaining about it. This leads to my first question: is peer review taking too much time and effort? 

2. Clementine: I was indeed complaining, but like most colleagues I realize that peer review is an essential part of our work: this is how we validate each other’s work, this is how we assess its interest and likely impact, and this is how we collectively improve how papers are written. Now, the problem is that so many papers get submitted nowadays, it becomes hard to do a proper job on all of them. Even after declining many invitations to review, I have to be pragmatic and focus on the main question: is the submitted paper worth publishing in that journal? If the paper is obviously good work by well-known people, it can be waved through with minimal scrutiny. If it is mediocre work, the rejection should be short and final – I do not want to get dragged into a technical debate with the authors. If however I am unsure whether the paper should be accepted after a quick first reading, then I need to scrutinize it, and to find a good reason for either rejecting it, or accepting it after some improvements. 

3. Deirdre: So you spend more time reviewing second-rate works than really interesting papers? I am glad I stopped reviewing years ago. 

An environmental boycott of Elsevier

Since 2012, thousands of academics have been boycotting the academic publisher Elsevier, whom they blame for overpricing its journals, and more generally for resisting open access to the scientific literature. Of course, most major academic publishers are guilty of the same, but Elsevier stands out as the worst offender. For instance, Elsevier was the last major publisher to join the Initiative for Open citations, years after all the others. Elsevier did not join the Initiative for open abstracts, and they play a leading role in the legal persecution of Sci-Hub.

In February 2022, Elsevier was denounced for helping the fossil fuel industry via its publishing and consulting activities. Again, other publishers are doing that too, and Elsevier is only the worst (or more prominent) offender. This has led to a (no longer active) petition by the Union of Concerned Scientists. Elsevier was not impressed, and now there is a campaign called Stop Elsevier.

This campaign includes a boycott. Participants may commit to

• Refuse to Review

• Refuse to Submit

• Write to Editors

• Refuse to Edit

• Take Direct Action

• Share Boycott on Social Media

There are also various options for allowing these commitments to be publicly shared, as in the original Cost of Knowledge boycott. For the moment, they are still kept private, though.

The new boycott won’t sink Elsevier by itself, but it could strengthen its well-deserved reputation for unprincipled greed. This would help discourage academic institutions from doing business with Elsevier.

No more rejections: eLife reinvents the academic journal

Academic journals as gatekeepers

Nowadays, the role of academic journals is not so much disseminating information, as choosing which submitted articles to accept or to reject for publication. Gatekeeping was surely important when journals were actually printed, in order to save paper and shelf space. Nowadays, it serves as a token of quality, which can make or break the careers of the articles’ authors.

The journal’s publishing decisions are based on the peer review process, which involves the work of reviewers and editors. The process can be fair — or biased, it can be rigorous — or cursory, it can be constructive — or confrontational. How would we know, when the result is only one bit of publicly available information? Not even one bit actually, since when an article is rejected, we do not even know that it was submitted.

In good cases, the process can in fact yield more useful information: because it results in the article being improved, and/or because the reviewers’ reports are made public. However, this generally does not occur when the article is rejected, even in best-practices-in-their-field journals such as SciPost.

eLife’s new model

To avoid wasting the reviewers’ work, and to make it impossible to judge articles by the journals that publishes them, the obvious solution is to eliminate accept/reject decisions. This is the basis of eLife’s new model, announced here. As with most radical progress, the challenge is to succeed in spite of the system’s corruption and its perverse incentives. In eLife’s case, this requires that the resulting articles count as publications for whoever manages the authors’ careers. To this end, eLife is assigning DOIs, and the authors can designate an official Version of Record.

Will eLife will thrive or perish? Will its new model be emulated? Will much-needed systemic change follow? Hard to tell. What I can do for the moment is to discuss the model’s logic in more detail, and point its strengths and weaknesses.

The editor is back

A striking feature of the new model is the crucial role of editors. In old times, editors used to run their journals without help from reviewers — Einstein was outraged when presented with a report from an anonymous reviewer in 1936, for the first time in his career. Nowadays, editors delegate much work and responsibility to reviewers, who are often asked to provide explicit recommendations on whether to accept or reject submitted articles.

In eLife’s new model, reviewers still write reports, but they can focus on giving their scientific opinion without worrying about journal’s publication criteria. Editors now make important decisions:

• Performing an initial screening to decide whether the submitted article deserves consideration, or should be desk-rejected. Given the large numbers of articles that are written nowadays, some sort of screening is unavoidable.

• After peer review has been done, rating the article for significance and strength of support.

The purpose of the rating system is to help readers quickly decide whether the article is worthy of trust and attention. Crucially, ratings are not optional and delegated to reviewers as in SciPost: they are done systematically, and involve the editors. This will hopefully allow the ratings to be done consistently and meaningfully, by people who are well-placed to compare many articles. To evaluate the quality and interest of an article, explicit ratings are potentially much better than a crude accept/reject decision.

This system requires a lot of work and commitment from editors: this could well be a weakness. However, not rejecting articles and doing all the review work in public can certainly save a lot of the editors’ and reviewers’ time, if not at eLife in particular, at least at a systemic level.

More details on the rating system

The rating system or “eLife assessment” system is described here:

• 5 ratings for significance of findings, from useful to landmark.

• 6 ratings for strength of support, from inadequate to exceptional.

These ratings are supposed to be embedded in a short text that summarizes the editors and reviewers’ opinion of the article. Having standard ratings promises to allow easy comparison between articles, and should be very valuable to readers. At the same time, it makes it possible to do statistics and build metrics, with all their potential for misuse. These metrics would however be directly based on an explicit evaluation of the article, and could therefore be more meaningful than citation numbers, let alone journal impact factors.

This rating system is not far from what I have been proposing in general and in particular. Possible improvements to the eLife system could include letting authors suggest ratings for their own articles, and adding a rating for clarity i.e. quality of writing and presentation.

Reactions

• Thoughtful blog post by Sophien Kamoun.

• The usual criticism that researchers have to play by the rules of the system, and would commit suicide by participating in eLife’s new model. By this logic, eLife should have been content to perform good quality open peer review: attempts at further improvement are doomed.

• HHMI’s enthusiastic support. After Plan S, another case of funders pushing researchers towards good practices.

Suprisingly good CNRS guidelines on open access

When it comes to good practices, research institutions are often good at declarations of principles, and not so good at implementation. For example, it is easy to declare that research assessment should be qualitative and not rely too much on bibliometrics, but harder to do it in practice.

This is why I am pleasantly surprised by recent CNRS guidelines on open access. These guidelines take the form of a letter to CNRS researchers who have to fill yearly reports on their work. 

 

The salient points about open access are:

• Over the last three years, the proportion of articles by CNRS researchers that are openly accessible rose from 49% to 77%. (OK, these data mean little in the absence of more details.)

• CNRS is adopting a rights retention strategy, as proposed by Coalition S: it recommends that all articles be distributed under a CC-BY license. In particular, this allows the articles to be made openly accessible right at the time of publication.

• CNRS is not asking researchers for their lists of publications: instead, CNRS just takes publication lists from the national preprint archive HAL.

• The weak point of all this seems to be the impractical and clunky nature of HAL. However, as reminded in the letter, HAL is increasingly interoperable with disciplinary archives such as ArXiv and BiorXiv. And indeed, my recent articles were automatically harvested by HAL from ArXiv.

This goes in the direction of having a strong open access mandate while requiring no extra work from researchers. To get there, it remains to make the CC-BY license mandatory, and the upload of articles to HAL fully automatic.

Open access by decree: a success of Plan S?

“Science family of journals announces change to open-access policy”: the title of this Nature news article may sound boring, but the news are not:

Science is allowing authors to distribute their articles under Creative Commons license, free of charge, without embargo.

Sounds too good to be true? Well, of course, there are caveats. First, this is the author’s accepted version, not the published version. (OK, minor point.) Second, this is a trial policy, which may or may not be made permanent. And third, this only applies to authors who are mandated to do so by their Coalition S funders.

Who, you may ask, are these happy authors? I will defer to Wikipedia for reminders about Plan S and Coalition S, and move to the next question: how did Coalition S achieve this? Last summer, Coalition S announced a Rights Retention Strategy that mandates fundees to distribute their work under CC license, without embargo, no matter what the publisher makes them sign. In their own words,

This public licence and/or agreed prior obligations take legal precedence over any later Licence to Publish or Copyright Transfer Agreement that the publisher may ask the author to sign.

This type of “open access by decree” may sound like little more than an exercise in blame-shifting. To an author who is caught between a funder and a journal’s incompatible demands, it should not matter who blacklists the other: the author cannot publish in that journal. However, Plan S got bad press among some researchers for appearing to blacklist journals. Now it is the journals who will have to blacklist funders, and to reject submitted articles for technicalities.

The success of the maneuver depends on publishers’ reactions. Coalition S is too big to ignore, so publishers have to take a stand. Nature journals have recently announced their adoption of Gold open access options, for a charge of the order of 9.500 euros per article. This was denounced as outrageously expensive. Björn Brembs even took it as a refutation of the widespread idea that the author-pays model would lead to lower costs.

However, Coalition S-funded authors can now choose between publishing in Science for free, or in Nature for 9.500 euros. It does seem that Science is competing on price! Not so fast: Nature’s high price is partly an illusion, as it comes in the context of transformative agreements, which are supposed to transition Springer Nature’s journals to open access. The idea is that academic institutions’ journal subscriptions would also cover open access publishing of their researcher’s articles. For an author who is covered by a transformative agreement, publishing open access in Nature is effectively free, which may be why Science had to offer the same for free, just to stay competitive.

At this stage, it is not clear which authors will face which choices of open access options and pricing. It is therefore a bit early for seeing the effects of the Rights Retention Strategy. At least, we now have the admission by an important publisher that open access is not an extra service that should bring them extra revenue.

(1/2) Open access mystery: why did the ERC backstab Plan S?

The European Research Council (ERC) just announced that they would withdraw their support for Coalition S, the consortium of research funders behind Plan S. Plan S is the valiant but not universally welcome attempt to impose strong open access requirements to research articles, without paying more money to publishers.

The ERC is Europe’s most prestigious research funder, and a main backer of Plan S. Without Plan S, the ERC has no open access strategy, and without the backing of ERC, Coalition S may not be big enough for succeeding. Why would ERC make this U-turn? I do not know, but let me gather a few potentially relevant pieces of the puzzle. The pieces are of three types:

• some context on the ERC and more generally on Europe’s research plans,
• the recently announced rights retention strategy by Coalition S,
• ERC’s meager and not very credible justification for their withdrawal.
  

With open peer review, open is just the beginning

Abstract

Open peer review does not just mean publishing existing reviewer reports, but should also lead to writing reports primarily for the public. We make a specific proposal for structured reviewer reports, based on the three criteria validity, interest and clarity.
This post is partly based on a joint proposal with Anton Akhmerov for improving the structure of reviewer reports at SciPost. Feedback from Jean-Sébastien Caux on that proposal is gratefully acknowledged.

 

Benefits of open peer review: the obvious and the less obvious

In the traditional model of academic peer review, reviewers’ reports on the submitted article are kept confidential, and this is a big source of inefficiency and waste. If the article is published, the readers can neither assess how rigorous the process was, nor benefit directly from the reviewers’ insights. If it is rejected, the work has to start all over again at another journal.

Open peer review, defined here as making the reports public, could help journals remedy the penury of reviewers: if applied to rejected articles, by avoiding duplicating effort, and if coupled with naming reviewers, by giving them better incentives to do the work. However, the consequences of open peer review may be more far-reaching. Published reports can indeed be used for evaluating the article’s interest and quality. In aggregate, they could be used for evaluating journals and researchers. For these purposes, they would certainly be better than citation counts.

Plan S implementation: in danger of mission creep?

The principles behind plan S have already sparked lots of debate, including an open letter denouncing the plan, based on objections that I found not very convincing. Now that the plan’s promoters have published their draft implementation guidance (and are inviting comments on it), the discussion can become more specific. Given the boldness of the principles, their implementation cannot be painless, and is bound to raise criticisms if not resistance. It is therefore both crucial and difficult to get the implementation right, and to inflict the minimum amount of pain that is necessary for achieving the goals.

This means that the implementation should be tightly focussed on open access and cost reduction, to the exclusion of other possible goals that a reform of the publishing system might have. I will discuss whether the draft implementation guidance is indeed focussed enough. But first, let me be more specific about what a minimal implementation might look like.

How strong are the objections to Plan S?

After a coalition of European science funding agencies announced their Plan S initiative for open access, a number of researchers wrote an open letter criticizing the move, under the title “Reaction of Researchers to Plan S: Too Far, Too Risky”. To summarize, they fear that Plan S would increase costs, lower quality, and restrict academic freedom. In order to evaluate how seriously these fears should be taken, let me start with a 5-point analysis of the issues, before discussing the open letter’s specific concerns.

Point 1: Traditional journals are overpriced by an order of magnitude.

 

Overall, publishers earn about 3800-5000 euros per article they publish. The true costs of organizing peer review are much lower: a few hundred euros per article at SciPost or PeerJ. The difference is therefore not mainly due to commercial publishers’ profits, but rather to large inefficiencies at all traditional publishers. (Have a look at the salaries of the American Chemical Society’s executives.)

Some publishers also use the money for subsidizing other activities, such as conferences (learned socities) or science journalism (Nature, Science). While these activities are valuable, they are not the main purpose of scientific publishing, and should not determine its future.

CNRS rejects Couperin's claimed victory in Springer big deal

After long and tortuous negotiations, the French consortium Couperin has claimed victory in its recent agreement with Springer, after having secured price decreases. This claim seems reasonable, as prices of big deals with publishers tend to increase steadily. Of course, critics can still point out that Springer remains very expensive compared to smaller, more efficient publishers. But at least Springer seems amenable to some compromises in negotiations. And one should not forget that the greediest and most obnoxious publisher remains Elsevier, who even refused to join the Initiative for open citations.

I was therefore surprised when CNRS announced its rejection of the Springer deal, although CNRS takes part in Couperin and was actively involved in the negotiations. The email announcement came from Alain Schuhl, a CNRS official who is also a member of Couperins’ governing council. (See the email and its English translation below.) This email was a warning to CNRS researchers that access to Springer journals was now cut off. However, articles from 2017 and earlier are still available, as they are coverd by the previous subscription.

The explanation for the rejection is that the deal’s price was too high according to the Ministry’s open science plan. This explanation makes little sense for two reasons:

Where should academics write if they want to be read?

Out of the 2 million academic articles that are published each year, many are not read by anyone but their authors, and most have no more than a handful of readers. For someone who writes in order to spread ideas, and does not publish just to avoid perishing, this can be quite discouraging. Of course, not everything one has to say is of interest to many people. Still, part of the problem could come from the academic article as a venue, and one may wonder whether other writing venues (such as a blog) could reach more readers.

In this post I will list various venues for scientific writing, and try to do an order-of-magnitude comparison between them. I will not simply estimate how many readers are reached: a tweet may reach many people, but it is read in seconds and can be quickly forgotten. Rather, I will try to estimate the ratio between the time spent by all readers on a text, and the time spent writing it.

Springer threatens to go rogue, and retreats

In its negotiations with Springer, the consortium Couperin that was in charge of most French researchers’ subscriptions had been modestly asking that Springer renounce “double dipping”, i.e. does not get paid twice for the same articles – once via subscriptions, once via open access APCs. But this would have meant decreasing subscription prices by 15%, while Springer has been used to yearly increases of the order of 3-5%.

Today’s news are that negotiations have broken down, and most French researchers are set to lose legal access to most articles published by Springer on April 1st. (See CNRS’s note on the subject, in French.) In such a conflict, researchers can stand up against the publisher by

• not complaining when they lose access to journals, and getting the articles elsewhere (which nowadays mostly means at Sci-Hub),
• boycotting the publisher, i.e. not submitting articles to its journals, and not working for them as a referee or editor.

Coincidentally, I have received an invitation to referee an article for JHEP, a journal published by Springer. I have declined as follows:

Most French research institutions are set to lose access to Springer journals on April 1st. I am aware that this does not affect access to JHEP, which is covered by SCOAP3, and that in our field all articles are on arXiv anyway. However, as a matter of principle, I would like to protest Springer’s extortionate commercial practices, and to show solidarity with colleagues who will lose access to their own work. Therefore, I am suspending all new collaboration with JHEP, and I am declining to review this article.

Update on April 3rd: According to an email from Couperin, access to Springer journals has in fact not been cut off. It seems that Couperin got away with rejecting Springer's latest offer, and saying that they were willing to continue negotiating. So Springer was bluffing, and does not dare take a hard line against Couperin. If such a weak and ill-prepared consortium, with little support from researchers (who else is boycotting Springer?) can defeat Springer, pretty much anyone can.

Update on April 11th: There is now a petition called "Springer, we can do without".

Update on October 18th: An agreement has been found between Couperin and Springer. Couperin boasts that it has obtained a modest price decrease: this merely reverses a few years out of several decades of unjustified increases. And the 9-10 months where Couperin's member institutes kept access to Springer journals without a subscription, will eventually be paid retroactively. (Except for institutes who did not accept the new agreement.) Still, Springer's behaviour is not too bad for a predatory publisher. And the price decrease is in absolute terms a large amount of money, which could make much difference if it was injected into frugal publishing platforms. So I am no longer boycotting Springer.

Couperin is now negotiating with Elsevier, aiming to achieve comparable savings, and pushing for open access. Nevertheless, it is not clear to me how Couperin's deal with Springer, and negotiating stance with Elsevier, are compatible with the Plan S to which CNRS officially subscribes: if Couperin believed in Plan S, it should at least refrain from committing beyond 2020, the date at which Plan S is supposed to come into force.

The open secrets of life with arXiv

If you only think of arXiv as a tool for making articles openly accessible, consider this: in 2007, a study showed that papers appearing near the top of the daily listing of new papers on arXiv, will eventually be more cited than papers further down the list – about two times more cited. And there is a daily scramble for submitting papers as soon as possible after the 14:00 EDT deadline, in order to appear as high as possible on the listing. The effect is not as perverse as it seems, as there is no strong causal relation between appearing near the top and getting more citations. (More likely, better papers are higher in the listing because their authors want to advertise them.)

The consequences of arXiv’s systematic use in some communities are actually so deep that a speciation event has occurred among researchers, and a new species of arXivers has appeared. Here I will try to explain how arXivers live, in order to help non-arXivers understand arXivers, and have an idea of what could happen to them if the currently proliferating clones of arXiv gained widespread use.

Couperin vs Springer and Elsevier: towards less extortionate deals?

Historically, the French consortium Couperin has obtained poor results in negotiating with predatory publishers, mostly consenting to their high and increasing prices. This is not necessarily Couperin’s fault, although it does not help that Couperin’s leadership appears weak and ill-informed. Rather, this is a consequence of the basic economics of scientific publishing, with publishers systematically abusing their strong position. Even the Finnish consortium FinELib, which was determined to seek a good deal and enjoyed a fair amount of support from researchers, recently consented to one more extortionate deal with Elsevier.

However, recent developments suggest that Couperin could fare better in current and upcoming negotiations:

Germany won't pay for Nature's "scientific porn", and other messages from Couperin's open science days

Earlier this week, there was a mini-workshop in Paris called Couperin’s open science days 2018. (Original title in French: Journées sciences ouverte 2018.) I followed most of it via webcast, and I will now summarize some of the salient points. The videos are available online, but most of them are in French.

The German way: Horst Hippler and Ralf Schimmer

 

The most important messages came from Germany: the country whose academic institutions have thought seriously about scientific publishing, and have organized themselves so as to drive the needed reforms. The most salient manifestation so far has been the standoff with Elsevier, and it was nice to have further details on the strategy.

After Elsevier, should we boycott Springer?

While the ongoing “Cost of knowledge” boycott of Elsevier may not be very effective, the likely “no deal” hard exit of Germany from Elsevier subscriptions renews the boycott’s relevance, and maybe its urgency. It is indeed likely that most German universities and research institutions will lose access to Elsevier articles in 2018.

As a researcher, why would I continue publishing in journals that are in principle inaccessible to most of my German colleagues? Universal access to the literature via Sci-Hub is under increasing legal assault and should not be taken for granted. In these circumstances, boycotting Elsevier is no longer only a matter of fighting an obnoxious publisher, but also a basic necessity of ensuring that articles are accessible to their intended audience. (Unless one thinks that the intended audience is not the scientific community, but the paying Elsevier subscribers.)

Now it turns out that if I boycott Elsevier because of Germany, I may have to boycott Springer because of France.

Self-publishing a book with Glasstree

Three years and three major revisions after it first appeared on Arxiv and GitHub (why GitHub? see this blog post), my review article on two-dimensional conformal field theory may be mature enough for appearing in book form. But with which publisher?
To answer this question, I should first say why I would want to have a book in the first place, since the text is already on Arxiv.

Why don't academics write in Wikipedia?

Since several years ago, Wikipedia is being widely used by academics. As a theoretical physicist, I often use it as a quick reference for mathematical terminology and results. Wikipedia is useful in spite of its many gaps and flaws: there was no general article on two-dimensional conformal field theory until I started one recently, the article on minimal models is itself minimal, and googling conformal blocks sends you to a discussion on StackExchange, since there is nothing on Wikipedia.

The paradox is that many academics see these gaps and flaws in the coverage of their own favourite subjects, yet do nothing to correct them. Let me discuss three possible reasons for this passivity: fear of Wikipedia, lack of time, and laziness.

The jungle outside the ivory tower

 

Attracting and retaining academic contributors has long been recognized as a challenge by Wikipedians, to the extent that there are guidelines on how to do it.

Publishing in SciPost: a must?

Now that I have published my first article in SciPost, let me comment on that experience.

 

Open peer review!

 

The main reason I was attracted to SciPost in the first place is that it practises open peer review, which means that the referee reports are publicly viewable. (The referees can choose to remain anonymous.) If one wants to improve the communication of research results, publishing referee reports is the obvious first step, as it requires no extra work, and has potentially large benefits on the quality of the process. Actually, publishing reports on a rejected article can even save some work if the article is later submitted elsewhere. (SciPost however erases reports on rejected articles.)