Sunday, 24 July 2016

SciPost: the right tool for commenting arXiv articles?

ArXiv has not changed much since it started in 1991, and it is only starting to consider the obvious next steps: allowing comments on articles, followed by full-fledged open peer review. Scientists have not all been waiting idly for the sloth to make its move, and a few have tried to build systems for doing that. Here I will discuss a recent attempt, called SciPost.


 A strong editorial college

The most distinctive feature of SciPost is its editorial college, made of well-known theoretical physicists. These people do not just lend their names to the project. Given how SciPost functions, they have a lot of work:
  • Approve registration of new contributors.
  • Approve requests for new Commentary pages. Such requests need to be sent by contributors in order to be able to comment on arXiv or journal articles.
  • Approve comments themselves.
  • Oversee the refereeing of submitted articles, according to a quite complicated procedure.
The composition and degree of involvement of the editorial college make it clear that the emphasis is on the quality of contributions. Commenting platforms with weak quality control can easily degenerate into discussion forums. (Cf the late Selected Papers Network.) SciPost seems designed to avoid this.


An adequate platform

Having tested the website by looking around and writing a small comment, I have found that it works well. My registration, request for a Commentary page, and comment, were approved in less than 24 hours, sometimes much less. Latex formulas are allowed. The search function works as expected.
While this is a good start, I hope that the platform’s capabilities will improve in due time:
  • A potentially crucial point would be to have Commentaries linked from arXiv using trackbacks.
  • Contributors who comment on an article could have the option to inform the authors by an automatically written email. And readers could have the option to follow a number of articles and contributors.
  • Different versions of the same article are apparently treated as unrelated objects: it would be nicer to have one Commentary for all versions, with the possibility to indicate which comments apply to which version(s).
  • More generally, SciPost could do less categorizing, and more tagging. There need not be essential distinctions between different versions of the same article, between Reports, Replies and Comments on an article, between authors and other contributors. These distinctions could be indicated by tags, rather than woven into the structure of the platform.
If the platform became more flexible, it could be used in interesting ways that have not necessarily been foreseen by its creators. For example, if anonymous Comments were allowed, then I could post an anonymous report that I wrote for some other journal.


A new flavour of open peer review

SciPost’s organization of open peer review is disctinctive for
  • the strong involvement of editors,
  • the principle that reports are publicly viewable, while reviewers can remain anonymous,
  • the deletion of reports when articles are rejected.
Deleting information that has been publicly available does not look like a good idea. In contrast, PeerJ publishes reports only when articles are accepted. But it would be better to always keep reports publicly available: reports of a rejected article are particularly valuable, as they can prevent unnecessary duplication of effort if the article is submitted elsewhere.
Ultimately, one could even eliminate the notions of accepted and rejected articles. While not being that radical, SciPost takes steps in this direction by having reviewers rate articles in a more nuanced way than just accepting or rejecting. Reviewers are supposed to rate the validity, significance, originality and clarity of articles, with six possible grades (from “poor” to “top”) in each case. (The originality rating might be a bit subjective.) It would be good if such ratings were available to all commentators, and not just reviewers.


The problem of attracting contributors

The main difficulty for any initiative like SciPost is to attract enough contributors, and to become a standard tool for a large enough community. With its editorial college, focus on quality, and intitutional backing, SciPost has a chance of attracting quality-conscious contributors.
However this comes at the expense of ease of use. New contributors wanting to post a comment should not have to wait for three successive approvals from moderators (for registering, opening a Commentary page, and posting the comment itself). Wikipedia and StackExchange have shown that allowing new contributors to bring content online immediately is not necessarily detrimental to quality control. In the case of SciPost, weak contributions need not be blocked: what matters is that strong contributions are recognized, displayed more prominently, and easy to find.



SciPost is probably the best currently active platform for publicly commenting arXiv articles, at least in theoretical physics. I hope that it gains the popularity that it deserves, and that its developers build on their good start and make it more flexible and easy to use.