Open peer review!
The main reason I was attracted to SciPost in the first place is that it practises open peer review, which means that the referee reports are publicly viewable. (The referees can choose to remain anonymous.) If one wants to improve the communication of research results, publishing referee reports is the obvious first step, as it requires no extra work, and has potentially large benefits on the quality of the process. Actually, publishing reports on a rejected article can even save some work if the article is later submitted elsewhere. (SciPost however erases reports on rejected articles.)
In closed peer review, the only indication of quality is the reputation of the journal. Using the reputation (or the impact factor) of a journal as a proxy for the quality of an article is however stupid. In contrast, open peer review guarantees the quality of the peer review process at the level of the article. Readers should prefer open peer reviewed articles. Therefore, authors should prefer journals that do open peer review.
In the case of SciPost, the referees not only write reports, but also give some qualitative grades in six domains: validity, significance, originality, clarity, formatting, grammar. While this is in principle a good idea, I do not see why each referee should comment on each aspect on the article. For example, I could complain that the first referee wrote a narrowly technical report, and has no understanding of the significance and originality of the work. Fortunately, readers can see that for themselves.
SciPost’s procedure is not for the faint-hearted: I and my coauthors were a bit shocked by the first referee report, which came without comments from the editor, and without indication whether there would be other reports. Fortunately there was a second report, and it was more insightful.
On the use of arXiv
According to the procedure, articles are submitted to SciPost from arXiv. While this is good and normal in the case of the initial submission, this also applies to subsequent revisions. Posting each revision on arXiv sounds strange, because in principle there could be many revisions until the article is published. But arXiv is not designed to host intermediate steps in a peer review process. ArXiv is designed to communicate articles to readers, and having too many versions can confuse those readers. Actually, arXiv has technical features that make it awkward to have many versions: there is a delay of about one day between the submission and the appearance of each version, and only the first four revisions of an article are announced in the daily “Replacemements” list.
While SciPost requires the posting on arXiv of intermediate versions, but not of the published version, the publishing platform Episciences does just the opposite. However, authors who are not fully happy with the published version, should not have to inflict it to arXiv readers. Journals should respect arXiv’s role in scientific communication, and avoid treating it as a technical platform.
Formatting
SciPost has its own format for articles, and takes great care of references. In practice, I found that the amount of required formatting work was reasonable. In principle however, I do not see why journals should apply their own cosmetics to the articles they publish. Moreover, having technically perfect references is useful not to readers, but to bibliometric robots, whose work needs maybe not be encouraged. So far, SciPost has missed the opportunity to free itself from needless formatting rules.
The platform: possible improvements
The platform for submitting and tracking articles works well, but could be improved on a number of points:
- It would be good to know in advance what the submission form looks like, without having to actually fill it. In particular, SciPost’s subject classes differ from arXiv’s, and should be made explicit somewhere.
- I would greatly appreciate if all the data that I enter into the platform were accessible to me from my account. This would include submission forms, comments, and correspondence with editors.
- It would be good to be able to preview texts such as comments on articles or replies to referee reports. Previewing before sending is all the more relevant, as these texts can contain Latex formulas. Being able to save them and send them later would also be good.
Many thanks for your valuable input! Let me try to address some of the points you raised, following your outline.
ReplyDeleteOn open peer review/reports:
Indeed, as you correctly point out, the openness of the process provides a great quality-increasing incentive. Concerning reports of rejected articles, our policy at the moment is to leave this to the discretion of the authors, the reason being that they might elect to resubmit a strongly revised version elsewhere, and feel that earlier critical comments would taint later judgement. This policy will be reconsidered by the Board at the next Virtual General Meeting.
Also, we indeed require referees to provide six `grades' for each paper, to gather as much information as possible from the evaluator. As you correctly note however, Editorial Fellows (and general readers) must interpret these gradings in their own light (we do ask the referee to `grade' their own capability to review).
Your suggestion of keeping authors more accurately updated on (to be) delivered reports and comments is noted, and this aspect will be improved upon.
On the use of arXiv:
Indeed at the moment we require all (substantial) revisions of the paper to be (re)submitted through the arXiv. The reason for this is to guarantee that readers do not lose track of which version is most up-to-date. Only at the production stage do we allow authors to send in a modified version with small corrections without updating the arXiv.
Besides this, there are also other issues with arXiv, most importantly that not all forms of Lecture Notes are accepted as submissions. For this (and the earlier-mentioned) reason(s), SciPost is currently implementing its own file upload system for Contributors, which will hopefully solve all these issues.
Formatting:
Ensuring a certain level of formatting is of some importance in ensuring overall quality of the Journal(s). We indeed aim to make the life of authors as simple as possible, but don't view enforcing this quality as missing an opportunity. A Journal with lax formatting simply looks too amateurish. As you correctly point out, we in particular take great care with references, to facilitate readers' ease and Crossref metadata compatibility.
The platform: possible improvements:
The bulk of the submission form indeed appears immediately after pre-filling it with the arXiv number. We will consider making it directly visible.
Most of the data you enter into the platform is available from your personal_page, under the relevant tabs; we will however revise this and improve it where necessary.
Your suggestion for a preview of texts entered for comments/replies/reports is an extremely useful one. This has in fact been immediately implemented, and is now already functional on the site.
Let me use this opportunity to thank you very much for these valuable inputs, which are very much appreciated. SciPost belongs to the community of researchers, and if features and facilities are requested by the community, they will be considered for direct implementation.
Thank you for the encouraging and detailed answer. Your reactivity to suggestions is remarkable, and I hope that other researchers will suggest valuable improvements.
Delete