- the quantitative approach of counting citations and/or computing alternative metrics,
- the qualitative approach of conducting peer review, and giving quality tokens to articles. A quality token can be publication in a journal, or a grade.
Giving grades, implicit or explicit.
Assuming some form of peer review is conducted, how can the results be concisely summarized?
Judging articles from the journals which publish them is widely recognized as bad practice. This bad practice is however very widespread, maybe because it has the advantage of simplicity. In each field of research journals are grouped in three or four tiers according to an informal but universally accepted hierarchy, so peer review implicitly results in giving grades which take three or four values -- something very helpful to impatient evaluators. But these grades are assigned in a rather contorted way, since the reviewers are asked not to assign grades, but to judge whether articles are suitable for publication in a given journal.
It would therefore be useful to rate articles in a more direct and meaningful way. This has been attempted by some innovators, most recently by the open peer reviewing platform Publons. Publons asks reviewers to give two scores to each article: one score for quality, one score for significance. Each score is a number from 1 to 10, "where 10 is amazing, 5 is average, and 1 is poor". The most obvious problem with this system is that the meaning of each possible score is not clearly defined: in which cases should you give a score of 6 rather than 7?
Principles of rating.
For a rating system to be useful and meaningful, some principles must be obeyed. I propose that the most important principles are:
- Clarity: Each possible grade should have an explicit meaning. In particular grades should not be numbers, as adding or averaging them would destroy their meaning.
- Transparency: Each grade should come with all the underlying arguments and considerations. Grades are meaningful only if the full story is available.
- Mutability: Grades can evolve when the articles are modified, or when later research sheds more light on the subject.
Sketching a rating system.
Let me sketch a system which would follow the above principles. Each article could have three grades, one in each of the following three categories:
A - major discovery
B - strong new results
C - incremental progress
D - not worth reading
A - proved beyond reasonable doubt
B - strong evidence
C - some evidence, but important doubts remains
D - main claims are unsupported
A - very satisfactory
B - some polishing would be welcome
C - important clarifications are needed
D - please rewrite article
Authors could be asked to rate their own articles. Their grades would not have the same status as grades from independent reviewers, but could still be helpful indications, especially when compared with authors' and reviewers' grades for other articles by the same authors.
The real problem.
Grades are only one aspect of a new scientific publishing model. For innovative platforms such as Publons or the Selected Papers Network, the crucial issue is probably to get people to publicly comment on each others' articles -- "publicly" meaning that the comments should be made public, not that their authors should always renounce anonymity. To solve this issue, it is necessary, but far from sufficient, to avoid making mistakes in the relatively simple matter of grades.