There are three ways to participate in grassroots reviewing.
- Make a comment on (a detail of) the article.
- Write a review.
- Write a synthesis assessment.
Everyone can write comments on the article, the reviews or the synthesis. These can be comments just like ones below a blog post, but with the software of Hypothesis it is very easy to select a part of the manuscript (PDF file or web page) and write a comment about it.
A comment can be short and only make a remark about one sentence or as long as a typical review and discuss the full manuscript. I would suggest that anonymous comments should be possible and that all comments will be pre-moderated. But that should be settings so that different communities can handle this differently.
I just read an open review manuscript of an EGU journal and thought something was wrong. Here the reviewers are anonymous, but anyone else making comments need to do this named. For a named comment I would have had to invest more time to be sure my comment made sense. Had an anonymous comment been possible, I had probably asked a question. The downside is that anonymous comments will likely have less quality and increase the number of comments and thus the work for the editors and authors.
Reviews would look similar to those written for traditional journals, but they would be published. Furthermore the reviewer would be asked to assess the importance of the paper, whether it should be published and to classify it.
The classification would mostly be journal dependent. The main categories could be reviews, methodological studies, replications, experimental and observational work. In statistical homogenisation I would suggest the categories: homogenisation methods, validation and error quantification, properties of inhomogeneities and applications. They can be detailed again.
Reviews can be published anonymously, but the editor of the article would need to know who it is to assess their expertise. (The other editors of the journal should not know this because they could be authors or have other conflicts of interests.) With named reviews the reviewer can take credit for their work, but my impression is that most prefer to be anonymous, that makes it easier to critique colleagues with which one still needs to collaborate in future. In addition, like for comments, writing a named review takes more time. After some time, for example ones a year, a tally can be made of how many articles someone published and reviewed to create some social reward for reviewing.
For the specific comments also reviewers can use hypothesis to make this easier. Reviews can be updated and are always possible (post-publication review).
Also partial reviews could be worthwhile, reviews that only assess part of a study, but do not make an overall assessment. For example, in case of a paper that homogenizes a climate dataset and goes on to analyse climatic changes, the reviewers of the journal on statistical homogenization could make an assessment on the quality of the homogenization, which can inform the editors of a more general climate journal in making their assessment of the complete study.
The synthesis would be written by one of the editors based on the reviews and their own expertise. The synthesis would look like the reviews: a general assessment, a classification and an assessment of the importance of the study.
In case the article is not published, I would suggest to keep the page up and list it on an index of rejected papers, rejections and its justification is useful information.
The synthesis does not have to be an average of the reviews, the editor can assess the strength of the reviews and reviewers often only have expertise on part of the paper.
The synthesis could be named or anonymous. I presume different communities have different preferences.