Peer review is a critical part of the publishing process at JM3, as it is for most science journals. We require a minimum of two independent reviews before we will accept a manuscript, though it remains the editor’s decision on whether a manuscript is ultimately accepted or rejected. There are many kinds of peer review, so to be specific JM3 practices an editor-driven external peer review of author-submitted manuscripts. Reviewers (also called referees) are blinded, meaning that authors never know the identity of the reviewers. I’ll write more about the overall editorial process at JM3 in my next editorial, but here I’d like to focus specifically on the reviewing of a manuscript.
The peer-review process serves two immediate goals: to help editors decide which manuscripts to publish and which to reject (filtering), and to give authors advice on how to improve their papers (criticism). Additionally, the “stamp of approval” of being published in a peer-reviewed journal can aid authors in their careers, as well as having many other benefits. But it is my philosophy that everything about the science publishing enterprise should be focused on the reader, and so it is with the peer-review process. The filtering and criticism that accompanies an editorial peer-review process helps to get the best papers into the hands of the most interested readers efficiently.
But for the peer-review process to fulfill its goals, the reviews must be of good quality. What constitutes a quality review? Alas, I suspect that none of us has ever been trained in proper science-paper reviewing—we generally figure it out through experience. Anyone who has published a fair number of papers knows that some reviews are of much higher quality than others (independent of the ultimate fate of any given manuscript). A good review teaches the author about writing and about science, resulting not only in one better paper, but in making every subsequent paper the author writes better. It also makes the job of the editor significantly easier. A bad quality review does none of this.
Over the last three years I’ve written a series of editorials on what it takes to write a good scientific paper.1 These topics constitute a reasonable list of things a reviewer should be looking for in any paper that might hope to be published. Appended to this editorial is a summary of the advice I gave in those editorials, organized in the form of a checklist.
To be clear, neither editors nor reviewers need to use a formal checklist when writing a review. The attached list is a guideline to help both editors and reviewers make sure that the most important aspects of a scientific paper are considered. As one might expect, the checklist also happens to be a great list of things an author should consider before submitting a manuscript. It is always good advice for an author to think like a reader, and the first readers will be the editors and reviewers.
After reading and critically evaluating a manuscript, the reviewer must now convey that evaluation to the journal editors. In all cases, a respectful and constructive tone should be used. The format of a review is not critical, but each review should contain certain vital information. The first paragraph should contain these three key points:
• Provide a brief (1-2 sentence) synopsis of the paper
• Explain what is novel in this paper (1-2 sentences), both what the authors claim and your assessment
• Explain why the work is significant, or not (1-2 sentences)
If the reviewer finds it difficult to put any or all of these points into one or two sentences, chances are the manuscript has not done a good job conveying its key messages—a potential red flag.
The second paragraph should give an overview of the quality of the research being reported. If there are any significant flaws in the logical progression from method to data to analysis to conclusions, bring them up here and what could be done to fix the flaws. In this paragraph, focus on the big issues (if there are any). If all is good, say so.
The third and final section of the review should be a list of specific points that the author should address. These points can be small or large, from graphics formatting to paper organization. Remember, though, that copyediting will be done by the journal staff after acceptance, so don’t worry about language or format issues unless they interfere with your ability to properly understand and review the manuscript, or if improper language causes what is said to deviate from what is meant.
What does a bad quality review look like? A list of generic complaints or conclusions without specific references to the details of the manuscript is not very helpful (for example, saying that the work is not novel but not providing any example prior publications that cover the same topic). The worst kind of review is one that simply states the reviewer’s accept/reject conclusion. This is essentially of no value to an editor.
Reviewers are absolutely essential to the success of a peer-reviewed science journal. Reviewers volunteer their valuable time (typically 3–8 hours per manuscript) for no obvious benefit, other than the altruistic goal of giving back to their community. For all those who have contributed reviews to JM3, I thank you. Perhaps this editorial, with the attached checklist, will make your job a little easier next time.
1.
Should the manuscript be rejected?
Reject the manuscript if one or more of the answers to the following questions is no. Support all no answers with specific reasons.
• Does the content of the manuscript match the scope of the journal?
If no: Is there a journal with a better match?
• Does the manuscript present novel results (with the exception of review papers and the like)?
If no: Did the author(s) fail to distinguish what was novel? Where was similar content published?
• Are the results significant enough to be worth reading about (and thus worth publishing)? Will it impact the thoughts or actions of its readers?
If no: Is it possible to increase the significance with more data, different analysis, improved theoretical treatment, etc.? Would a different audience (different journal) find the work more significant?
• Does the data support the conclusions (i.e., is the quality of the research sufficiently high)?
If no: Can the conclusions be scaled back to what the data allow, and if so, would the results still be significant? Is it possible to add more data/theoretical treatment/etc. to enable the conclusions to be supported?
• Is the writing of sufficient quality to allow the above points to be evaluated?
If no: What suggestions would help the author(s) get the manuscript in better shape (e.g., English-language editing, better organization, etc.)?
2.
If the manuscript is not rejected, what should be changed to make it acceptable for publication?
Reviewers can use the following checklist as a guide for creating a comprehensive review of the work, with suggestions for improvements. For authors, asking the questions and following the instructions below will result in a paper more likely to be accepted for publication.
2.1.
Organization, Length, and Clarity
• Is the work well organized and structured so that conclusions logically follow from results that logically follow from the methods used? Do those conclusions answer the research questions initially posed?
• Make sure the length of the manuscript is appropriate. Does the knowledge gained by the reader justify the time spent reading?
• Is the thought process clear? Is clear language used (claiming neither more nor less than can be justified)?
2.2.
Introduction
• Indicate the field of the work, why this field is important, and what has already been done (with proper citations).
• Indicate a gap, raise a research question, or challenge prior work in this territory.
• Outline the purpose and announce the present research, clearly indicating what is novel and why it is significant.
• Avoid: repeating the abstract; providing unnecessary background information; exaggerating the importance of the work; claiming novelty without a proper literature search.
2.3.
Method (Materials, Theory, Design, Modeling, etc.)
• Describe how the results were generated with sufficient detail so that an independent researcher (working in the same field) could reproduce the results sufficiently to allow validation of the conclusions.
• Can the reader assess internal validity (conclusions are supported by the results presented)?
• Can the reader assess external validity (conclusions are properly generalized beyond these specific results)?
• Has the chosen method been justified?
• Are data analysis and statistical approaches justified, with assumptions and biases considered?
• Avoid: including results in the Method section; including extraneous details (unnecessary to enable reproducibility or judge validity); treating the method as a chronological history of what happened; unneeded references to commercial products; references to “proprietary” products or processes unavailable to the reader.
2.4.
Results and Discussion
• Present the results of the paper, in logical order, using tables and graphs as necessary.
• Explain the results and show how they help to answer the research questions posed in the Introduction. Evidence doesn’t explain itself; the results must be presented and then explained.
• Typical stages in the discussion: summarizing the results, discussing whether results are expected or unexpected, comparing these results to previous work, interpreting and explaining the results (often by comparison to a theory or model), and hypothesizing about their generality.
• Discuss any problems or shortcomings encountered during the course of the work.
• Discuss possible alternate explanations for the results.
• Avoid: presenting results that are never discussed; presenting discussion that doesn’t relate to any of the results; presenting results and discussion in chronological order rather than logical order; ignoring results that don’t support the conclusions; drawing conclusions from results without logical arguments to back them up.
2.5.
Conclusions
• Provide a very brief summary of the Results and Discussion.
• Emphasize the implications of the findings, explaining how the work is significant and providing the key message(s) the author wishes to convey.
• Provide the most general claims that can be supported by the evidence.
• Provide a future perspective on the work.
• Avoid: repeating the abstract; repeating background information from the Introduction; introducing new evidence or new arguments not found in the Results and Discussion; repeating the arguments made in the Results and Discussion; failing to address all of the research questions set out in the Introduction.
2.6.
Acronyms
• The title should not use acronyms unless (a) the subject is almost exclusively known by its acronym or is widely known and used in that form, and (b) the acronym does not commonly have more than one expansion.
• Always spell out the acronym the first time it is used in the body of the paper.
• Avoid acronyms in the abstract unless the acronym is commonly understood and used multiple times in the abstract. If an acronym is used in the abstract, it must be spelled out (defined) in the abstract, and then spelled out again the first time it is used in the body of the paper.
2.7.
Citations (References)
• Include citations that provide sufficient context to allow for critical analysis of this work by others.
• Include citations that give the reader sources of background and related material so that the current work can be understood by the target audience.
• Include citations that provide examples of alternate ideas, data, or conclusions to compare and contrast with this work, if they exist. Don’t exclude contrary evidence.
• Include citations that acknowledge and give credit to sources relied upon for this work.
• Are the citations up to date, referencing that latest work on this topic?
• It is the job of the authors to verify the accuracy of the references.
• Avoid: spurious citations (citations that are not needed but are included anyway); biased citations (references added or omitted for reasons other than meeting the above goals of citations); excessive self-cites (citations to one’s own work).
2.8.
Figures and Tables
• Ensure that the figures accurately and carefully document the data and their context.
• Ensure that the figures allow for comparisons and inferences of cause and effect, avoiding spurious readings.
• Figures should have captions and legends to allow them to be understood independent of the text, if possible.
• Ideally, a figure caption will do three things: describe everything in the graph, draw attention to its important features, and (when practical) describe the main conclusions to be drawn from it.
• All figures should be referred to in the text, with first references in numerical order.
• A piece of data has four parts: a description (what is it?), a number, a unit, and an uncertainty estimate. Try to put all four parts of the data in the figure.
• Error bars should be present; explain clearly what they represent. If any data points have been removed, explain.
• By all means, use color when it can enhance the graphic (since most articles are now read on-line), but make sure that no information is lost when printed in black and white.
• Tables are best for looking up specific information or exact values, and graphs excel at displaying trends and making comparisons.
• When the number of data points is small, a table generally is preferred over a graph.
• Use log-scales to reveal trends in the data, not hide them. Log-scales emphasize relative changes, while linear scales are best at showing absolute changes.
• Choose plot scales (x- and y-axis start and stop values, for example) to avoid white space: try to use at least 80% of each scale to display data.
• Avoid: titles on the graph (title information should be in the figure caption); pie charts; bar charts unless there isn’t a better option; spurious 3D effects, such as the use of 3D bars in a bar chart; gridlines and other clutter; inconsistent formatting of figures; commercial displays in the guise of diagrams or figures.
2.9.
Abstract
• The abstract should be a concise (200 words or less), stand-alone summary of the paper, with 1-2 sentences on each of these topics:
• Background: What issues led to this work? What is the environment that makes this work interesting or important?
• Aim: What were the goals of this work? What gap is being filled?
• Approach: What went into trying to achieve the aims (e.g., experimental method, simulation approach, theoretical approach, combinations of these, etc.)? What was actually done?
• Results: What were the main results of the study (including numbers, if appropriate)?
• Conclusions: What were the main conclusions? Why are the results important? Where will they lead?
• The abstract should be written for the audience of this journal: don’t assume too much or too little background with the topic.
• Ensure that all of the information found in the abstract also can be found in the body of the paper.
• Ensure that the important information of the paper is found in the abstract.
• Avoid: using the first paragraph of the introduction as an abstract; citations in the abstract; acronyms (but if used, spell them out); referring to figures or tables from the body of the paper; use of the first person; use of words like “new” or “novel,” or phrases like “in this paper,” “we report,” or “will be discussed.”
2.10.
Title
• The title should be clear and informative and should reflect the aim and approach of the work.
• The title should be as specific as possible while still describing the full range of the work. Does the title, seen in isolation, give a full yet concise and specific indication of the work reported?
• Don’t mention results or conclusions in the title.
• Avoid: overly clever or punny titles that will not fare well with search engines or international audiences; titles that are too short to be descriptive or too long to be read; jargon, acronyms, or trade-marked terms.
