Even if you’ve never been involved in scientific research, you’re probably aware that it involves a process called “peer review”. I want to take a minute to explain how this actually works. This is based on my personal experience, although I think much of it generalizes to other academic fields, including those outside of science.

1. Sending to referees

It starts with the submission of a manuscript to a journal. A lot of work has already gone into the manuscript, including input from collaborators and colleagues, but this is where peer review formally begins.

The journal assigns the manuscript to an editor, and then the editor chooses a few (usually 3) referees to look at the paper. Now, choosing referees can be quite difficult, because they need to be close enough to the field that they can understand and critique the manuscript. In fact, it’s common for referees to decline, because they think the manuscript is too far outside their field. And yet, referees can’t be so close that they’re direct competitors. Authors typically provide a list of competitors to the editor to avoid conflict of interest (or even worse, theft of ideas). But editors aren’t required to follow this advice, and authors never know because they don’t know the names of the referees.

2. Referee reports

Now, it’s the referees’ turn. They read the manuscript closely and critically (or get their underlings to do it). They summarize the paper as they understand it, discuss any problems that they spotted, and make a recommendation as to whether the paper should be accepted. Typical critiques include:

• There’s another interpretation that should be considered.
• The interpretation conflicts with some aspect of the data.
• The manuscript conflicts with [citation to paper], and this should be addressed.
• The data are noisy and the claimed trend isn’t apparent.
• The manuscript is unclear or difficult to understand.
• The study isn’t important enough to be published in this prestigious journal, either because it doesn’t have any wider implications, or because it’s already been done.

The editor takes the referee reports into consideration, and may choose to accept or reject the manuscript. Even when referee reports are mixed, the editor can exercise their own judgment based on which referees they trust the most, and which ones make the most substantive arguments. But typically, the editor doesn’t make a decision yet, and instead sends the referee reports to the authors for a reply.

3. Reply to referees

The authors look at the referee reports, and write a letter replying to each and every point made by the referees. For each point, there is almost always some adjustment made to the manuscript. The idea is that even if a referee says something off-base, it could represent a common misunderstanding which should be addressed directly in the paper. While there can be overhauls to the manuscript, may changes are quite small. For instance, if a referee suggests an alternate interpretation, the authors might simply add a single sentence with a single citation.

This letter is unwaveringly polite and gracious, but I can say from personal experience that the authors don’t necessarily feel so gracious. It’s common for authors to vent frustrations to colleagues. “Referee A clearly has an axe to grind and just wants to sink our paper. Referee C just has no idea what they’re talking about.” And many researchers will swap stories about referees being unfair to them. If authors have a major problem with a particular referee, they might privately communicate their concerns to the editor, who might quietly drop the referee.

Once the replies are sent to the editor, the editor may accept, reject, or send it out for more referee responses. Typically, there are 1-3 rounds of referee responses.  This can take several months.

4a. Acceptance and publication

If the paper is accepted, that’s when the authors celebrate. Grad students might publish around one paper a year, so it’s a significant event.

The paper isn’t published immediately. It’s sent to copy-editors, who format it, and make corrections in accordance to obscure rules in the journal’s style guide. This is called the “proofing” stage, and while it may involve a few iterations of correspondence, it’s no big deal.

Once the paper is published, you sit back and watch it collect citations (or not, as is more commonly the case). I should note that while the formal peer review process has concluded, there’s also a process by which researchers can publish a “Comment” on a published paper, usually to say something very critical. This has never happened to me.

4b. Rejection and followup

If the manuscript is rejected, there are a few things authors can do. There was one time that we tried to appeal, because the editor had rejected our manuscript based on three positive reports, and one critical one. But the appeal didn’t work and I don’t think this is very common.

Most often, authors simply submit to another journal. If the problem was that the study wasn’t interesting enough, it might be submitted to a less prestigious journal. If referees raised more serious concerns, the manuscript might undergo major revisions. Of course, you can’t just submit and resubmit a manuscript forever. The process is exhausting, and I imagine at some point the same referees are looking at it repeatedly. Eventually the study is dropped in favor of other priorities, or transformed into something completely different.

Final notes

I want to emphasize that peer review is an imperfect process. Papers can quite easily become rejected or accepted unfairly. It can be hard to find people with the necessary expertise to be a referee. And if referees are too close to the topic, they have a lot of power to sabotage their rivals. Even when everything goes right in peer review, it is in no way an confirmation of the paper’s conclusions. That just means it passes minimal standards of rigor.