We are the watcher on the walls. We are the fire that burns against the cold, the light that brings the dawn, the horn that wakes the sleepers, the shield that guards the realms of men…for this night and all the nights to come.
OK, so I’m not Jon Snow (in that I’m not a bastard or disastrously handsome), and we are not waging war against white walkers. But in some respects, peer review is the last line of defence against bad science escaping into the world.
Peer review is a key part of the scientific process. Having a manuscript scrutinised by your peers, whether that is informally or formally, is a necessary step to check that your ideas make sense, and that your results are reliable. I’ve written before about how having your articles reviewed by experts in your field can be a gut-wrenching process, but if it is done well then your science is improved as a result.
However, the peer review process is far from perfect. Objectivity is often hard to achieve, and there are many weaknesses in using people from the same research field to review each others papers. Studies have shown that reviewers for example can be heavily influenced by the reputation of the author, and that the badge of peer-reviewed can stop people from looking more closely at the results of a study. In the medical sciences in particular, many articles that are often headline news are published with results that cannot be reproduced.
One suggested way to improve the process is to make every aspect of the study available to a reviewer, from the data to the code you used to produce the results. It is increasingly common now to see “source code available online” at the conclusion of a scientific paper. As much as this scares me and my often illegible scripts, this can only help to improve the transparency and reproducibility of research, as well as the coding skills of researchers.
Another way is to make the peer review process itself more transparent. Reviews are often given anonymously, which gives reviewers the ability to be lazy, or unnecessarily cruel in their judgement of an article. People have long have called for open peer revision, to make reviewers more accountable for their opinions. However, anonymity could be a good thing if you, for example, are a younger researcher giving a negative review to a senior professor in your field. In this way, anonymity protects the reviewer and gives her the freedom to voice her honest opinions.
Another idea is the notion of post-publication review, when an article is published, but then comments and reviews are attached, making the study more of a conversation between authors and reviewers. Organisations like The Winnower are promoting this idea, encouraging journal clubs and research groups to publish journal club discussions on their website.
A journal in the field of climate science that adheres to a similar method is Climate of the Past. This open access journal publishes the majority of its papers in Discussion format first. Any interested member of the community can register and post a comment, and several expert referees are also asked to provide more detailed reviews.
This week I was lucky enough to review one of these Discussion papers, on some recently recovered historical data for northern New Zealand. It was one of the first reviews I have done, and I was fascinated by the paper, so I did my very best to provide useful feedback for the authors.
I read the article once, on the train, to get a feel for the research and its story. I annotated with ticks, questions marks or circles, and the occasionally some notes. Then, I read it again, this time next to a computer, typing comments as I went and expanding on my earlier scribbles. I also separated the references, tables and figures from the rest of the article, so that I could easily to refer to the relevant images and data.
Next I checked over my comments, and combined them into tiny, minor and major comments. After two close reads of the article, I felt that I could also add some introductory statements about the paper, and my recommendation (which was to publish subject to minor revisions). The whole process would have taken about seven hours.
After checking and double checking my comments, I submitted them. And although it was an option, I chose not to be anonymous when I clicked submit. I chose to be nonymous.
I have to be honest, it was pretty terrifying as an inexperienced reviewer, to put my name to the review. What if I had done a terrible job? What if the editor and authors said to me “you know nothing Jon Snow” and had me immediately fired? What if I had gone too heavy on the en dash identification?? But I worked hard on the review and I was pretty certain that my comments would help improve the paper, even if just a little bit. And you know what? Yesterday I received a thank you from the author.
How do you review an article? Do prefer to stay anonymous, or put your name to your comments? Please, review my review of reviews below!
lindenashcroft.com 
As reviewer I prefer anonymous reviews. It is a voluntary job, which already takes a lot of time and if it could also have negative repercussions, I think it would be hard to get people to do the job. If I am positive about a manuscript, it is fine to have the option of writing a named review, but I would personally not do so when I feel a manuscript should be rejected, no matter how clear the situation is, maybe even more so in such a situation.
I say this, even if I had quite some bad experiences with anonymous reviews as author and at the time wrote a post against anonymous peer review. The main protection against such abuses of peer review should be the editors. And ironically they are normally named, but often do their job miserably.
Anonymity is limited. Once you know the field, you can often guess quite well who the reviewer is. People have their pet topics and peeves. Fortunately, such guesses are not perfect and that already deflects much anger. A few days ago I asked a colleague for a paper and he replied he thought I had been the reviewer and thus had it already. No, I was not. 🙂 Once a colleague guessed I was the reviewer of a rejected manuscript and he still talks with me. But it was just a guess (which I will deny nor confirm).
LikeLike
But in some respects, peer review is the last line of defence against bad science escaping into the world.
Peer review is just one of the many filters that help find credible information.
Before review are hiring the right people, finding co-authors (supervisors) willing to risk their reputation on the paper, presenting the work at conferences and surviving the questions.
After review it are getting comments or a full rebuttal article in response, (which will likely only happen if the erroneous article is still somehow influential), blogs (social media) discussing the article, newspaper articles and quotes from colleagues (for publicly interesting papers), being cited and thus more visible.
There is a trade off between letting “bad science escape” and letting interesting ideas be known.
LikeLike
Thanks for sharing your wisdom Victor! I really enjoyed your blog posts on peer review, and your changing opinions over time. I too see-saw a bit on the idea of anonymous reviewers. In my experience, you can often tell who a reviewer is, even if they are anonymous, as I think would have been the case with my recent review. Not putting your name to a review allows a level of honesty, particularly for younger researchers or negative reviews, but then putting your name at the end of a review means that you have to be willing to back up your opinions. I will keep monitoring my opinion as my reviewer experience grows.
As for your second comment I agree, there are many other ‘watchers on the wall’, not just the official peer review process. Sharing drafts with colleagues, presenting preliminary work at conferences, talking with people at morning tea are all a form of peer review in some way. The post-publication reviews you mention are also increasingly important things for researchers to consider. I feel as though scientists used to used peer review as more of a shield than they do now.
LikeLike