We are the watcher on the walls. We are the fire that burns against the cold, the light that brings the dawn, the horn that wakes the sleepers, the shield that guards the realms of men…for this night and all the nights to come.
OK, so I’m not Jon Snow (in that I’m not a bastard or disastrously handsome), and we are not waging war against white walkers. But in some respects, peer review is the last line of defence against bad science escaping into the world.
Peer review is a key part of the scientific process. Having a manuscript scrutinised by your peers, whether that is informally or formally, is a necessary step to check that your ideas make sense, and that your results are reliable. I’ve written before about how having your articles reviewed by experts in your field can be a gut-wrenching process, but if it is done well then your science is improved as a result.
However, the peer review process is far from perfect. Objectivity is often hard to achieve, and there are many weaknesses in using people from the same research field to review each others papers. Studies have shown that reviewers for example can be heavily influenced by the reputation of the author, and that the badge of peer-reviewed can stop people from looking more closely at the results of a study. In the medical sciences in particular, many articles that are often headline news are published with results that cannot be reproduced.
One suggested way to improve the process is to make every aspect of the study available to a reviewer, from the data to the code you used to produce the results. It is increasingly common now to see “source code available online” at the conclusion of a scientific paper. As much as this scares me and my often illegible scripts, this can only help to improve the transparency and reproducibility of research, as well as the coding skills of researchers.
Another way is to make the peer review process itself more transparent. Reviews are often given anonymously, which gives reviewers the ability to be lazy, or unnecessarily cruel in their judgement of an article. People have long have called for open peer revision, to make reviewers more accountable for their opinions. However, anonymity could be a good thing if you, for example, are a younger researcher giving a negative review to a senior professor in your field. In this way, anonymity protects the reviewer and gives her the freedom to voice her honest opinions.
Another idea is the notion of post-publication review, when an article is published, but then comments and reviews are attached, making the study more of a conversation between authors and reviewers. Organisations like The Winnower are promoting this idea, encouraging journal clubs and research groups to publish journal club discussions on their website.
A journal in the field of climate science that adheres to a similar method is Climate of the Past. This open access journal publishes the majority of its papers in Discussion format first. Any interested member of the community can register and post a comment, and several expert referees are also asked to provide more detailed reviews.
This week I was lucky enough to review one of these Discussion papers, on some recently recovered historical data for northern New Zealand. It was one of the first reviews I have done, and I was fascinated by the paper, so I did my very best to provide useful feedback for the authors.
I read the article once, on the train, to get a feel for the research and its story. I annotated with ticks, questions marks or circles, and the occasionally some notes. Then, I read it again, this time next to a computer, typing comments as I went and expanding on my earlier scribbles. I also separated the references, tables and figures from the rest of the article, so that I could easily to refer to the relevant images and data.
Next I checked over my comments, and combined them into tiny, minor and major comments. After two close reads of the article, I felt that I could also add some introductory statements about the paper, and my recommendation (which was to publish subject to minor revisions). The whole process would have taken about seven hours.
After checking and double checking my comments, I submitted them. And although it was an option, I chose not to be anonymous when I clicked submit. I chose to be nonymous.
I have to be honest, it was pretty terrifying as an inexperienced reviewer, to put my name to the review. What if I had done a terrible job? What if the editor and authors said to me “you know nothing Jon Snow” and had me immediately fired? What if I had gone too heavy on the en dash identification?? But I worked hard on the review and I was pretty certain that my comments would help improve the paper, even if just a little bit. And you know what? Yesterday I received a thank you from the author.
How do you review an article? Do prefer to stay anonymous, or put your name to your comments? Please, review my review of reviews below!