Friday 11 September 2015

Peer review: evolution, experiment and debate

John Sack, Founding Director of HighWire Press introduced the morning panel on peer review at the 2015 ALPSP Conference.

Dr Aileen Fyfe, PI of the ‘Publishing the Philosophical Transactions’ project at the University of St Andrews reflected on how Henry Oldenburg used an editorial driven model for Philosophical Transactions. The Royal Society approved all issues for publication. But it wasn't at the article level, it was to confirm there were no threats to the country: it was ratification of a sort.


In the 1760s considered papers by looking at the abstracts and would take a vote. This was to protect the reputation of the Society and did not check facts or reasoning. Meanwhile in France, the Académie royale des sciences asked academicians might be asked to report jointly on submitted papers on the truth claims being made in the papers. However, they did not judge each other, only outsiders. This high level checking scrutiny was abandoned in the 1780s as it was deemed too difficult. In the Philosophical Transactions in the 1860s referees who were a member of the society would make recommendations for publications. They provided literary comments about the article. The Fellows were writing about each other's work as well as outsiders. This now included a judgment on originality and significance.

The Philosophical Magazine in the 1920s:
Nature in the 1950s-70s would publish papers if they weren't actually wrong with erratic refereeing. They relied on papers that came from good institutions and/or known labs. In summary, the history of peer review is not as simple as you might imagine. Much better to understand this before we move forward to revise and update peer review going forward.

Dr John R Inglis, Executive Director and Publisher at Cold Spring Harbor Laboratory Press wryly noted there are many critics of modern peer review, but the prevailing view tends to be that it may not be perfect, but it's better than nothing. What do scientists think about peer review? Most are satisfied with it, think it helps scientific communication and think it has improved their papers. However, many think it can be improved, think it holds back science and now believe it is now unsustainable with increasing journals and science.

Most scientists think peer review should improve the quality of a paper, determine its originality and the importance of its findings. It helps to ensure previous work is acknowledged, help select the best papers for the journal and detect falsehoods. There are many ways that peer review is changing including double blinding, transparency, publishing reviews alongside papers, checking figures for manipulation, use specialised data, validating authors and reviewers and forbidding author-offered reviewers.

There are also changes to where peer review is done. often it is outsourced to peer review platforms like Rubriq, Peerage of Science, Editage and PubLons. There are also changes to when peer review is done. After publication there are a range of options to comment on papers: journal specific commenting functions, PubMeds Commons, PubPeer, ResearchGate and Academia.edu, ScienceOpen and SelectedPapers.net.

Cold Spring Harbor Press launched their pre-print server for biology bioRxiv. It's a not-for-profit free service that distributes draft papers for open comment. Posting is quick. Papers get a date stamp and a DOI. There is a commenting function and they link to the history of the evolution of the paper. It results in rapid transmission of results for community consideration. They have more than 2000 manuscripts posted from over 40 countries and more than 800 institutions. There are rising rates of submission and usage. Every subject category is respected and most manuscripts eventually appear in journals.

They know that 30% of papers have been revised and 33% of all papers have been published in more than 190 journals. They have extensive feedback via social media including 25,000 tweets. There are plans to make submission easier for authors. The use of pre-prints is changing. The behaviour of biologists is changing and journals policies are changing. Inglis closed by quoting the warnings contained in the Research Information Network report on Peer Review.

Dr Simon Kerridge is Director of Research Services, University of Kent and Chair of the Board of Directors at the Association of Research Managers and Administrators. Peer review is generally for journal articles, monographs and other long form research outputs, research data and other forms of scholarly output. From his point of view, it also includes research project proposals, research environment and strategy, and research impact. There are many purely academic reasons for doing peer review, but recognition is a part of it too. Promotion, esteem, time and money are all factors. Many universities have 'citizenship' as criteria for promotion where peer review is a factor. There may be internal or external mentoring or structured support. Becoming a journal editor or reviewer looks good on a CV.

By raising your profile you gain more recognition, but how is this recorded/advertised? Very few journals list reviewers. Some funders list 'peer review college' and most conferences list reviewers. Most academics list their own reviewing and universities try to keep full lists. There are internal work allocation models that provide recognition for peer review. Some journals reward reviewers with reduction in charges, 'peer review miles' to offset future fees and other waivers. Some conferences have reductions and some funders do pay reviewers or their institution. There are some universities that pay bonuses for peer review (e.g. REF peer review panel). With internal peer review it is unlikely you will get paid.
Dr Kirsty Edgar, Leverhulme Early Career Fellow at the University of Bristol provided an early career researcher (ECR) point of view on peer review. She reflected that it always seems to be the third review that's bad! ECRs want to improve research, get academic seal of approval, improve the dissemination of the research. But most importantly, they want to get through the process, publish in the highest impact journal possible, improve their CV and get a job.

There are several issues. There is little in the way of support or training, although this is improving. Are you getting a fair deal as a reviewee? Will you get promotion or fellowship? Will people read your work and will you be able to afford to publish my work?

There are some solutions: improve training or change the system in a small way such as peer choice, cascading reviews and open peer reviews. You can also fundamentally change the system by getting rid of journals, or on a slightly less radical agenda, introduce pre-publication, data and post-publication review. Edgar cited the eLife model of the support they provide to early career researchers. Edgar closed with some recommended reading: Sense About Science, the Voice of Young Science blog by James Steele and the BioMed Central Blog by Sarah Hayes.


Posted by at
Labels: , , , , , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)