Peer review: evolution, experiment and debate

John Sack, Founding Director of HighWire Press introduced the morning panel on peer review at the 2015 ALPSP Conference.

Dr Aileen Fyfe, PI of the ‘Publishing the Philosophical Transactions’ project at the University of St Andrews reflected on how Henry Oldenburg used an editorial driven model for Philosophical Transactions. The Royal Society approved all issues for publication. But it wasn't at the article level, it was to confirm there were no threats to the country: it was ratification of a sort.

Peer review a la française 1760s - fascinating presentation from @aileenfyfe #alpsp15 pic.twitter.com/IvpaphGjkr

— Martyn Lawrence (@MartynLawrence) September 11, 2015

In the 1760s considered papers by looking at the abstracts and would take a vote. This was to protect the reputation of the Society and did not check facts or reasoning. Meanwhile in France, the Académie royale des sciences asked academicians might be asked to report jointly on submitted papers on the truth claims being made in the papers. However, they did not judge each other, only outsiders. This high level checking scrutiny was abandoned in the 1780s as it was deemed too difficult. In the Philosophical Transactions in the 1860s referees who were a member of the society would make recommendations for publications. They provided literary comments about the article. The Fellows were writing about each other's work as well as outsiders. This now included a judgment on originality and significance.

The Philosophical Magazine in the 1920s:

Fyfe: 20th century review didn't necessarily imply validation or agreement. Published as contribution to debate #alpsp15

— Alice Meadows (@alicejmeadows) September 11, 2015

Nature in the 1950s-70s would publish papers if they weren't actually wrong with erratic refereeing. They relied on papers that came from good institutions and/or known labs. In summary, the history of peer review is not as simple as you might imagine. Much better to understand this before we move forward to revise and update peer review going forward.

Dr John R Inglis, Executive Director and Publisher at Cold Spring Harbor Laboratory Press wryly noted there are many critics of modern peer review, but the prevailing view tends to be that it may not be perfect, but it's better than nothing. What do scientists think about peer review? Most are satisfied with it, think it helps scientific communication and think it has improved their papers. However, many think it can be improved, think it holds back science and now believe it is now unsustainable with increasing journals and science.

Most scientists think peer review should improve the quality of a paper, determine its originality and the importance of its findings. It helps to ensure previous work is acknowledged, help select the best papers for the journal and detect falsehoods. There are many ways that peer review is changing including double blinding, transparency, publishing reviews alongside papers, checking figures for manipulation, use specialised data, validating authors and reviewers and forbidding author-offered reviewers.

There are also changes to where peer review is done. often it is outsourced to peer review platforms like Rubriq, Peerage of Science, Editage and PubLons. There are also changes to when peer review is done. After publication there are a range of options to comment on papers: journal specific commenting functions, PubMeds Commons, PubPeer, ResearchGate and Academia.edu, ScienceOpen and SelectedPapers.net.

Inglis. Some stats on bioRxiv... Post print feedback is growing #ALPSP15 pic.twitter.com/mKIuKPm7DF

— Simon Kerridge (@SimonRKerridge) September 11, 2015

Cold Spring Harbor Press launched their pre-print server for biology bioRxiv. It's a not-for-profit free service that distributes draft papers for open comment. Posting is quick. Papers get a date stamp and a DOI. There is a commenting function and they link to the history of the evolution of the paper. It results in rapid transmission of results for community consideration. They have more than 2000 manuscripts posted from over 40 countries and more than 800 institutions. There are rising rates of submission and usage. Every subject category is respected and most manuscripts eventually appear in journals.

They know that 30% of papers have been revised and 33% of all papers have been published in more than 190 journals. They have extensive feedback via social media including 25,000 tweets. There are plans to make submission easier for authors. The use of pre-prints is changing. The behaviour of biologists is changing and journals policies are changing. Inglis closed by quoting the warnings contained in the Research Information Network report on Peer Review.

Dr Simon Kerridge is Director of Research Services, University of Kent and Chair of the Board of Directors at the Association of Research Managers and Administrators. Peer review is generally for journal articles, monographs and other long form research outputs, research data and other forms of scholarly output. From his point of view, it also includes research project proposals, research environment and strategy, and research impact. There are many purely academic reasons for doing peer review, but recognition is a part of it too. Promotion, esteem, time and money are all factors. Many universities have 'citizenship' as criteria for promotion where peer review is a factor. There may be internal or external mentoring or structured support. Becoming a journal editor or reviewer looks good on a CV.

By raising your profile you gain more recognition, but how is this recorded/advertised? Very few journals list reviewers. Some funders list 'peer review college' and most conferences list reviewers. Most academics list their own reviewing and universities try to keep full lists. There are internal work allocation models that provide recognition for peer review. Some journals reward reviewers with reduction in charges, 'peer review miles' to offset future fees and other waivers. Some conferences have reductions and some funders do pay reviewers or their institution. There are some universities that pay bonuses for peer review (e.g. REF peer review panel). With internal peer review it is unlikely you will get paid.

Questions for the peer review panel. #alpsp15 pic.twitter.com/nx1XxhnKk5

— Suzanne Kavanagh (@sashers) September 11, 2015

Dr Kirsty Edgar, Leverhulme Early Career Fellow at the University of Bristol provided an early career researcher (ECR) point of view on peer review. She reflected that it always seems to be the third review that's bad! ECRs want to improve research, get academic seal of approval, improve the dissemination of the research. But most importantly, they want to get through the process, publish in the highest impact journal possible, improve their CV and get a job.

There are several issues. There is little in the way of support or training, although this is improving. Are you getting a fair deal as a reviewee? Will you get promotion or fellowship? Will people read your work and will you be able to afford to publish my work?

There are some solutions: improve training or change the system in a small way such as peer choice, cascading reviews and open peer reviews. You can also fundamentally change the system by getting rid of journals, or on a slightly less radical agenda, introduce pre-publication, data and post-publication review. Edgar cited the eLife model of the support they provide to early career researchers. Edgar closed with some recommended reading: Sense About Science, the Voice of Young Science blog by James Steele and the BioMed Central Blog by Sarah Hayes.

ALPSP Awards spotlight on... bioRxiv, the preprint sharing service for the life sciences from Cold Spring Harbor Laboratory

Nice place you've got there, Cold Spring Harbor Laboratory

With only a few weeks to go before our annual September conference, thoughts turn to the finalists of the ALPSP Awards for Innovation in Publishing, sponsored by Publishing Technology. The winners will be announced at the conference, but if you can't wait until then, here's the first in a series of spotlights on the finalists.

We asked John Inglis, Executive Director of Cold Spring Harbor Laboratory Press, to tell us more about finalist project bioRxiv, the preprint sharing service for the life sciences that the Laboratory initiated in November 2013.

ALPSP: Tell us a bit about your company

JI: Cold Spring Harbor Laboratory (www.cshl.edu) is located 35 miles east of New York, on the north shore of Long Island.  It was founded in 1890 and was historically important in the development of genetics and molecular biology last century.

Today it has first-class research programs in cancer, neuroscience, genomics, plant science, and quantitative biology.  But the Laboratory also makes a huge contribution to scholarship and the professional education of scientists through highly regarded conferences on its main campus and its center in Suzhou, China, laboratory and lecture courses, an innovative graduate school, a center and archive for the history of molecular biology, and the outstanding journals, books, and electronic media produced by its Press (www.cshlpress.org).  

So the Laboratory's long-established culture combines both doing science and sharing science with many thousands of investigators who visit our campuses and an even larger number worldwide who benefit from its publications.

bioRxiv metrics

ALPSP: You have submitted the bioRxiv project for the Awards. Tell us more about how it works and the team behind it.

JI: A preprint is the manuscript of a research paper that has not yet been accepted for publication by a journal. bioRxiv is a service for the open online distribution of preprints. It's free for both authors and readers. Authors categorize their manuscript in one of 25 life science subject categories and identify their findings as "new", "confirmatory", or "contradictory". The manuscript is submitted through an automated system that creates a pdf, attaches a DOI, and displays the abstract in the bioRxiv site. The pdf can be previewed and downloaded. Manuscripts are posted usually within hours of submission, with only minimal screening and no peer review. They can be revised any time and prior versions are retained on the bioRxiv site. Usage metrics and altmetrics for each posted manuscript are publicly available. When a version of a manuscript is published by a journal, a link is posted from the preprint to the paper.

bioRxiv is a service of Cold Spring Harbor Laboratory managed by a small team within CSHL Press that includes co-founder Richard Sever, production manager Linda Sussman, lead technical developer Ted Roeder, and administrator Inez Sialiano.  bioRxiv is hosted by HighWire Press, where it gets great support from Bert Carelli and the drupal development team and Kevin-John Black and the Bench>Press crew.  We also rely very much on a group of about 50 bioRxiv Affiliates, young principal investigators who screen submitted manuscripts and assure us that the content is indeed science.

ALPSP: Why do you think it demonstrates publishing innovation?

JI: There is technical innovation in the automated system that ingests and posts manuscripts with minimal human intervention. But much more importantly, we see evidence that bioRxiv is beginning to change information practices within the broad sweep of life science. The long established online preprint server arXiv covers physics, computer science,  mathematics, and since 2003, areas of biology with a quantitative component, such as population genetics. In only 8 months, bioRxiv has received more than 550 manuscripts and every one of the life sciences categories is represented. As expected, the largest categories are the more quantitative, such as evolutionary biology and bioinformatics, in which preprints are familiar but a substantial number of manuscripts are in cancer, genetics, cell and developmental biology, neuroscience, immunology, ecology  -  areas of science in which sharing preprints is a new behavior among authors.

Publication link

There are changes, too, among journals.  Since bioRxiv began, many journals and publishers have either changed or clarified their policies to state openly that they will accept for publication manuscripts that have been shared as preprints. 20% of the manuscripts currently on bioRiv have been published in final form in a long list of journals that include the leading general journals and prominent titles in more specific areas. 

These journals understand and support the distinction between the sharing of information that bioRxiv makes possible and the validation of knowledge through peer review, a function that still resides with journals.

Another area of innovation is the enabling of prepublication critical scrutiny and discussion of findings that bioRxiv makes possible. There is a commenting function on the site which so far has been used only for a small number of papers. But when it happens, there can be useful dialogue between authors and readers.  There is a lot of social media activity around bioRxiv posts, including twitter discussion and blogging. And authors tell us they get a great deal of direct, private comment on manuscripts through email. It's our impression, therefore, even at this early stage in the life of the service, that posting a manuscript to bioRiv offers the author an opportunity for community feedback that could improve the manuscript before it's submitted to a journal for publication. And perhaps, therefore, speed its acceptance, although of course we have no evidence for that.

In summary, bioRxiv is changing the way information in the life sciences is shared.  It's helping quicken the pace of research.  It's empowering early career scientists, who can share their results and gain visibility before they have built a publication record.  And it's enabling investigators to create and share a citable, discoverable record of work accomplished that can be evaluated by grants and hiring committees, even though it has not yet been formally published.

The Laboratory's mission

ALPSP: And finally, what are your plans for the future?

JI: The Laboratory is committed to bioRxiv for the long term but we're impatient to spread the word about the availability of the service as rapidly as possible. We still have much to do to make scientists worldwide aware of the service and its benefits. We are still adding features and functions to the site and next up will be some general data about the relative distribution of preprints among subject categories and the journals that have published final versions. We are talking to publishers about their preprint policies and where appropriate, advocating for change. We may well develop a list of preprint-friendly journals on the bioRxiv site.  Several publishers are interested in offering authors the possibility of posting their manuscript to bioRxiv at the same time it's submitted to a journal. We have tested this functionality in the Bench>Press submission system with one of the CSHL Press research journals Genome Research and many authors liked it, so we're working now with other journals that use B>P and would like to offer the opportunity via other submission systems. Other things we are considering include a rating system that readers can attach to posted manuscripts.

We are excited about the rising rate of submission, increasing usage, extraordinarily interesting and important science being posted, and growing awareness and discussion of the information being shared through bioRxiv.  Its future will be driven in a major way by what the community tell us they want the service to be.

The ALPSP Awards for Innovation in Publishing will be made at the ALPSP International Conference. Book now to reserve your place.