Spotlight on Publons - shortlisted for the 2017 ALPSP Awards for Innovation in Publishing

On the 14th of September we will be anouncing the winner of the 2017 ALPSP Awards for Innovation in Publishing, sponsored by MPS Limited at the 10th Anniversary ALPSP Conference.  In this series of posts leading up to the Awards ceremony we meet our six finalists and get to know a bit more about them.  

First up, we speak to Andrew Preston, Co-Founder and Managing Director of Publons. 

Tell us a bit about your company

Publons is the home of peer review. Our cross-publisher platform collects peer review activity, providing researchers with a verified record of all of their reviewing contributions. Publons makes it possible to:

●     Recognise the contributions of reviewers worldwide

●     Improve and expand the reviewer pool through our free online training course

●     Gain novel insights into global peer review and researcher behaviour

●     Find, screen, contract, and motivate reviewers (an important part of an editor's job)

All of these threads feed into our mission to speed up research through the power of peer review.

 

What is the project that you submitted for the Awards?

We submitted the Publons reviewer recognition service. This is essentially what you find on Publons.com and comprises of a free offering for researchers and a paid service for publishers.

 

Tell us more about how it works and the team behind it

Researchers sign up to create a free account. Publons then makes it really easy to import, verify, and store a record of every peer review they've performed and every manuscript they've handled as an editor, across any journal in the world. Researchers also have access to personal statistics and a downloadable, verified review record, which provides evidence of their service and standing in their field.

Records can be added regardless of whether the manuscript goes on to be published. Publons works with researchers and journals to set policies about what information can be displayed publicly. We support all review policies, ranging from completely open to double-blind, and we also enable post-publication review.

Partner journals are able to add Publons to their peer review workflow. Every time a researcher reviews for one of our 1,350 partner journals they are able to opt in to have their review seamlessly added to their Publons record. Researchers love this, making it really easy for a publisher to provide an enhanced service to reviewers.

 

Why do you think it demonstrates publishing innovation?

Peer review is at the heart of the research ecosystem. We rely on it to validate findings, evaluate their importance, and to flag issues in the publication. When peer review suffers, so does the quality, pace, and public perception of research.

Publons is the first platform to bring some transparency to the peer review process. It took significant technical innovation to develop the infrastructure to handle the peer review policy and editorial management system for really any journal in the world. The scale of uptake -- 180,000 reviewers and 1,350 journals from many of the top publishers in the world -- highlights the importance of the market need we're filling.

What are you plans for the future?

Publons was recently acquired by Clarivate Analytics. Together, as a trusted neutral player in the market, we believe we can tackle the challenge of bringing trust and efficiency to peer review at scale. We plan to expand our offering to make it even easier for researchers to get recognition for their work and for publishers to have a deeper understanding of researcher workload and expertise.

Right now though we are preparing for Peer Review Week - a global event celebrating the essential role that peer review plays in maintaining scientific quality and looking forward to finding out who the winners of the Publons Peer Review Awards are too.

Publons Co-Founders - Andrew Preston (right) & Daniel Johnston
 Photo by: Image Service,Victoria University of  Wellington

Andrew Preston is the co-founder and Managing Director of Publons. He was an active researcher in physics, first as a PhD student at Victoria University of Wellington, then as postdoctoral researcher at Boston University. He founded Publons with a mission to speed up research by improving peer review. Publons recognises reviewers for their work and -- with more than 170,000 researchers and over 1,350 partner journals -- is the world’s largest peer review platform. Publons was acquired by Clarivate Analytics in June 2017. 

Website: www.publons.com

Twitter: @arhpreston @publons  
Facebook: facebook.com/publons
Publons: https://publons.com/author/1/andrew-preston
LinkedIn: http://nz.linkedin.com/in/arhpreston/

See the ALPSP Awards for Innovation in Publishing Finalists lightning sessions at our Annual Conference on 13-15 September, where the winners will be announced. 

The ALPSP Awards for Innovation in Publishing 2017 are sponsored by MPS Ltd. 

'Scientifically sound' What does that mean in peer review? Cameron Neylon asks...

Cameron Neylon, Director of Advocacy at the Public Library of Science, challenged the audience at The Future of Peer Review seminar.

He suggested that if we're going to be serious about science, we should be serious about applying the tools of science to what we (the publishers) do.

What do we mean when we say 'scientifically sound'? Science works most of the time, so we tend not to question it because of this. Should we review for soundness, as a pose to reviewing for soundness and importance.

How can we construct a review process that is scientifically sound? The first thing you would do in a scientific process is to look at the evidence, but Neylon believes the evidence is almost totally lacking for peer review. There are very few good studies. Those that exist show frightening results.

We need to ask questions about the costs and the benefits. Rubriq calculated there are 15 million hours of lost time reviewing papers that were rejected in 2012. (This video post illustrates the issues they raise). This is equivalent to around $900m if you calculate reviewers' time. How can we tell this is benefitting science? We need to decide whether we would be better spending that money and time on doing more research or improving the process.

Neylon asked what would the science look like to assess the effectiveness of peer review? There are some hard questions to ask. We'd need very large data sets and interventions including randomised control trials. But there are other methods that can be applied if data is available. Getting data about the process that you are confident with is at the heart of problem.

The obvious thing is to build a publishing system for the web. Disk space is pretty cheap and bandwidth can be managed. Measure what can be measured. Reviewers are pretty good at checking technical validity of papers. Importance is more nebulous. Taking this approach, Neylon believes that you end up with something that looks like PLOS One.

The growth curve for PLOS One is steep as it tackles these issues. In addition to this growth trajectory, 85% of papers are cited after 2 years: well above average for STM literature. There still remains a challenge of delivering that content to the right people at the right time. Technical validity depends on technical checks. PLOS One has six pages of questions to be answered before it goes to an editor. How much could we validate computationally? Where are computers better than people?

What has changed since similar talks 5 years ago? New approaches that were being discussed then are happening now (e.g. Rubriq). The outside world is much more sceptical about what happens with public funding. According to Neylon, one thing is for sure when it comes to peer review. The sciences need the science.

These notes were compiled from a talk given by Cameron Neylon at ALPSP's The Future of Peer Review seminar (London, November 2013) under CC-BY attribution. Previous presentations by Cameron can be found on his SlideShare page.

Ulrich Pöschl on advancing post-publication and public peer review

Ulrich Pöschl

Ulrich Pöschl is based at the Max Planck Institute for Chemistry and is professor at the Johannes Gutenberg University in Mainz, Germany. 

He initiated interactive open access publishing with public peer review and interactive discussion through the journal Atmospheric Chemistry and Physics and the European Geosciences Union.

In his talk at The Future of Peer Review seminar, he presented a vision of promotion of scientific and societal progress by open access and collaborative review in global information commons; access to high quality scientific publications (more and better information for scientists and society); documentation of scientific discussion (evidence of controversial opinions and open questions); and demonstration of transparency and rationalism (role model for political decision process).

Pöschl believes the most important motivation of open access is to improve scientific quality assurance. Why is it not a threat to peer review? Traditional peer review is fully compatible with open access. Information for reviewers is strongly enhanced by open access. Collaborative and post-publication peer review can be fully enabled by open access. Predatory open access publishers and hoaxes are a side-issue: a transition problem and red herring (partly caused by the vacuum created by the slow move of traditional publishers).

Pöschl went on to outline a range of problems that affect peer review. Quality assurance can be an issue with manuscripts and publications often carelessly prepared and faulty. The tip of the iceberg can be fraud. Common practice can lead to carelessness. Consequences can be waste and misallocation of resources.

Editors and referees may have limited capacity and/or competence. Traditional pre-publication review can lead to retardation and loss of information. Traditional discussion can be sparse and subject to late commentaries. For Pöschl, he doesn't have time for pure post-publication review (open peer commentary) as he has enough to do with his scientific work.

The dilemma at the heart of peer review is speed versus quality. There are conflicting needs of scientific publishing: rapid publication versus thorough review and discussion. Rapid publication is widely pursued. The answer? A two stage process. Stage 1 involves rapid publication of a discussion paper, public peer review and interactive discussion. Stage 2 comprises review completion and publication of the Final Paper.

The advantages of interactive open access publishing are that it provides an all win situation for the community of authors, referees, editors and readers. The discussion paper is an expression of free speech. Public peer review and interactive discussion provides lots of benefits, but in particular, they foster and document scientific discourse (and save reviewer capacities).

Four stages to interactive open access publishing

Pöschl outlined a multi-stage open peer review in a paper in Frontiers in Computational Neuroscience (2012). These stages are:

1. Pre-publication review and selection
2. Public peer review and interactive discussion
3. Peer review completion
4. Post-publication review and evaluation

This needs to be in combination or integration with repositories, living reviews concept, assessment house concept, ranking system/tiers and article level metrics.

At Atmospheric Chemistry and Physics, the rejection rate is as low as 10%. Submission to publication time is a minimum of 10 days and up to 1 month. The publication charge is 1000 euros and they have up to 50% additional comments pages. The achievements for combining these approaches include top speed, impact and visibility, large volume, low rejection rates and costs. The journal is fully self-financed and sustainable.

Pöschl passionately believes that these stages can be adjusted to other priorities and can therefore work for other disciplines and research communities. Future perspectives to take into account include an efficient and flexible combination of new and traditional forms of review and publication, as well as multiple stages and levels of interactive publishing and commenting.

Why is peer review is such an enduring factor in the research process? Mark Ware provides an overview

Mark Ware: why is peer review an enduring factor?

Mark Ware, author of the Publishing Research Consortium's report Peer Review: An Introduction and Guide kicked off The Future of Peer Review seminar.

Peer review is not broken in his mind. It is overwhelmingly supported by researchers, but that doesn't mean it can't be improved. Publishers need to take new developments into account.

Peer review is one part of the publishing cycle as well as the broader research cycle. It is important. Benefits include improving the quality of published articles (and this comes from the research community). It provides filters, seal of approval and a reward system that works for researchers.

However, no human activity is perfect so what are the limitations? They include validity, effectiveness, efficiency, the burden on researchers and fairness. In a world where there's a drive for transparency, we have to take these criticisms seriously. Peer review is ripe for improvement in many areas.

So who's who in peer review?

• Authors (co-authors, contributors) - the definition of authorship has become more formalised in recent years.
• Editors and editorial boards - editor role is crucial, misconception that it is the reviewers that make the decision. This ignores the fact that peer review is a process. Editors use their judgment. The best examples are a constructive discussion with the aim of making the paper the best it can be.
• Reviewers
• Publishers and editorial staff - these roles are often overlooked by those that claim reviewers do all the work. Editorial independence is a reason why we might want this. Peer review process diagram: the danger is that we think this is the system, but actuallyit is one small part of a publishing process.
• Readers (post-publication review).

Peer review flow chart: just one part of process

One size does not fit all. There are clear differences and preferences in biomedical, physics and astronomy, chemistry, mathematics, social sciences and humanities.

Retractions are booming:
In 1977 there was c.2 retractions per 100k publications.
In 2010 this had risen to 50 per 100k publications.

What's driving new approaches?

What are the problems we're trying to solve?

Which problems are we trying to mediate?


What are the opportunities to improve? Fairness and bias, delays, inefficiency, reproducibility, research data and information overload all figure.

Pre-publication innovations include:

• 'Soundness not significance'
• Cascade review
• Portable review
• Open and interactive
• Registered reports.

Post-publication innovation includes:

• Comments and ratings
• Metrics and altmetrics
• Article evaluation and tiering systems (e.g. Frontiers)
• Overlay journals.

The peer review landscape – what do researchers think? Adrian Mulligan reflects on Elsevier's own research.

Adrian Mulligan ponders: what do researchers think?

What do researchers think? Peer review is slow. Generally in STM journals, average peer review is 5 months. Social sciences can go up to 9 months. From when you submit an article through to final submission. Peer review can be prone to bias and can hold back true innovation (here he cites the Nature articles from October 2003, Coping with peer rejection: Accounts of rejected Nobel-winning discoveries highlight conservatism in science).

There is a small group of people who decide what is happening with a manuscript and they tend to be quite conservative. It's time consuming and redundant and does not improve quality. This view was reflected in The Guardian article by Dr Sylvia McLain 'Not breaking news: many scientific studies are ultimately proved wrong!' 17 September 2013. Jeffrey Brainard reported in The Chronicle in August 2008 'Incompetence tops list of complaints about peer reviewers' on how there are too few qualified overworked reviewers.  Recently, The Scientist article 'Fake paper exposes failed peer review' by Kerry Grens (October 6, 2013) highlighted how peer review may not be good at preventing fraud or plagiarism.

Elsevier undertook a research project to find out What do researchers think? From author and reviewer perspective. They surveyed individuals randomly selected from published researchers and received 3,008 responses. Most researchers are satisfied with current peer review system: 70% in 2013 (1% higher than in 2009 and 5% higher than in 2007). The satisfaction level is higher among chemists and mathematicians, and lower among computers scientists, social scientists (inc. arts, humanities, psychologists and economists). Chinese researchers are the most satisfied and there is no difference by age.

Most believe that peer review improves scientific communication. Almost three quarters agreed that the peer review process on unsuccessful submissions improved the article. The amount of researchers who had gone through multiple submissions is relatively low (29% submitted to another journal, article submitted average of 1.6 times before accepted). Few believe peer review is holding back science, but the amount is growing: in 1997 - 19% agreed, in 2009 it was 21%, and in 2013 it had risen to 27%.

Pressure on reviewers is increasing including time and a lack of incentives. Some reviewers lack the specialist knowledge required and it could be argued that too many poor quality papers are sent for review.

Mulligan observed that on a national level, the contribution in terms of submissions should be equal in terms of a country's contribution of reviews. China publishes far fewer papers compared to reviews and the reverse is true for the US. He noted that Chinese researchers are more likely to accept an invitation to review.

Over a third of those who responded believe peer review could be improved. Reviewers need more guidance, researchers are less willing to volunteer their time to conduct reviews and the fact that the system is biased/needs to be completely anonymous should be reviewed. Another challenge for the industry is whether or not peer review can genuinely detect fraud and plagiarism.

More people are shifting to open peer review, however, the preference in North America is for more traditional peer review (single blind and double blind). So what is Elsevier doing? They are reducing the number of reviews - transferring from one journal to the next. They are recognising reviewers' contributions - rewarding reviewers with certificates awards. And they are getting reviewers together to improve speed and quality.