In this guest post, Michael Casp and Anna Jester look back at the ALPSP 2022 Annual Conference and Awards.
Peer review in the Digital Age relies
heavily on email communications and web-based peer review platforms to contact reviewers and organize
recommendations. In many ways, peer review today is just a recreation of the pre-internet
mail, fax, and file cabinet template, but pasted online. With the current advancements in preprints,
social media, and communication platforms, it is possible – even likely – that
the model of peer reviewing, and the technology that supports it, have more
evolutions to come.
As communication begins to move beyond
traditional text formats, so does content. Getting “beyond the PDF” has been a
staple at scholarly publishing conferences for years, and as it becomes more
commonplace, we are navigating its demands on the peer review process.
But it’s not just about technology. We must
also focus on developing and empowering the next generation of reviewers in
order to maintain a robust and sustainable reviewer pool. This happens through
teaching and developing early academics, which can offer a technology-versed,
diverse pool of reviewers that can continue to utilize the technologies we
develop.
At the ALPSP 2022 Conference we were treated to many innovative ideas that will
have a direct impact on peer review, potentially changing it and hopefully
improving it, for the researchers and publishers and ultimately providing
better value for society.
Beyond Email
Peer review’s reliance on email is a given.
That is, unless you’re in China, where the app WeChat has in many ways
supplanted email as the default communication system. Western publishers that
seek to engage with Chinese researchers might struggle if they only use email.
However, Charlesworth presented the ALPSP audience with a possible solution
with their new product, Gateway. Gateway
uses an API to allow journal-related communications
to be sent to authors, reviewers, and editors via WeChat. This solution allows journals
to meet Chinese researchers where they are, rather than trying to pull them
into an (let’s face it, antiquated) email system.
Given this shift, eLife presented the ALPSP
audience with Sciety, a new
preprint evaluation platform that allows academics to review, curate, and share
preprints and reviews with their peers. Preprint servers have also started to
become social hubs for researchers to connect in more of a real time
environment than traditional publishing tools. This system holds the promise of
opening up peer review and publishing to a wider user base, allowing more
people to curate and review research than ever before. The challenge presented
by the scale of preprints is immense, and Sciety has the potential to
reorganize how we deal with all this research in a social-focused way.
One more data point to mention: with the pervasive
use of social media throughout the world, it is no surprise that academics
would have their own version. Despite controversies,
ResearchGate has maintained its
position as the largest academic social network, connecting about 17 million
users. With this many scholars connected, it’s possible we could see something
like peer review networks emerge, though that doesn’t seem to be ResearchGate’s
focus at the moment.
Beyond the PDF
A decade or so ago, getting “beyond the
PDF” was still a new idea being speculated about at conferences. It is now a
reality, with authors providing data sets, code, and detailed methods notebooks
alongside their traditional manuscripts. As a partner to those authors, we’ve
come up with ways to publish this content, but it can present special problems
during peer review.
For starters, journals that employ an
anonymized review model can find it quite difficult to extract identifiable
information from complex content like code or reproducible methods. Sometimes
an author’s institution might be inextricably linked to this content, making
anonymization impossible – or at least impractical.
Other forms of content, like ultra-high-resolution
images, can present logistical problems. Industry standard peer review
management systems have hard limits on the size and format of files they can
manage. For example, fields like pathology can rely on extremely detailed
images of microscope slides, and these multi-gigabyte files are hard to move from author, to
editor, to reviewer. Paleontology research can also require larger-than-usual
images, as sharing highly detailed photos of artifacts is crucial to the field.
Dealing with these kinds of challenges at the peer review stage can require a
lot of creativity and patience for all involved without a more flexible
solution.
Massive datasets can also present review
challenges. Beyond the logistics of moving large
files, there are also often more basic concerns, like is this data organized
and labeled in a useful (and reusable) way? Is it actually possible to do a
real review of a large dataset in the time that reviewers have to give to a
paper? FAIR data principles
are targeted at answering some of these questions, and services like Dryad and Figshare
seek to help by curating and quality controlling
datasets, ensuring they meet basic standards for organization, labeling, and
metadata. But these services come with an additional cost that not everyone can
bear. And a data review still depends on a reviewer willing to go the extra
mile to actually review it.
Moving peer review beyond the PDF is still
a work in progress, but many of these are solvable problems as our technology
and internet infrastructure improve. Our J&J Editorial
staff regularly handle content like videos, podcasts, and datasets. At EJournalPress’, our platform is integrated
with third parties including Dyrad, Code Ocean, and Vimeo. These integrations
are an added convenience, as most journals and societies have to have direct
agreements with third parties for the integrations to be fully utilized. But we
often have to work around the peer review system, rather than with it, relying
on basic cloud file servers (e.g., Dropbox or OneDrive) instead of more
purpose-built technology.
Open/Pre-submission Review
Another decade-old conference trope was the
constant talk about new open peer review models. You might recall that people
were split on the wisdom of this approach, but the rise of preprints has done a
lot to push open peer review and pre-submission review into the limelight.
Organizations like Review Commons are working with EJournalPress to make pre-submission and
peer review a viable choice for authors by building review transfer
relationships with more traditional journals. The Review Commons model is to
take preprint submissions and have them peer reviewed. These reviews can then
be shared with participating journals if authors choose to submit. Journal editors
can use the existing Review Commons reviews to evaluate whether or not to
publish the work. In data presented at ALPSP 2022, manuscripts that came into
journals with reviews rarely needed to be sent out for further review. This has
many benefits, saving the editor time in soliciting reviews, and giving a
journal’s (probably over-taxed) reviewer pool a little break.
Review Commons is currently free for
authors, being supported by a philanthropic grant. It will be fascinating to
see if they are able to pivot towards a sustainable financial model in the
future.
We won’t exhaust you with the long list of
other open peer review initiatives, but suffice it to say, a lot of smart people
are working hard on making this a standard part of research communication.
Developing the Next Generation of
Reviewers
None of what we’ve written so far will
matter one iota if there aren’t enough people in place to do the actual content
reviews. One of the interesting revelations we had while managing journal peer
review was the incredible range that exists in review quality. From the
in-depth, multi-page discussions of every point and nuance of an author’s
manuscript, to the dreaded “Looks good!” review, anyone in peer review can tell
you that we can (and must!) do a better job training our reviewers. We can
offer guideline documents and example reviews, but some people need a more
engaging approach to understand and deliver what editors expect.
It would be lovely if reviewing was a
required part of the science curriculum. It currently seems to happen in a
piece-meal, ad hoc fashion, many times driven by people who are willing to just
figure it out themselves. A more standardized approach is called for,
especially as reviewable content becomes more complex and varied.
One of the best examples we’ve seen of
reviewer training was actually a writer’s workshop for researchers wishing to
submit to a medical journal. The EIC of the journal
led this workshop, asking authors to submit their manuscripts ahead of time to
serve as examples for the workshop. During the workshop, the EIC talked through
several of these manuscripts, giving the authors invaluable feedback and what
amounted to a free round of review prior to the official journal submission.
Though this workshop was ostensibly for
authors, it was equally valuable for reviewers. Participants got to watch the
EIC go through a paper in real time, ask questions, pose solutions, and talk
through the subject matter with someone who had written hundreds and reviewed
thousands of manuscripts. This program has always stuck out as a great way to
train authors and reviewers, while also building the relationship between the
journal and its community. Win-win-win!
Formal peer review training benefits all
parties, such as the Genetics Society of America’s program which also includes direct feedback from
journal editors. If you’re thinking of implementing something like this, your
organization may wish to conduct a pilot prior to a full rollout. Another great
model for peer review training is to pair mentors and mentees to simultaneously
provide training and increase the number of high quality trained peer reviewers in the field broadly. If
your team is willing to study the results of your reviewer training efforts, be
sure to submit them to the next International Congress on Peer Review and Scientific Publication so that we can all
benefit from your findings.
Demographics and Diversity
Many of the journals and publishers we work
with are prioritizing diversity within their community by making efforts to
extend their reach to people who might have been historically left out of the
conversation. These organizations are also looking inward to seeing what their
current level of diversity looks like in order to improve it.
Many organizations have begun collecting
demographic information regarding their authors, reviewers, and editors. We
recommend a thoughtful approach when embarking on this project, as it can be
fraught with pitfalls and unexpected consequences if you don’t get it right.
Before your organization embarks on this endeavor, please consider best practices regarding data collection and clearly define the initiative’s
goals. Wondering where to start? Do a landscape scan of what other organizations aim to do with this data and please use standardized questions for self-reported diversity data collection.
Fortunately, many people are working on
demographics data initiatives, and there is lots of support and ideas available
from our community.
Summary
To put it mildly, there is a lot going on
right now. The technology we use has the potential to upend the traditional
research communication process, and in some cases (like preprints) it already
has. With a host of new data, content, and equity concerns, people involved in
the peer review process have more to deal with than ever before. And it’s unlikely
that we’re doing enough to equip them with the knowledge and training they need
to succeed. But we can do better, and I’m heartened to see the many people in
and around our industry who are trying to improve the situation. From our end, eJournalPress is supporting societies
and journals as they work to collect and evaluate demographic information and
metadata, and J&J Editorial staff
are always investigating ways to support journal innovations through a
combination of technology and experienced staff.
I often think about peer review in the
context of that old Churchill quote about democracy: “It has been said that
democracy is the worst form of government, except all of those other forms that
have been tried from time to time.” Peer review might not be the best method of
scientific evaluation, but it’s the best we’ve got, and who knows, maybe we’ll
make something even better. But until then, we’ve got work to do.
Anna Jester, VP of Marketing and Sales, eJournalPress, Wiley Partner Solutions
Michael Casp, Director of Business Development, J&J Editorial, Wiley Partner Solutions
Wiley Partner Solutions was gold sponsor of the ALPSP Conference and Awards held in Manchester UK in September 2022. The 2023 ALPSP Conference will be held in Manchester from 13-15 September 2023.
Wiley is one of the world’s largest publishers and a global leader
in scientific research and career-connected education. Founded in 1807, Wiley
enables discovery, powers education, and shapes workforces. Through its
industry-leading content, digital platforms, and knowledge networks, the
company delivers on its timeless mission to unlock human potential. Visit us at
Wiley.com. Follow us
on Facebook, Twitter,
LinkedIn
and Instagram.
Mastodon is a software development non-profit from Germany founded in 2016 by Eugen Rochko and offers microblogging in an open source realm. There are currently four members in the core team - Eugen, Claire, Felix Hlatky and Dopartwo and 1.8m monthly active users (+385% as of 14 November 2022). Funded by sponsors, there are three packages to support Mastodon, with the lowest already sold out. You may notice the logo in blue, but this is being phased out as part of their new rebrand to purple. I haven't been able to yet find out why they named their business after an extinct mammal, especially one whose name (named by French naturalist Georges Cuvier in the early 19th century) translates to 'nipple tooth' because of their distinctly shaped small teeth! 
