Tuesday 11 August 2020

Spotlight on Rigor and Transparency Index powered by SciScore - shortlisted for the 2020 ALPSP Awards for Innovation in Publishing

In this series, we hear from the finalists for the 2020 ALPSP Awards for Innovation in Publishing.  The winners will be announced at the ALPSP Virtual Conference on 16 September.  

In this post, we hear from Anita Bandrowski and Martijn Roelandse about the RTI, Rigor and Transparency Index, powered by SciScore.

logo Rigor and Transparency Index

Tell us a bit about your organization.
We are a group of inventors that come from a long tradition of breaking technology and social barriers. We have a technology toolbox, but always ask “is the answer technology or culture?”

What is the project/product that you submitted for the Awards?

In a way, we have something similar to the impact factor, however it scores different things. The RTI, Rigor and Transparency Index, is the average rigor criteria completeness score for each journal.

graphic SciScore illustration

What is included? The rigor and transparency criteria (sex/gender as a biological variable, randomization of subjects into groups, investigator bias reduction through blinding, group size determinant via power calculation, identification of statistical tools, as well as the identification and authentication of key biological resources such as antibodies, cell lines, and organisms). Authors, funded by several funding agencies such as NIH or submitting to various journals such as Nature, are expected to address these criteria in their work, but some may not. The RTI is the average number of these criteria found in journal articles in a given year. 
We think that if journals know where there are particular problems or where they stand compared to their peers, it may inspire them to work with authors to improve the quality of manuscripts.

Tell us a little about how it works and the team behind it

The RTI is based on our tool, SciScore, which has now been integrated into eJournal Press and will be available in Editorial Manager in the fall. This tool acts as an automated reviewer, which can check whether rigor criteria are addressed. It looks for sentences that discuss the particular criteria, for example “experiments were performed on both male and female mice, with subjects balanced for sex in each group” fulfills the “sex/gender as a biological variable” criteria. The authors or reviewers can then determine if the criteria are met in the context of the study, something that automated tools really can’t do at this point. While SciScore acts as a reviewer, the RTI index is the average SciScore for papers published in a given journal and given year.

The team behind this endeavor includes Anita Bandrowski who heads the Antibody Registry and the RRID initiative; Maryann Martone, one of the architects of FAIR; Martijn Roelandse, a former head of content innovation at a major publisher; and Burak Ozyurt, a computer scientist that has just taken the Google BERT AI, and made it run ...on a laptop ….in his spare time. We bring varied skills to find interesting solutions to very difficult problems.

In what ways do you think it demonstrates innovation?

Innovation is a tough thing to assess. Certainly there are plenty of ways that someone can score journals. However, as far as I am aware, all ways of scoring journals are based on the data around the article itself, i.e, how many times it is cited or tweeted about. The RTI delves deep into the science described in the paper and attempts to assess how good that is. It seems to me that this is something quite different, therefore potentially innovative.

What are your plans for the future?

“Pinky we are going to take over the world!!!” - The Brain (...or maybe not)

We are passionate about making science a little better, and a little more efficient. We want every study to be done in a rigorous way, we want every reagent to be explicitly specified so people reading scientific papers can quickly replicate key findings. If successful, this should accelerate the pace of scientific discovery, not just the rate of publishing.

photo Anita Bandrowski
Anita Bandrowski works in the department of neuroscience at UCSD taking part in the daily operations of projects such as the Brain Cell Census Network and the SPARC project (tracing the neural to organ connections). In addition, she runs the RRID initiative (rrids.org), heads SciScore development efforts, and appreciates the finer intricacies of both wine and cats. 

photo Martijn Roelandse
Martijn Roelandse is founder/consultant at martijnroelandse.dev and has worked with the SciScore team for over 10 months now. He has a PhD in neuroscience, worked for 10+ years in scholarly publishing with a major publisher and 2015 ALPSP Award finalist with Bookmetrix. With friends he owns 4 barrels of whisky and he is really fond of his dog Bo.

COVID-19 Project Twitter bot 

You can hear from all of this year's finalists at the ALPSP Awards for Innovation Lightning Session on Tuesday 8 September.  Visit the ALPSP website to register and for full details of the ALPSP Virtual Conference and Awards 2020.
The 2020 ALPSP Awards for Innovation in Publishing are sponsored by PLS.

No comments:

Post a Comment