Tuesday 27 September 2016

Beyond Article Level Metrics

Melinda Kenneway, co-founder of Kudos, chaired the panel discussion exploring what is Beyond Article Level Metrics at the ALPSP Conference in September.

Change is coming, but it's not just metric wars, it's much bigger than that. Ben Johnson, Policy Advisor for HEFCE stressed the need for responsible metrics. They are everywehere - a layer on top of peer review. And for researchers, quality is wrapped up in publishing. The Research Excellence Framework (REF) costs £250 million. While it's only 2.5% of the overall cost of research, it is always questioned in terms of value for money.

Whatever the concerns around peer review, he observed, it is still considered the gold standard. Inappropriate indicators can create perverse incentives so these indicators need to be underpinned by an open and interoperable data infrastructure. This means that ORCID ids and DOIs become even more important moving forwards.  Johnson then announced the UK Forum for Responsible Metrics. Further details are available on the HEFCE website.

Jennifer Lin, Director of Product Management and Crossref, talked about the foundations needed to support the development of metrics. There has been an explosion of new, more diverse metrics that fit under the heading of 'altmetrics'. They hold power, constituting power, values, and livelihoods. But the infrastructure is mostly invisible, and you only feel it when it breaks. We need to understand the infrastructure behind metrics, understand metrics better, and the effect they have on research and researchers. Would a single non-proprietary body be ideal to coordinate our efforts on metrics?

Liz Allen, Director of Strategic Initiatives at F1000, talked about the opportunities to share science and its impact. There are suggestions that the concept of the journal is outdated, and open access is merely tinkering at the edges.
Publishing processes can get in the way. They don't have editors, but do pre-pub checks for ethics, quality and readability. Their open approach allows them to give full credit to reviewers. Open Science feels like the democratisation of research - it's very exciting. They are working with funders on these types of initiatives, including the Wellcome Trust on Wellcome Open Research.
Dr Claire Donovan FRSA, Reader at Brunel University London, was the final panellist. She talked about the broader impact of research but warned about 'metric fatigue' as the practice of measuring impact ran ahead of the theory. In the UK the 2014 REF had 20% impact, but may increase to 25% for 2020. This had led to concerns that impact may be more time consuming and potentially gamed in the next assessment. Donovan observed that research is a craft industry, with lots of bespoke outputs, but we're trying to assess it as if it were mass produced. (You can read her slides here.)
The Beyond Article-Level Metrics panel was held at the ALPSP Conference 2016. View all the sessions on the ALPSP YouTube channel.

No comments:

Post a Comment