Showing posts with label Kudos. Show all posts
Showing posts with label Kudos. Show all posts

Tuesday, 27 September 2016

Beyond Article Level Metrics

Melinda Kenneway, co-founder of Kudos, chaired the panel discussion exploring what is Beyond Article Level Metrics at the ALPSP Conference in September.

Change is coming, but it's not just metric wars, it's much bigger than that. Ben Johnson, Policy Advisor for HEFCE stressed the need for responsible metrics. They are everywehere - a layer on top of peer review. And for researchers, quality is wrapped up in publishing. The Research Excellence Framework (REF) costs £250 million. While it's only 2.5% of the overall cost of research, it is always questioned in terms of value for money.

Whatever the concerns around peer review, he observed, it is still considered the gold standard. Inappropriate indicators can create perverse incentives so these indicators need to be underpinned by an open and interoperable data infrastructure. This means that ORCID ids and DOIs become even more important moving forwards.  Johnson then announced the UK Forum for Responsible Metrics. Further details are available on the HEFCE website.

Jennifer Lin, Director of Product Management and Crossref, talked about the foundations needed to support the development of metrics. There has been an explosion of new, more diverse metrics that fit under the heading of 'altmetrics'. They hold power, constituting power, values, and livelihoods. But the infrastructure is mostly invisible, and you only feel it when it breaks. We need to understand the infrastructure behind metrics, understand metrics better, and the effect they have on research and researchers. Would a single non-proprietary body be ideal to coordinate our efforts on metrics?

Liz Allen, Director of Strategic Initiatives at F1000, talked about the opportunities to share science and its impact. There are suggestions that the concept of the journal is outdated, and open access is merely tinkering at the edges.
Publishing processes can get in the way. They don't have editors, but do pre-pub checks for ethics, quality and readability. Their open approach allows them to give full credit to reviewers. Open Science feels like the democratisation of research - it's very exciting. They are working with funders on these types of initiatives, including the Wellcome Trust on Wellcome Open Research.
Dr Claire Donovan FRSA, Reader at Brunel University London, was the final panellist. She talked about the broader impact of research but warned about 'metric fatigue' as the practice of measuring impact ran ahead of the theory. In the UK the 2014 REF had 20% impact, but may increase to 25% for 2020. This had led to concerns that impact may be more time consuming and potentially gamed in the next assessment. Donovan observed that research is a craft industry, with lots of bespoke outputs, but we're trying to assess it as if it were mass produced. (You can read her slides here.)
The Beyond Article-Level Metrics panel was held at the ALPSP Conference 2016. View all the sessions on the ALPSP YouTube channel.

Wednesday, 9 September 2015

ALPSP Awards for Innovation in Publishing - the finalists for 2015

The announcement of the winner of the ALPSP Awards for Innovation in Publishing is nearly upon us. Much anticipated and sometimes controversial, there's no denying the quality, breadth and range of the finalists.

In an intense lightning session the night before the Awards dinner, each of the shortlisted organizations presented for four minutes each to profile their submission.

The finalists were interviewed in the run up to the conference. Read each post at your leisure then debate who you think should have won on Thursday evening after the announcement!

Bookmetrix from Altmetric and Springer SBM
CHORUS - advancing public access to research
eLife Lens open-source reading tool from eLife
Impact Vizor from HighWire Press
JSTOR Daily online magazine
Kudos toolkit for researchers and their publishers
Overleaf authorship tool
RightFind XML for Mining from the Copyright Clearance Center
The Xvolution board game from NSTDA

The ALPSP Awards for Innovation in Publishing are sponsored by Publishing Technology. Not at the Awards dinner? Check back on the ALPSP website for the results!

Wednesday, 29 July 2015

Spotlight on ALPSP Awards 2015 finalist Kudos

The annual ALPSP conference will soon be upon us (so make sure you book your ticket before the late fee kicks in). One of the highlights is the announcement of the ALPSP Awards for Innovation in Publishing, sponsored by Publishing Technology, at the conference dinner.

This year's short list demonstrates yet again constant innovation in the search for new and improved ways to disseminate scholarly communication. But don't take my word for it. Here's the first in a series of spotlights on the finalists where they get to tell you, in their own words, what they are doing and why it is innovative.

In this first of the finalist posts, Melinda Kenneway, co-founder of Kudos, tells us about their platform helping researchers spread the word about their work.

ALPSP: Tell us a bit about your company

Kudos is a web-based platform to help more research get found, read and cited. Kudos came from an idea I've thought about, and talked about, for over a decade. When I was at OUP, I managed the marketing for a large program of journals, and we expended huge amounts of effort and expense looking to build readership for research articles. Campaigns that were most effective were those that the editor or author initiated. Not a surprise, as an email from a colleague or other prestigious member of a community, was always going to be more likely to be read than one from the publisher’s marketing department! The problem was, how to do this at scale … we couldn't manage this article-level promotion for everything we published – it was literally thousands and thousands of articles every year.

Since that original idea, much in the world has changed. The rise of article level metrics, broader assessment of impact and the increasing uptake by the research community of social media is creating a new opportunity for authors to take more direct control of the post-publication performance of their work. Coming from an industry background, the founders of Kudos saw an opportunity to provide tools to individual researchers to assist with this; but rather than do this in isolation to the rest of the scholarly communications industry, to work to aggregate the efforts of authors, publishers, universities, societies and funders – using existing infrastructure, tools and services to help drive impact.

That is the vision for Kudos. We’re currently still in the early days of our development, having launched in May last year initially with a toolset for authors to explain, enrich and share links to their publications. Kudos is free to authors, funded primarily by revenue from a publisher service that provides visibility on their authors’ activities within Kudos, and opportunities to support and amplify their outreach efforts.

What is the project that you submitted for the Awards?

We've submitted the Kudos researcher and publisher service, which we launched in May 2014. In just over a year we've grown to over 55,000 registered users with the support of some 40 publishers, including our founding partners: Taylor & Francis and the Royal Society of Chemistry.

Our users are drawn from all over the world, representing a range of career stages. Over a third are well established in their careers, so clearly, interest in publication performance by the research community is universal. Authors registered in Kudos are linked to over 2 million articles. Authors have created over three million views of their article profile pages on the Kudos site, and driven over 30,000 referrals to publisher sites since we launched.

Tell us more about how it works and the team behind it.

'Kudos was founded in 2013 by Melinda Kenneway, Charlie Rapple and David Sommer. We are three industry consultants with over half a century’s experience between us. This holistic overview has been instrumental in our approach to creating a solution that works with, rather than against, the current system. Not that we’re saying things can’t be improved – far from it; but we’re not trying to create a revolution so much as facilitate an accelerated evolution.

Kudos works by providing three simple tools to authors – they can create a profile page for each of their publications (an article, book or book chapter), drawing metadata from CrossRef (title, authors, publisher etc.) as a foundation for that page. They then add a short title, and brief explanations of what the work is about and why it’s important.. All this valuable, browseable new data is then made available to search engines, publishers, and other sites and services to help broaden reach. Authors can also add individual perspectives, so co-authors can tell the story of that publication from their viewpoint – this new feature, recently launched, has been particularly popular. Authors can also enrich their publications, linking up related resources such as images, datasets, videos and news coverage. We then provide a simple way for authors to share a link to their publication page via email, their blog, or social media sites, this is then trackable so they can assess subsequent impact on their personal dashboard against a range of metrics.

Publishers and universities can gain visibility on this activity through a Kudos dashboard that shows them which of their authors are registering, using the tools, and who is getting traction – for example, those articles whose altmeric scores are changing most rapidly indicating ‘of the moment’ media attention and those authors who are being most successful at sharing, with click-through access to information on who else is influential in driving impact within their networks (retweets etc.). The publisher dashboard provides opportunities for one-to-one engagement with authors to help increase the impact of their efforts, and identify influential advocates within communities. Publishers also benefit from access to the lay metadata and links that the authors add, which can be provided back in the form of a widget, to redisplay on the publication page on the publisher site. All publication profile pages on Kudos link prominently to the publisher’s authoritative version of the article or book chapter, on their site.

Why do you think it demonstrates publishing innovation?

Kudos is unique in being the first service to give authors tools to increase the impact of their research across all publishers, and the first to combine metrics from many sources in one place for authors – then map these against outreach activities. We’re also for the first time giving publishers visibility on the outreach activities of their authors and its effectiveness.

Beyond this, is the potential for even more significant innovation - through creating a service that is open and integrated into the existing and evolving scholarly communications infrastructure. We believe the consumer model for creating controlled communities, with the aim of keeping people within them to create monetizing opportunities, doesn't work for research. The community is too large and diverse, and the role of publishers, universities, funders and societies is too important to simply bypass. The real innovation for us will come in the future as we work to forge partnerships with the multiple existing and emerging organisations, tools, systems, resources and services that individually, or together in any combination, can help drive research performance.

What are you plans for the future?

Our roadmap is substantial. Our challenge is fulfilling this as fast as we would like! Because the founders are all drawn from within the industry, this has helped Kudos have a high profile at this early stage in our life. We've been keen to talk about our ideas at conferences and in workshops, and excited to have had so much support from publishers, so early on. However, we remain a very young company, with the challenges of working in a lean way to an ambitious vision. We've successfully generated some initial revenues, won grants, and attracted early funding – this has helped accelerate some of our roadmap pans – we’re particularly pleased to have been able to develop additional features for authors, or example - to start helping them manage their personal digital visibility more broadly via profile sites; and we will shortly be releasing a significantly enhanced publisher dashboard. We’re also currently in pilot phase with a service for universities, which includes almost a dozen major institutions such as the London School of Economics and Political Science, Stockholm University and Carnegie Mellon University.

But there’s a lot more to be done – all our existing products have many ongoing enhancements planned, and we’re also currently looking at developing a society service to help societies engage more deeply with their members and showcase high performance. 

Our longer term vision to embrace a future where researchers will have multiple ways of talking about their work, legitimately sharing information on publications, finding collaborators, reaching out to peers and connecting new audiences with their work. As the landscape grows in complexity, so will the need for a central tool-set, deeply integrated with these emerging services. Kudos will provide this – embedding ourselves where researchers want to access our tools, then providing them – and the organisations that support them – with insight on which of these drives performance, based on the measures that matter to them. By doing this, we aim to make research more discoverable, more useful, and maximise the effectiveness of outreach activities.'  

Melinda Kenneway is Executive Director at Kudos. After completing a degree in Experimental Psychology at the University of Oxford, Melinda began her career in academic publishing working at Oxford University Press for 13 years, where she gained the position of Global Marketing Director for the Journals Division. In 2004 she founded TBI Communications, a specialist marketing agency serving academic publishers, societies and libraries. In 2013 she co-founded Kudos – a web-based platform that supports the research community in increasing the impact of research output. 

Thursday, 11 September 2014

Metrics and more

Melinda Kenneway on metrics
Publication metrics are part of a much bigger picture. Where resources are restricted and there is a lot of competition, metrics become more essential to help prioritise and direct funding. The 'Metrics and more' 2014 conference session was chaired by Melinda Kenneway, Director at TBI Communications and Co-founder of Kudos. The panel comprised Mike Taylor, Research Specialist at Elsevier Labs, Euan Adie, Founder of Altmetric and Graham Woodward, Associate Marketing Director at Wiley. Kenneway opened by observing that as an industry we need to consider new types of metrics.

Publication performance metrics include:
  • Anti-impact factor: DORA
  • Rise of article level metrics
  • introduction of almetrics
  • New units of publishing: data/images/blogs
  • Pre-publication scoring (Peerage of Science etc)
  • Post-publication scoring (assessment, ranking etc)
  • Tools for institutional assessment

Researcher performance metrics include:

  • Publication output
  • Publication impact
  • Reputation/influence scoring systems
  • Funding
  • Other income (e.g. patents)
  • Affiliations (institutional reputation)
  • Esteem factors
  • Membership of societies/editorial boards etc
  • Conference activity
  • Awards and prizes

Institutional performance metrics include:

  • University ranking systems
  • Publication impact metrics
  • STAR/Snowball metrics
  • Research leaders and career progression
  • Patents, technologies, products, devices
  • Uptake of research

Graham Woodward, Associate Marketing Director at Wiley, provided an overview of a trial of altmetrics on a selection of six titles. On one article, after a few days of having altmetrics on the site, they saw the following results: c. 10,000 click throughs; average time on page over three minutes; over 3,500 tweets; an estimated 5,000 news stories; 200 blog posts; and 32 F1000 recommendations.

Graham Woodward
They asked for user feedback on the trial and the 50 responses provided a small but select snapshot that enabled them to assess the effectiveness of the trial.

Were the article metrics supplied on the paper useful? 91% said yes. What were the top three most useful metrics? Traditional news outlets, number of readers and blog posts. 77% of respondents felt the experience enhanced the journal.

Half of respondents said they were more likely to submit a paper to the journal. 87% used the metrics to gauge the overall popularity of the article, 77% to discover and network with researchers who are interested in the same area of their work and 66% to understand significance of paper in scientific discipline.

What happened next? The completion of six journal trial was followed by an extension to all OA journals. They have now rolled out metrics across the entire journal portfolio

Euan Adie from Altmetric reflected on the pressures and motivations on researchers. While there is a lot of pressure within labs for young researchers, funders and institutions are increasingly looking for or considering others types of impact, research output and contribution. There is an evaluation gap between funder requirements and measuring impact. That's where altmetrics come in. They take a broader view of impact to help give credit where it is due. HEFCE are doing a review of metrics within institutions at the moment.
Euan Adie

Seven things they've learnt in the past year or so.

  1. Altmetrics means so many things to so many people. But the name doesn't necessarily work. It is complimentary rather than alternative and it is about the data, not just the measure.
  2. It's worked really well for finding where a paper is being talked about where they wouldn't have known before, but also the demographics behind it.
  3. Altmetrics is only a weak indicator of citations, but the whole point is to look beyond. Different types of sources correlate to different extents.
  4. Don't take all altmetrics indicators as one lump, there are many different flavours of research impact.
  5. When you have an indicator and you tie it to incentives, it immediately corrupts the indicator. While he doesn't believe there is massive gaming of altmetrics there is an element of this with some people. It's human nature.
  6. The top 5% of altmetric scores are not what you expect.The most popular paper is a psychological analysis of the characters in Winnie the Pooh.
  7. Peer review is a scary place. Scientists and researchers can be pretty nasty! Comments can be used in a different (more negative) way than expected. But that is not necessarily a bad thing.
Mike Taylor believes we are approaching a revolution rather than an evolution. What we have at the moment is a collision of varying different worlds because the value of interest in metrics is increasing. What makes for great metrics, and how do we talk about them? Do we want the one-size-fits-all approach? We have data and metrics and in between those two things there is theory, formulae, statistics and analysis. Within the gap between the two things there are a lot of political issues. 

Taylor reflected on the economies of attention (or not) and how you assess if people are engaged. With an audience, when hands go up, you know they are paying attention, but no hands doesn't mean they aren't. Metrics so far are specialist, complex, based on 50 years of research, are mostly bibliometrics/citation based and much is proprietary. The implications for changing nature of metrics: are: as metrics are taken more seriously by institutions, the value of them will increase. As the value increases, we need to be more aware of them. As a scholarly community we need to increase awareness about them. Awareness implies critical engagement, mathematics, language, relevance, openness, agreement, golds standards, and community leadership.

Mike Taylor
We are experiencing a collision of worlds. Terms like 'H-Index' are hard to understand, but are well defined. Terms like 'social impact' sound as if they're well defined, but aren't. There are particular problems about the 'community' being rather diverse. There are multiple stakeholders (funders, academics, publishers, start-ups, governments, quangos), international perspectives and varying cultures (from fifty years of research to a start-up). 

Taylor suggested an example metric - 'internationalism'. Measures could include: how well an academic's work is used internationally; how well that academic works; through readership data; citation analysis (cited, citing); co-authorship; funding data (e.g. FundRef); conference invitations e.g. ORCID; guest professorships; text-analysis of content.

Taylor doesn't think metrics is a place where publishers will have the same kind of impact that they might of 30 years ago. He said to expect to see more mixed metrics with qualitative and quantitative work. Taylor concluded that metrics are being taken more seriously (being used in funding decisions). Many stakeholders and communities are converging. 

Big data + cloud computing + APIs + openness = explosive growth in metrics. 

It is a burgeoning research field in its early days. Publishers need to be part of the conversation. We need to enable community leadership and facilitate decision making.

Thursday, 11 July 2013

Charlie Rapple on leveraging authors' expertise and networks to increase discoverability and usage

Charlie Rapple, co-founder of Kudos
Charlie Rapple, Associate Director at TBI Communications, provided a stark reminder of the environment researchers and publishers operate in at our seminar 'It's all about discoverability, stupid!' held on 10 July.

The problem? There are more and more articles which people have less and less time to read. This results in too many articles that are uncited.

A paper by Michael Mabe and Mayur Amin from 2001 projected 3.26% per annum growth in article numbers (Scientometrics, 51:1 (2001) 147-162) which equates to a lot of papers. In the report The Role of the Critical Review Article in Alleviating Information Overload (Annual Reviews 2011) it was found that 81% of early career researchers felt they should read more of the literature than they do, and 25% suggested they would need to read for more than 24 working hours a week to keep up.

If there is less time for reading, then too many articles will go without being cited. A further article - Five-year impact factor data in the Journal Citation Reports - from 2009 by Péter Jacsó in Online Information Review, explores new indicators based on the level of uncitedness of articles in journals.

The fact that reader time is at a premium is an issue for the reader, author, publisher and librarian. There is a filtering challenge. The standard way of communicating what is in a formal scientific paper is through an abstract, but when you read it, all but an expert might struggle to get to grips with the content. As a result they may miss reading the paper.

One such example is a paper co-authored by R Giles Harrison at the University of Reading entitled 'Electrical signature in polar night cloud base variations' (Environmental Research Letters, March 2013). As a student, if you read the abstract, the first paragraph of which is extracted below, you might not be inspired to go to the full text download.

"Layer clouds are globally extensive. Their lower edges are charged negatively by the fair weather atmospheric electricity current flowing vertically through them. Using polar winter surface meteorological data from Sodankylä (Finland) and Halley (Antarctica), we find that when meteorological diurnal variations are weak, an appreciable diurnal cycle, on average, persists in the cloud base heights, detected using a laser ceilometer."

However, if you view the accompanying video posted on YouTube, you immediately see the headline story, put in simple terms, and communicated by a researcher who is passionate about and committed to his work.



This approach was subsequently picked up in the wider press enabling the paper and author to gain much wider coverage raising the impact of the work.

Harnessing the authors' expertise and explaining research for different audiences is an essential part of this approach. Putting the research in context and explaining why is it important while mapping to human interest or hot topics can help raise the profile of the work and widen the readership. If you can boil down what's unique about it while providing some background on who or what was an inspiration, you can convey the passion behind your work and go beyond formal formats. This is likely to lead to research being cited. Currently, it is time-consuming and manual to draw out the "story" and only a minority of research gets this treatment

Capturing lost value
There are a lot of assets already built on authors' expertise including: lay summaries, impact statements, benefit to society statements, novelty statements etc. All this type of information could be enormously helpful to PR teams and researchers, but is currently held in silos. Part of the problem is the limited infrastructure for capturing and putting these assets to work. There is a real opportunity for the author as a more authoritative hub using existing social media networks such as Facebook and Twitter as well as research-oriented sites such as MyExperiment.org, ResearchObject.org, Mendeley and CiteULike.

Rapple cited the work of Melissa Terras from UCL who blogged and tweeted about uploading papers to the repository, initially to relieve the boredom. She told the story behind the research with every blog post, and was intrigued to note spikes in downloads of her papers that corresponded with her use of social media. This led her to observe:

"If you tell people about your research, they look at it." 

This demonstrates the potential to close the loop, break down the silos and reclaim the value of contextual information that is currently lost or fragmented.

Bringing it all together
These three problems of too much information, not enough time; valuable assets left in silos; and valuable networks being under-utilised require a solution that involves better use of metadata and multimedia. Rapple has been working on a new project, currently in beta, called Kudos, that tries to address this.

Whatever the solution, it will need to take into account:
  • Article level metrics (altmetrics and usage marketing)
  • Author services (open access, academic spring, community, advocacy)
  • Discovery (SEO, social media, international reach, metadata, marketing to individuals)
  • Closing the loop (data driven services, integration not duplication, social machines)
  • Intelligent reading (filtering, multimedia, public accessibility)
  • Research evaluation (funding, REF, STAR, ORCID, H-Index etc)