Metrics and more

Melinda Kenneway on metrics

Publication metrics are part of a much bigger picture. Where resources are restricted and there is a lot of competition, metrics become more essential to help prioritise and direct funding. The 'Metrics and more' 2014 conference session was chaired by Melinda Kenneway, Director at TBI Communications and Co-founder of Kudos. The panel comprised Mike Taylor, Research Specialist at Elsevier Labs, Euan Adie, Founder of Altmetric and Graham Woodward, Associate Marketing Director at Wiley. Kenneway opened by observing that as an industry we need to consider new types of metrics.

Publication performance metrics include:

• Anti-impact factor: DORA
• Rise of article level metrics
• introduction of almetrics
• New units of publishing: data/images/blogs
• Pre-publication scoring (Peerage of Science etc)
• Post-publication scoring (assessment, ranking etc)
• Tools for institutional assessment

Researcher performance metrics include:

• Publication output
• Publication impact
• Reputation/influence scoring systems
• Funding
• Other income (e.g. patents)
• Affiliations (institutional reputation)
• Esteem factors
• Membership of societies/editorial boards etc
• Conference activity
• Awards and prizes

Institutional performance metrics include:

• University ranking systems
• Publication impact metrics
• STAR/Snowball metrics
• Research leaders and career progression
• Patents, technologies, products, devices
• Uptake of research

Graham Woodward, Associate Marketing Director at Wiley, provided an overview of a trial of altmetrics on a selection of six titles. On one article, after a few days of having altmetrics on the site, they saw the following results: c. 10,000 click throughs; average time on page over three minutes; over 3,500 tweets; an estimated 5,000 news stories; 200 blog posts; and 32 F1000 recommendations.

Graham Woodward

They asked for user feedback on the trial and the 50 responses provided a small but select snapshot that enabled them to assess the effectiveness of the trial.

Were the article metrics supplied on the paper useful? 91% said yes. What were the top three most useful metrics? Traditional news outlets, number of readers and blog posts. 77% of respondents felt the experience enhanced the journal.

Half of respondents said they were more likely to submit a paper to the journal. 87% used the metrics to gauge the overall popularity of the article, 77% to discover and network with researchers who are interested in the same area of their work and 66% to understand significance of paper in scientific discipline.

What happened next? The completion of six journal trial was followed by an extension to all OA journals. They have now rolled out metrics across the entire journal portfolio

Euan Adie from Altmetric reflected on the pressures and motivations on researchers. While there is a lot of pressure within labs for young researchers, funders and institutions are increasingly looking for or considering others types of impact, research output and contribution. There is an evaluation gap between funder requirements and measuring impact. That's where altmetrics come in. They take a broader view of impact to help give credit where it is due. HEFCE are doing a review of metrics within institutions at the moment.

Euan Adie

Seven things they've learnt in the past year or so.

1. Altmetrics means so many things to so many people. But the name doesn't necessarily work. It is complimentary rather than alternative and it is about the data, not just the measure.
2. It's worked really well for finding where a paper is being talked about where they wouldn't have known before, but also the demographics behind it.
3. Altmetrics is only a weak indicator of citations, but the whole point is to look beyond. Different types of sources correlate to different extents.
4. Don't take all altmetrics indicators as one lump, there are many different flavours of research impact.
5. When you have an indicator and you tie it to incentives, it immediately corrupts the indicator. While he doesn't believe there is massive gaming of altmetrics there is an element of this with some people. It's human nature.
6. The top 5% of altmetric scores are not what you expect.The most popular paper is a psychological analysis of the characters in Winnie the Pooh.
7. Peer review is a scary place. Scientists and researchers can be pretty nasty! Comments can be used in a different (more negative) way than expected. But that is not necessarily a bad thing.

Mike Taylor believes we are approaching a revolution rather than an evolution. What we have at the moment is a collision of varying different worlds because the value of interest in metrics is increasing. What makes for great metrics, and how do we talk about them? Do we want the one-size-fits-all approach? We have data and metrics and in between those two things there is theory, formulae, statistics and analysis. Within the gap between the two things there are a lot of political issues. 

Taylor reflected on the economies of attention (or not) and how you assess if people are engaged. With an audience, when hands go up, you know they are paying attention, but no hands doesn't mean they aren't. Metrics so far are specialist, complex, based on 50 years of research, are mostly bibliometrics/citation based and much is proprietary. The implications for changing nature of metrics: are: as metrics are taken more seriously by institutions, the value of them will increase. As the value increases, we need to be more aware of them. As a scholarly community we need to increase awareness about them. Awareness implies critical engagement, mathematics, language, relevance, openness, agreement, golds standards, and community leadership.

Mike Taylor

We are experiencing a collision of worlds. Terms like 'H-Index' are hard to understand, but are well defined. Terms like 'social impact' sound as if they're well defined, but aren't. There are particular problems about the 'community' being rather diverse. There are multiple stakeholders (funders, academics, publishers, start-ups, governments, quangos), international perspectives and varying cultures (from fifty years of research to a start-up). 

Taylor suggested an example metric - 'internationalism'. Measures could include: how well an academic's work is used internationally; how well that academic works; through readership data; citation analysis (cited, citing); co-authorship; funding data (e.g. FundRef); conference invitations e.g. ORCID; guest professorships; text-analysis of content.

Taylor doesn't think metrics is a place where publishers will have the same kind of impact that they might of 30 years ago. He said to expect to see more mixed metrics with qualitative and quantitative work. Taylor concluded that metrics are being taken more seriously (being used in funding decisions). Many stakeholders and communities are converging. 

Big data + cloud computing + APIs + openness = explosive growth in metrics. 

It is a burgeoning research field in its early days. Publishers need to be part of the conversation. We need to enable community leadership and facilitate decision making.

Data, the universe and everything. How data can drive your business

Melinda Kenneway: loves data

Melinda Kenneway, Chair of Data, the universe and everything: How data can drive your business seminar, opened the day by emphasizing how data underpins modern business. She summed up the premise for the day by quoting Jeff Weiner from LinkedIn:

‘Data really powers everything that we do.’ 

If we’re not selling content, what are we selling? Data is also becoming a product or service itself now. An era of big data can help us understand and deliver a much better experience for customers In 2012, 90% of all the data that existed in our entire history has been created in the previous 2 years. We need to get the basics of data right, but also have a strategy.

It’s knowing what you need to know rather than trying to find out about everything. Have clear objectives and stick to them. Don’t forget to consider privacy and legal issues about data. Deliver useful services that build on what the market thinks is acceptable. And if you don’t like data now? You should get out of marketing.

Colin Meddings from DataSalon focused on why you should care about cleaning up your data. To start with it can embarrass or annoy your customers when it is wrong and it’s open to error when users don’t put in info correctly.

Colin Meddings: not vague about data

Continuing with quotes from the great and the good, he cited an observation from 2013’s UKSG conference by Liam Earney from JISC:

‘Publishers can’t even tell a library what they subscribe to.’ 

We’re too vague about absolute fundamental data. The basic process of publishing involves: submit, review, copy edit, publish, purchase, read, cite, licence. All of these aspects generate data. The volume of data is huge and all these areas are where errors or mistakes can happen and where data can conflict.

Reflect on your efficiency with dealing in data. Do you struggle with data wrangling where you have to continually review and refresh datasets? See data as an asset in your business. It is worth investing in. You will get a return. Don’t forget that there’s a legal requirement around data to ensure it is accurate and fit for purpose. He recommended the DQM Data Governance Maturity Model. Be aware. Become reactive. Turn this into proactive. Then make it a managed process. And finally, it becomes optimal. 

Meddings outlined the seven deadly sins of data quality:

1. Missing data 
2. ‘Siloed’ data 
3. Invalid data 
4. Out of date info (people move around) 
5. Inconsistencies 
6. Right information, wrong field 
7. Duplicate and conflicting data. 

Put some effort and resource into sorting data. Task to someone specifically, create champions, set targets. It’s great if you can employ a team, but if you can’t, get senior management buy-in and work with those who have aptitude/passion for data. Start with one problem, don’t try to fix all at once.

Do a data audit. It can be manual. People who work with data everyday will know where the problems are. Sometimes an automated audit is better – good at finding information about your data for identifying empty fields, etc.

The Future for Smaller Publishers: Strategic marketing

Camilla Braithwaite: strategic marketing is key

Camilla Braithwaite, Marketing Manager at TBI Communications, spoke on the power of strategic marketing at The Future for Smaller Publishers seminar this week. She urged delegates to make the most of their assets and to build on organisational strengths and advised that in order to future proof yourself, you need to put yourself in a position to adapt and seize opportunities.

Strategic marketing is how a company can position itself within the competitor environment and is the business end of marketing (not promotional) which focuses on who customers are, what their needs and problems are, and what products and services you can develop to meet these requirements.

Smaller publishers face a number of challenges including (a lack of) market share and limited resources. They have to tackle the needs of a younger generation and find a way to negotiate open access mandates and new technology. In addition, competition for resources and building brand awareness can also be tricky.

Understanding your environment is the first step
Consider competitors' activities - who are they and what are they doing? Think in terms of your customer's workflow - who competes for your attention (it's not just other publishers, non-traditional competitors need to be considered). Author awareness is key - be aware of their rights. With many new entrants volume of content has a major impact on businesses large or small so discoverability and awareness are as important as ever. As there is continuing dissatisfaction with big profits from both librarians and researchers, this presents an opportunity for smaller organisations.

SMEs have a number of key strengths
These include proximity to community base, niche knowledge, understanding, being viewed as the 'good' in publishing, and flexibility. A small publisher's greatest asset is the connection to your community. Use your grass roots approach for advocacy, use your customer connections. Many smaller publishers will have Facebook, Twitter, a blog or forums they can use and you can encourage, reward and debate with customers through these channels. A lot of publishers are taking an advocacy approach using social media (e.g. British Ecological Society and SAGE Social Science) where you can position yourself at the heart of the community.

Build your brand - make more of it
What else does your brand represent for your customers? Is it seen as relevant and useful? Can you improve its profile among a new generation of researchers? Use the external perspective of key stakeholder groups to build up your brand values - then identify gaps to work on building something stronger and more relevant for them.

Be agile and innovative
Because you are smaller you can respond to changes around you and harness creative ideas within your organisation. Innovative doesn't have to mean expensive or be about bells and whistles. Jason Hoyt, co-founder of ALPSP Publishing Innovation Award winner PeerJ, recently said that rather than being content driven, publishers need to be more technology driven. What about being customer driven? Innovation workshops are a way to help you leapfrog barriers to meet a need that customers don't know they need yet.

Good sales and marketing strategies focus on strong territories and partnerships
This will enable you to build on areas with the most potential and is a good way to get global representation. Set targets and use a dashboard to monitor, control and understand your sales pipeline. Assess where you have market penetration which can be increased with some work and use targeted sales support to help achieve that (e.g. telephone).

Think about introducing clever pricing
Model it around the size of institution and adopt tiered pricing models. Analyse current sales and model different subscribers to identify four different tiers. This works for customers, is manageable for subscription agents and can improve your sales. A key part of keeping attrition low with tiered pricing is the communications with institutions when you introduce it. Keep as transparent as possible.

Charlie Rapple on leveraging authors' expertise and networks to increase discoverability and usage

Charlie Rapple, co-founder of Kudos

Charlie Rapple, Associate Director at TBI Communications, provided a stark reminder of the environment researchers and publishers operate in at our seminar 'It's all about discoverability, stupid!' held on 10 July.

The problem? There are more and more articles which people have less and less time to read. This results in too many articles that are uncited.

A paper by Michael Mabe and Mayur Amin from 2001 projected 3.26% per annum growth in article numbers (Scientometrics, 51:1 (2001) 147-162) which equates to a lot of papers. In the report The Role of the Critical Review Article in Alleviating Information Overload (Annual Reviews 2011) it was found that 81% of early career researchers felt they should read more of the literature than they do, and 25% suggested they would need to read for more than 24 working hours a week to keep up.

If there is less time for reading, then too many articles will go without being cited. A further article - Five-year impact factor data in the Journal Citation Reports - from 2009 by Péter Jacsó in Online Information Review, explores new indicators based on the level of uncitedness of articles in journals.

The fact that reader time is at a premium is an issue for the reader, author, publisher and librarian. There is a filtering challenge. The standard way of communicating what is in a formal scientific paper is through an abstract, but when you read it, all but an expert might struggle to get to grips with the content. As a result they may miss reading the paper.

One such example is a paper co-authored by R Giles Harrison at the University of Reading entitled 'Electrical signature in polar night cloud base variations' (Environmental Research Letters, March 2013). As a student, if you read the abstract, the first paragraph of which is extracted below, you might not be inspired to go to the full text download.

"Layer clouds are globally extensive. Their lower edges are charged negatively by the fair weather atmospheric electricity current flowing vertically through them. Using polar winter surface meteorological data from Sodankylä (Finland) and Halley (Antarctica), we find that when meteorological diurnal variations are weak, an appreciable diurnal cycle, on average, persists in the cloud base heights, detected using a laser ceilometer."

However, if you view the accompanying video posted on YouTube, you immediately see the headline story, put in simple terms, and communicated by a researcher who is passionate about and committed to his work.



This approach was subsequently picked up in the wider press enabling the paper and author to gain much wider coverage raising the impact of the work.

Harnessing the authors' expertise and explaining research for different audiences is an essential part of this approach. Putting the research in context and explaining why is it important while mapping to human interest or hot topics can help raise the profile of the work and widen the readership. If you can boil down what's unique about it while providing some background on who or what was an inspiration, you can convey the passion behind your work and go beyond formal formats. This is likely to lead to research being cited. Currently, it is time-consuming and manual to draw out the "story" and only a minority of research gets this treatment

Capturing lost value
There are a lot of assets already built on authors' expertise including: lay summaries, impact statements, benefit to society statements, novelty statements etc. All this type of information could be enormously helpful to PR teams and researchers, but is currently held in silos. Part of the problem is the limited infrastructure for capturing and putting these assets to work. There is a real opportunity for the author as a more authoritative hub using existing social media networks such as Facebook and Twitter as well as research-oriented sites such as MyExperiment.org, ResearchObject.org, Mendeley and CiteULike.

Rapple cited the work of Melissa Terras from UCL who blogged and tweeted about uploading papers to the repository, initially to relieve the boredom. She told the story behind the research with every blog post, and was intrigued to note spikes in downloads of her papers that corresponded with her use of social media. This led her to observe:

"If you tell people about your research, they look at it." 

This demonstrates the potential to close the loop, break down the silos and reclaim the value of contextual information that is currently lost or fragmented.

Bringing it all together
These three problems of too much information, not enough time; valuable assets left in silos; and valuable networks being under-utilised require a solution that involves better use of metadata and multimedia. Rapple has been working on a new project, currently in beta, called Kudos, that tries to address this.

Whatever the solution, it will need to take into account:

• Article level metrics (altmetrics and usage marketing)
• Author services (open access, academic spring, community, advocacy)
• Discovery (SEO, social media, international reach, metadata, marketing to individuals)
• Closing the loop (data driven services, integration not duplication, social machines)
• Intelligent reading (filtering, multimedia, public accessibility)
• Research evaluation (funding, REF, STAR, ORCID, H-Index etc)