How do we measure an individual's research impact?

This is a subject that I thought I already understood fairly well, having expressed on this blog and elsewhere various sceptical views about the whole notion of impact, including some of the commentary about it that has come out in connection with REF discussions and the like.

Some of the debate about impact has certainly been pitched at a fairly low level, and has often not taken us a great deal further than the well worn territory of either assessing 'research quality' - of an individual or a department (or, more formally, a unit of assessment) - in quite subjective terms; or making the assessment on the basis of publication numbers, journal rankings and citation counts. Not only has this not advanced matters much beyond the old RAE system, but it is open to all the familiar arguments: how many articles to count; how far back to go; how to judge the quality of an individual article, etc. Like a lot of important things in life, research quality seems to be one of these things where 'we know it when we see it'.

However, a few days ago I was intrigued to find that I didn't really understand this topic of measuring research impact as well as I thought. It happened like this. I had been approached by a couple of academic bodies in Italy to help them with two tasks: (i) assessing the research quality of individual academics, by reading and ranking a selection of their papers; (ii) assisting Italian universities to make new appointments at professorial level. I presume my name was put forward by a colleague in Italy, but in order to take things forward the Italian academic bodies naturally asked me to provide my CV and to complete a couple of simple forms. Whether, in the end, the Italians ask me to do these academic tasks, remains to be seen, but completing their forms was unexpectedly challenging.

For one of the forms asked me for the h-index and g-index of my research, something that left me totally baffled as I've never heard of them. Just to deal with the forms I did a quick Google search using the string 'Paul Hare h-index'. To my surprise this came back with h = 5 and g = 7. I still had no idea what they meant so I just put them on the forms and sent them off as requested.

Once that was done, though, I had to investigate further.

It then turned out that the h-index originated in a paper by Hirsch published in a physics journal in 2005. His idea is a really simple one. If a researcher has an h-index of 22, say, it means they have 22 papers with 22 or more citations. The h-index is the maximum number of a researcher's papers where this property holds. Thus someone might have published 80 or so papers, but most will only have a handful of, or even no citations. With 80 papers, the h-index could turn out to be 25, if that researcher - among all his/her output - has 25 papers with at least 25 citations (and not 26 papers with at least 26 citations).

The g-index, by the way, is a modification of the h-index to weight more heavily the articles that gain more citations.

What you realise when you think about all this is that anyone's measure of h (or g) will depend on the specific database that is examined - what papers are included in it, what citations are counted, etc. This means it's not easy to get a wholly reliable measure of h, because available research publication and citation databases are nowhere near good enough. Whatever database is used, for any given researcher their h value should rise over time, implying that it's probably not a great indicator for early-career researchers. Moreover, if different databases are used to investigate a given individual, one can expect to find different h values. All one can say, then, is that the person's true h-value is at least as large as the largest value one manages to find.

To illustrate the effect of using an alternative database, I downloaded an amusing bit of free software called Publish or Perish (Harzing, A.W., 2007, Publish or Perish, available at this link). This searches through the Google Scholar database, and using this approach I found the following values for my research: h = 20 and g = 33. I still don't really know whether these values are good, bad or indifferent, as I haven't checked the corresponding values of any colleagues and haven't seen these research performance indicators reported before.

Should we try to use indicators like this in the forthcoming REF? It seems to me worth thinking about, despite the practical difficulties I have mentioned above.

What do academics do?

This is something my neighbours probably wonder about, and my mother certainly does! And it's quite tricky to answer, because we do so many different things.

Our contracts don't help much here, as they are usually pretty vague. Our job description is usually no more than a line or two, stating which department we are in and that we shall do whatever work is agreed with our line manager. It doesn't always even mention teaching or research, so I suppose in extremis we could be asked to do the gardening or clean the corridors.....I'd better not pursue this line of thinking, though.

More seriously, the presumption is that our main activities are teaching students and doing research, together with whatever administrative tasks are needed to fulfil these duties. When I first came into academia in the early 1970s, this was all taken for granted, and there were no targets, performance indicators, and the like to guide us; nor was there any teaching assessment (either by students or externally run) or research assessment. Nowadays, of course, life is more complicated. As in many other areas of life, academics are no longer trusted - either by their institutions or by the wider community - to be competent professionals who will simply get on with the job if left in peace. So we do have targets and regular monitoring of what we do, and these serve both to make sure we don't idle around too much, and to influence promotion and other rewards.

On the face of it this all sounds eminently sensible, and some might say long overdue. But the key problem with targets is that they inevitably pick out bits of a job that are readily measurable, and miss out the less tangible, more qualitative aspects. For an academic, that means I can be told how many hours to teach, and perhaps even set a target of how many papers to write in a given year. But it's hard to make me deliver quality unless I have the inner motivation to do so, and even worse, focussing on such targets might make me neglect other important parts of the job that can't be summed up in a few targets.

The point is that academics are not only employees of a given institution, but part of a wider academic community - usually based around our individual subject areas - within which very strong values of collegiality are still very important. This wider context leads us to do all sorts of things for which we receive zero or minimal reward, for instance: reviewing research grant applications, external examining, refereeing papers, reviewing book proposals for publishers, writing book reviews, informally commenting on colleagues' papers before journal submission, and so on. It seems to me really important that we do these things, and do them well, but the institution-based performance indicators that increasingly influence what people do offer no encouragement whatsoever. I think this is a great pity.

I'm not sure what my mother would make of the above remarks, but I hope that some of my academic colleagues will find that they strike a chord.