Metrics and research impact

When the Research Excellence Framework (REF) was put forward as the new approach to assessing research in UK universities (replacing the RAE), it was widely claimed that it would offer several advantages over its predecessor, most notably:

• it would be a less costly system to run;
• it would measure the volume and quality of research better than before by making extensive use of metrics; and
• it would find ways of evaluating the impact of research carried out during the assessment period.

In this note, I consider each of these in turn.

Cost of the exercise

So how costly is costly? I have seen estimates produced by HEFCE and others suggesting that the former Research Assessment Exercise (RAE) cost around 1% of the funds eventually allocated as a result of the exercise (namely the QR funding stream discussed in an earlier post). To this one might want to add another 1-2%, perhaps, to allow for the costs incurred by individual institutions in preparing their submissions, and other institutional costs. This gives 2%, possibly as much as 3% of the research funds allocated using the RAE. Not a trivial sum, but not terrible either, especially when one compares it with the reported costs incurred by the ESRC and other research funders when they allocate responsive-mode research funding by evaluating individual research project proposals. The costs involved here are apparently around 20% of the research funds actually paid out. Two conclusions follow:

(a) The RAE wasn't all that costly, so it is quite hard to believe that saving money was a key reason for shifting to the new REF;

(b) If we really are worried about the costs of allocating research money, there should be more concern about the responsive mode as it is a lot more costly; and it would be far more so if evaluators and reviewers were actually paid for what they do.

Measuring research using metrics

As originally understood, 'metrics' in the REF referred to some of the more widely used citation indexes, and databases that logged publications in a wide range of outlets, supplemented by measures of journal quality and impact. The hope was that using such publicly available resources would save on costs by making it less necessary for institutions/units of assessment to assemble all their output data for themselves. On closer investigation, this approach turned out to be less appealing than initially thought. For even in the sciences and engineering, publications coverage was incomplete and in the arts and humanities the general picture was pretty hopeless. Citations, too, were incomplete, and reflected very much the different research practices of different disciplines. The situation might evolve, of course, but for now it seems that in many subject areas these fancy new tools might only be usable as supplements to what was already being done. Moreover, it quickly became evident that once metrics of some sort were agreed, they could give rise to lots of undesirable game playing, such as colleagues citing each other multiple times. If such distortions were thought to be widespread, the whole approach could be discredited (after all, people do respond to incentives!).

Research impact

The latest idea, quite a silly one in my view, is that we should pay a lot of attention to the 'impact' of our research - and hence look for ways of measuring and evaluating this impact. Impact on what, one might ask. Fundamentally, I suppose, we ought to be interested in the impact on 'knowledge', hard though that would be to judge. In my own area of the social sciences, we are supposed to think about the 'impact on society', and more specifically, in economics, 'impact on the economy', and sometimes even 'impact on UK exports' (yes, seriously!). This is such a ridiculous idea that I find it hard even to discuss it seriously, but let me try, at least briefly.

First, the impact of a piece of research is something that takes time to become apparent, sometimes decades, so evaluating research impact over as short a period as 5-10 years is generally nonsense. Second, within any research group, e.g. an academic department or School, there is a normally a very wide distribution of research output, with 2 or 3 individuals often producing more than half the output of the group. Moreover, for any of these highly productive individuals, their own output will vary a good deal in quality and (potential) impact - for most good researchers are lucky if they produce more than a couple of really startling ideas in an entire career. A lot of what they do simply builds on and develops earlier ideas. Given these normal features of quality research, an insistence on measuring impact, as currently proposed, will result in a lot of artificial and trivial indicators of impact being developed. Thus we might measure the number of invited lectures a researcher gives, the number of radio broadcasts, the number of TV appearances, and so on - these are the sort of measures already collected for the old RAE, part of the 'esteem' aspect of research. But none of this gives much of a clue as to the longer term social benefit of the research concerned - but the trouble is, I'm not sure that we can often say much about that until long after the research has been done.

It would probably help if those who are keen on measuring the impact of research would try harder to explain what it is they really want to measure.