REF - Transparency and Feedback

The latest issue of Times Higher Education includes an article by Mark Burnley about the REF. He complains that when the results of REF2014 are eventually published, while individual units of assessment will receive their grading in the form of a distribution (or profile) rather as occurred with RAE2008, individual academics will not get any direct feedback telling them how well or otherwise they are judged to have performed. Mark Burnley is clearly unhappy about this situation, and thinks it should be changed.

One of the reasons he gives is that if a given unit is assessed as having, say, 10% of its research of outstanding, world class quality, how can it know which 10% and who delivered that spectacularly good work? His point, I think, is that if the unit doesn't know that, it can't learn from the outcome of REF2014 and do even better next time. More generally, he claims that academics need individualised feedback from the REF to enable them to understand where their weaknesses are and hence, what they can do about them (for next time). However, I think this way of using the REF results is inappropriate, it's not what the system has been designed for, and there ought to be other ways for academics to get the feedback they need. Let me explain.

(a) The REF has been designed to evaluate units of assessment, not individuals, and its fundamental purpose is to provide performance indicators that will be used by the Funding Councils to allocate the research component of their grants to institutions (hitherto, the QR stream) for some years following REF2014. Thus its function is to guide the allocation of financial resources, not to evaluate individuals.

(b) Next, if a unit of assessment does indeed have 10% of its research adjudged 'outstanding' then it seems to me quite bizarre to imagine that members of that unit, or managers in the relevant institution, might find it difficult to identify which parts of their work are so extremely good. If they can't figure out for themselves which elements of their research are world class, then surely something is badly wrong......... After all, we would surely like to think that the REF panels, etc., will do their work as fairly and objectively as reasonably possible, so the designation 'outstanding' is hardly going to be conferred pretty much randomly. One hopes not, anyway!

(c) Last, quite regardless of the RAE, and now the REF, most academic institutions that are serious about developing and improving their research already have systems in place for assessing individual staff - setting targets periodically, monitoring performance, providing training, and so on. Not always, but very often, promotion is the reward for successful delivery of quality research output over a period, and for most of us, promotion is a rather strong incentive to perform well. Of course, acknowledging all this doesn't stop many of us from griping about the iniquities of our internal evaluation systems, and I'm sure many of us don't much like this sort of thing. But it's bound to be more finely tuned to suit a given institution than the rough and ready REF could ever be, so whatever we do internally is likely to give most academics a fairer hearing.

Overall, therefore, the REF has an important job to do, so let's simply allow it to do that job well, without trying to give it additional tasks for which it was never designed. And in case the reader might wonder, I'm not a huge fan of the REF, but it's there, and is not about to go away. We just have to live with it - it's part of our university world nowadays!

Evaluating research – Hungary

Hungary is easily my favourite country in Eastern Europe, and Budapest my favourite city, in both cases probably because I have been so often, know the place well, and have many good friends and colleagues there. So a few months ago, I was delighted when the Hungarian Academy of Sciences invited me to join a new committee they were setting up, namely the External Advisory Board (EAB) for the Academy’s Institute of Economics. Like many things that academics do, this is not a paid position, but the Institute does cover the expenses to get me out to Budapest for meetings. And the first meeting of the EAB took place in Budapest earlier this week.

Accommodation was arranged for me at a guest-house of the Academy, located in the old Castle District of the city, though when we got to the designated address my taxi driver didn’t believe it could be right, as it looked like an ordinary house or apartment building, albeit dating from the 18^th century. Luckily, a tiny sign beside the imposing solid wooden door read, ‘MTA Vendégház’, confirming that we were at the right place. Once inside, the welcome was friendly, facilities were fine, and all was well.

Next day our EAB meeting took place at the building that houses the Institute of Economics, a truly horrible concrete edifice whose sole redeeming feature is that once inside, it affords – at least when looking west – lovely views of the southern end of the Buda Hills. We were there to assess the Institute’s annual report for 2009 and to offer advice on its research strategy, but naturally I cannot comment on such things here since our formal business is confidential between the Academy, the EAB itself, and the Institute. However, I can say that the quality of the Institute’s work, and its leading researchers, bears no relation to the quality of their building; they do lots of quite outstanding research and many researchers at the institute have well established international profiles. Given that, it is always a pleasure to visit the Institute, and it was nice, while there, to encounter several researchers I know well from previous visits.

Just like universities in the UK, the Institute faces severe funding pressures these days, and this is forcing it to diversify into more ‘commercial’ types of research and research-related service provision simply to balance the budget. I imagine that some researchers, who are used to the more old fashioned, traditional model whereby they could spend a lot of time ‘sitting and thinking’, must find such trends quite uncomfortable and unwelcome. But again, just as with us, funding pressures are not going to go away, and we all have to find ways to live with them.

What about Budapest itself? As ever, it is a lively city, with lots of tourists still wandering around, both individually and in orderly groups. Some older buildings, such as the main cathedral on the Castle District, are undergoing restoration, and some major construction projects are also on going, notably the fourth metro line, Metro 4. Otherwise, everything seems as normal, except for one surprise when I went down to Moscow Square, a major public transport hub (metro, buses and trams all link up there). For I discovered that my favourite tram-line, the number 56, no longer exists! This is the tram I’ve taken for decades when going out to the Buda Hills – it went all the way out to a little place called Hüvösvölgy towards the northern end of the Hills. Although on this trip I had no time, sadly, to visit the Hills, I was relieved to see that the route of another tram, the number 61, has been extended to go out there. So next time…….