---
layout: post
title: BIS, metrics and non-selective QR-allocation
categories: [metrics, REF, QR, green paper]
tags: [metrics, REF, QR, green paper]
published: True

---

This post originally appeared in an [edited form on Wonkhe](http://wonkhe.com/blogs/bis-metrics-and-non-selective-qr-allocation/).

[The Green Paper for Higher Education](https://bisgovuk.citizenspace.com/he/fulfilling-our-potential) made no bones about the fact that it thinks that REF is too expensive. Fuelled by academic discontent with REF, the paper clearly seeks to co-opt academic protest in order to shape future exercises towards metrics-based approaches.

However, there's a snag for the government. The influential recent [independent review of metrics for research assessment](http://www.hefce.ac.uk/rsrch/metrics/) clearly stated that there is no measure at the moment that can capture the research landscape. This is obvious to anyone who works in a university; the current regime of peer review-based national research assessment, however flawed and expensive, both carries the confidence of universities and provokes a number of management behaviours (also alluded to in the Green Paper, though selectively, and only in a negative sense). As such, while it is straightforward to imagine ways to translate flawed evaluation procedures into mechanistic funding allocation systems, it can’t be done by flicking a switch. The devastating damage to the UK HE research sector would be widespread and intolerable.

Given the above, it's interesting that BIS/HEFCE recently advertised the public sector tender contract “[UK SBS BLOJEU CR150082HEFCE Provision Of Citation Indicators](https://www.contractsfinder.service.gov.uk/Notice/26d25c44-4c38-4d4c-93ca-9a580ac39724)”. Advertised for just 10 days starting on October 29th (one day before the Green Paper was released) and with a fulfilment deadline of 10 days after the contract was awarded, this contract asks for “robust citation data relating to institutions’ research outputs in the period 2008-2014” containing the following data:

For each journal article in the period 2008-2014 with at least one author from a UK HEI, we require: 

* Article DOI
* Field-weighted citation impact (FWCI)
* An indicator of the citation impact of the journal. This should not be the journal impact factor, but an indicator which seeks to address some of the issues identified with journal impact factors, such as the Source Normalized Impact per Paper (SNIP).
* Citation percentile
* Name(s) and UK nation of the UK HEI(s) included in the address field
* Journal subject classification
* Year of publication

This data is to be provided so that HEFCE can “undertake further analysis on the data”. In each case, “journal subject classification should be mapped onto REF 2014 units of assessment”.

The contract is worth £16,667, which means that the provider is not going to do the work within a ten-day period, but rather is going to export an existing dataset and give it to the government in a form that they can play around with. There are only two providers that I know of that will have this data to hand and ready to go within ten days: Elsevier and Thomson Reuters. I bet one of them gets the contract. We also know that BIS and ministers [meet frequently with these organisations](https://www.whatdotheyknow.com/request/meetings_between_bis_officials_a).

(Also, for information, the specifically requested Field-Weighted Citation Impact “is the ratio of the total citations actually received by the denominator’s output, and the total citations that would be expected based on the average of the subject field” (see page 55 of the [snowball metrics report](http://www.snowballmetrics.com/wp-content/uploads/snowball-metrics-recipe-book-upd.pdf) I don't know, but I do suspect that Elsevier is the only provider with the FWCI data as they provided [this report](http://www.rcuk.ac.uk/media/news/the-uk-has-overtaken-the-us-to-rank-1st-by-field-weighted-citation-impact/) to BIS previously.) 

So, what's going on? It's not worth being overly conspiratorial about it; it might be nothing and one would sincerely hope that a £16.5k tender could not be given equal weight alongside a hefty and legitimated independent review of metrics. Process must count. But the Metric Tide already did a comprehensive citation correlation analysis to previous REF results, backed by Elsevier's own sanction that the metrics simply aren't good enough to replicate the peer-review panels. So it's not likely a correlation exercise. The only thing I can think that it might be, then, is about modelling a simpler non-selective (or random sampling) exercise to allocate funding. This is, after all, a bigger dataset than the correlation analysis in the Metric Tide. It comprises all outputs with UK HEI authors, not just those submitted to the last REF.

Indeed, eliminating selection from REF would yield some of the savings that BIS would like. Given wider political agendas around teaching, scientific selectivity, and research concentration, one might also infer that the devastation might be deemed acceptable if it left Cambridge, Oxford, Imperial and/or UCL unaffected. If Manchester can do a bit better, that's probably OK as well. If they can find a way to use citation metrics to produce this kind of result (probably through some kind of skewed banding procedure that maintains the grip of the elite research-intensive universities on the bulk of research funding), while shaking up some of the other institutions, that might fly. But it would be awful for actual research quality. I also think that it would be extremely politically damaging to pursue such a route and would probably be contentious at various levels within the Conservative party itself.

It will all just depend upon how confident the government is that it can allocate funding in a way that does not need to carry the confidence of the HE sector, when all evidence suggests that universities are only likely to intensify their gaming and strategising behaviours alongside the introduction of more metrics. It's also clear that a metrics-based approach won't get rid of costs at HEIs; we'll all just end up paying Elsevier and Thomson Reuters for the data and analytics. It will also depend on how confident the government is that it can survive the backlash of going against a well-received, recent, comprehensive report that said that a purely metrics-based approach would be irresponsible, highly damaging, and invalid (although the government may be operating from a different premise). So if they go down this route, they're working themselves up for a big fight. On the other hand, they've already signalled that REF is too expensive in the Green Paper in their eyes. So if they don't change something, then they will be embarrassed. Hence, I suspect, the need to run the numbers now while everyone is distracted by the mess of TEF.