---
title: My responses to the Consultation on the Second Research Excellence Framework
layout: post
---
Here are my draft responses to the parts of the [Consultation on the Second Research Excellence Framework](http://www.hefce.ac.uk/pubs/year/2016/201636/) that attracted my interested. These are my individual thoughts, not those of any institution that I represent. They are also not my final submission.

__1. Do you have any comments on the proposal to maintain an overall continuity of approach with REF 2014, as outlined in paragraphs 10 and 23?__

With respect to the timings proposed in paragraphs 23 and 24: the period until submission is now very short. Since the decisions will only be published in the summer of 2017, but that these decisions may include changes to staff submissions, it is difficult for institutions to plan their submission. As below, if all “research-active” staff are to be submitted, this has given institutions just two years notice on this front. Although other changes such as variable output numbers will mitigate this, the short submission period may put undue pressure on some staff via their institutions.

__3a. Do you agree that the submissions guidance and panel criteria should be developed simultaneously?__

The simultaneous development of submission guidance and panel criteria is welcome and would ensure more cohesion in this area.

__3b. Do you support the later appointment of sub-panel members, near to the start of the assessment year?__
 
The later appointment of sub-panel members poses a problem. It is important, if REF is to maintain the support of the sector, that criteria and guidance represent disciplinary norms, even where there are contradictory opinions on what this represents within a field. It seems crucial, then, that the criteria appear as fully co-owned by sub-panel support in their development.

__5a. Based on the options described at paragraphs 36 to 38, what approach do you think should be taken to nominating panel members?__
 
I support a continued nomination process here with the organisations listed to be excluded. Care should be taken, though, in some fields such as Library Information Science to ensure that commercial entities with conflicts of interest are also excluded.

__5b. Do you agree with the proposal to require nominating bodies to provide equality and diversity information?__
 
I support the increased burden here for equality and diversity purposes. By some measures, the UK fares very badly on equality and diversity among the senior professoriat. To ensure that we work to counter this, the additional burden here seems worthwhile.

__7. Do you have any comments on the proposal to use HESA cost centres to map research-active staff to UOAs and are there any alternative approaches that should be considered?__

I support this proposal to prevent gaming by institutions through UoA assignment.

__8. What comments do you have on the proposed definition of 'research-active' staff described in paragraph 43?__
 
The definition of research-active staff here seems sound, especially with the exemption for RAs. In terms of the danger of contractual changes, a mitigating strategy would be to penalise institutions that proportionally reduce research-active staff by a certain percentage over the census period (i.e. take REF2014-timing HESA figures and then compare with REF2021 figures, offset/baselined against total academic staff at an institution).

__9. With regard to the issues raised in relation to decoupling staff and outputs, what comments do you have on:__

__9a. The proposal to require an average of two outputs per full-time equivalent staff returned?__
 
The potential challenge that I see here lies in the disadvantage to smaller UoAs as opposed to larger departments/research areas. Through the deregulation here that does away with oversight of individual circumstance exemptions, in a smaller unit submission there will be fewer other members of staff to draw upon to make up for outputs in, say, cases of maternity leave. In larger units, there will be more staff and it will be easier to up the level by spreading outputs among other members.

I wonder, too, whether the claimed burden reduction for selectivity is here overegged. That is, institutions will still internally rank and evaluate outputs for submission, they will just not have the additional constraint of ranking within each submitting staff member.

__9b. The maximum number of outputs for each staff member?__
 
I agree that a maximum cap is a good idea and the proposed limit of six here seems to work well.

__9c. Setting a minimum requirement of one for each staff member?__
 
A minimum of one output is a good idea and better than zero. It does not seem right to ask for the return of all research-active staff members but then to not assess at least something from each of those individuals.

__10a. Is acceptance for publication a suitable marker to identify outputs that an institution can submit and how would this apply across different output types?__
 
Acceptance for publication, marked by deposit within an institutional repository within three months, is a sensible timeframe for some types of output, but hardly for all. Monographs in the humanities, as just one example, can take a very long time to produce and so should be exempted if this measure is used.

However, a more sensible move may be to stipulate that new staff members at an institution may not submit outputs accepted within the first six months of their arrival. There could be a comment added that additional scrutiny on the time of deposit in relation to time of acceptance with respect to outputs submitted in the first year of a staff member's hiring. (But this will require more monitoring.)

__10c. Would non-portability have a negative impact on certain groups and how might this be mitigated?__
 
I believe that the arguments that non-portability would adversely affect certain groups, such as ECRs, is over-stated. Such arguments, when I have heard them, have circled around the problems of being hired. However, if no institution can expect to hire in order to use outputs, then the playing field is level. Providing exemptions to specific staff disadvantages others and will restrict the mobility of non-exempted staff. The benefits of non-portability incentivizing institutions to create a research environment that is conducive to producing good research and retaining staff seems more important here.

__10d. What comments do you have on sharing outputs proportionally across institutions?__
 
Sharing outputs would be a good solution to the problem of apportioning credit but I would note that it comes with additional monitoring requirements.

__11. Do you support the introduction of a mandatory requirement for the Open Researcher and Contributor ID to be used as the staff identifier, in the event that information about individual staff members continues to be collected in REF 2021?__
 
This is a good idea but I will note, again, that the timescale for its implementation is (too) short.

__18. Do you agree with the proposal for using quantitative data to inform the assessment of outputs, where considered appropriate for the discipline? If you agree, have you any suggestions for data that could be provided to the panels at output and aggregate level?__

No.

I continue to be concerned about the data quality and the use of citation measures to asses research quality. See the recent “Analysis of the international journal publishing activities in Switzerland with special emphasis on gold open access publishing” Data Paper by Max-Planck Digital Library, Big Data Analytics Group: Alexander Machado, Laura Hoppmann, Johannes Knaus & Margit Palzenberge at https://zenodo.org/record/167381#.WG-OqvGLT0o for an implicit critique of the SCOPUS database, for instance.

Furthermore, without reliable semantic information about the way citations are used, this method is not sound. For just one further example, see citation of Andrew Wakefield's now-retracted and fraudulent paper “Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children”: http://www.thelancet.com/journals/lancet/article/PIIS0140673697110960/fulltext.

__29. What comments do you have in relation to the inclusion of examples of impact arising from research activity and bodies of work, as well as from specific research outputs?__
 
I support this approach. I would also add that allowing impact activities conducted at an institution, based on a new staff member's work at a previous institution, should be permissible.

__36. Do you agree with the proposals for providing additional credit to units for open access?__
 
This is a good insertion. However, it would be better if the precise nature of the “credit” here could be determined, for instance by an allocation of a percentage within the environment score.

__43. What comments do you have on the proposed timetable for REF 2021?__
 
As above, I still feel that the timetable is too short given the lateness of this consultation.

__44. Are there proposals not referred to above, or captured in your response so far, that you feel should be considered? If so, what are they and what is the rationale for their inclusion?__
 
I note that a previous draft of this consultation document, prior to the Stern review, considered splitting the 4* category in two and raising the number of stars that could be awarded. I thought that this was a sensible approach and I am sad to have seen it dropped from the final version.