REF rankings – research power vs GPA
Times Higher Education published an interesting alternative to the results of the 2014 research excellence framework (REF) that compares the rankings of institutions by intensity weighted scores. The league tables rank institutions according to the grade point average (GPA) judged by the quality profile submitted by eligible staff. However, the approach is selective and does not take into consideration the number of eligible staff submitted by each department.
Times Higher Education published rankings based on research power, which is calculated by multiplying an institution’s GPA by the proportion of staff submitted. This method gives a better sense of which institutions are likely to win the most quality related research funding since it considers both the quality and volume of an institution’s submission. But if the institution has a large department, then the methodology does not work since large volume means greater research power, despite selectivity. Also, staff who were not submitted to REF receive a big fat zero, dragging down the GPA values even further.
In light of this, the methodology of research power should be considered. For example, Queen Mary University of London (where I work) was ranked 11th in the 2014 REF when calculated by GPA alone. QMUL submitted 21 UoAs, with 671 FTE eligible staff submitted out of 911 (or 74% eligible staff submitted). This resulted in a GPA value of 3.18 with a good position in the GPA league table as joint eleventh. When ranked by intensity weighted GPA using values from the Higher Education Statistics Agency, we did far worse and our position drops from eleventh to thirty fourth. This assumes that the 26% staff not submitted are eligible with a research quality score equivalent to zero, only.
Whilst the research intensity rankings makes a useful comparison with the standard GPA rankings, there are problems with the methodology and raises the issue of volume and selectivity. Clearly, institution’s that are more selective will have greater research quality with a better position in the GPA league table. The research power model accounts for eligible staff and has a significant impact on the institution’s rank position. It is therefore not surprising that institution’s with greater staff, for example, University College London will do better on intensity weighted GPA since they submitted higher proportions of staff (2566 out of 2810 or 91%) when compared to QMUL (671 out of 911 or 74%).
The question now arises what would happen to the research quality rankings if we had submitted all our eligible staff. Perhaps another alternative is to focus our discussions that present research outputs per UoA, since you cannot argue the game when individuals have been recognised as world leading (4*).