University Rankings, Diversity, and the New ... - Sense Publishers

USHER AND MEDOW
With only a few exceptions (notably, Spain’s Excelencia rankings), league table
systems then take the data on each indicator and turn it into a ‘score’. Usually, this
is done by giving the institution with the highest score on a particular indicator a
perfect mark of 100 and then awarding lower scores to other institutions based on
how close they were to the score of the top institution.
Once scores have been derived for each indicator, they are weighted, with greater
weight being accorded to indicators which are believed to be of greater importance.
The weighted scores from all indicators are then tallied to give a unified final score
for each institution. This weighting and aggregation is not always entirely transparent.
There are examples like the United Kingdom’s Guardian and the Dutch Elsevier
rankings where indicators do not possess specific weights; nonetheless, because
institutions are still displayed from top to bottom, it is implicit that all indicators
are being given an equal weight and the resulting scores are being aggregated in
the normal way.
Clearly, the choice of indicators and the weight given to each indicator make an
enormous amount of difference to the final output. Indeed, it is no exaggeration to
say that when publishers advertise their product as a guide to ‘the best’ institutions,
it is the publishers themselves who largely decide the best simply through their
choice of indicators and weightings. In effect, the act of choosing a set of indicators
and weightings imposes a one-size-fits-all definition of ‘quality’.
THE EVIDENTIARY BASIS OF LEAGUE TABLES – HOW DATA IS COLLECTED
A key issue in the preparation of league tables and rankings is the method by which
data is collected. There are basically three sources of data on institutions:
– Independent third parties. Frequently, government agencies will collect and publish
data on institutions in their jurisdiction, and this can be used as an objective
standard by which to compare institutions. This data is very often financial in
nature and is based on administrative data from grant-making bodies.
– University sources. The most complete and most detailed sources of data on
universities are of course universities themselves, and they are thus potentially a
very rich source of data.
– Survey data. Surveys of the opinions or experiences of various stakeholders can
be used to obtain comparable data on different institutions regarding educational
quality.
The use of each source of data has strengths and weaknesses. Independent
third-party administrative data (usually from governments or grant-making bodies)
is generally considered the ‘gold standard’ of comparative data since it is, at least
theoretically, both accurate and impartial. The problem is that this data is not
(usually) collected for the purpose of compiling league tables but rather as an
administrative by-product of ordinary business. As a result, over-reliance on this
source of data can lead to a situation where indicators are chosen simply on the
basis that data is available rather than because they contribute to a sensible definition
of quality. Certainly, there are few if any measures from these sources which
6