Peer review is once again a
key criterion in this year’s rankings. But research
quality is now gauged on five rather than ten years of
citations, making it more topical, says Martin Ince
The tables on pages 3-5 are the third edition of
The Times Higher/QS World University Rankings.
As in 2004 and 2005, they list the world’s top 200
universities according to a range of qualitative and
quantitative criteria. Our methodology this year follows
that we used in 2005 very closely.
Qualitative and quantitative forms of data each
account for half the total score. The qualitative data
is based on our belief that the people who know most
about university quality are those who work in them or
are closely connected to them.
For this reason, 40 per cent of the score allotted to
each university is derived from peer review carried out
among academics by QS Ltd, partners with The Times
Higher in compiling the World University Rankings.
This has involved gathering data from 3,703 academics
around the world. Each was asked which area of academic
life — science, medicine, technology, the social
sciences or the arts and humanities — they are expert
in, and then asked to name up to 30 universities they
regard as the top institutions in their area. This is a
robust and simple test, and is almost immune to fraud.
To achieve this large total of participants, we
amalgamated data from our surveys in 2004 and 2005 with
this year’s responses. However, only the most recent
response was used from any individual. In future years,
we shall not use data more than three years old.
This peer review shows that, although there are a few
dozen universities that are plainly world leaders, there
are also well-regarded universities in a surprisingly
large variety of countries, in both the rich and
developing worlds. Indeed, Top Universities
Guide, the book that accompanies this supplement,
shows that the top 500 universities in the world all
have their supporters. The top 200 come from 30
countries, while the top 500 come from 51.
This peer review is enhanced by a further 10 per cent
of the score based on the opinion of a vital group of
outsiders who observe the world’s universities closely.
These are graduate recruiters, especially those who work
internationally or on a substantial national scale. The
sample includes people from companies in manufacturing,
services, finance and transport, as well as from the
public sector. They were asked which universities they
like to recruit from, a question that we hope reveals
something about the quality of the students an
institution can attract and the teaching they receive
there. We sampled 736 recruiters.
Peer review is the standard way in which the quality
of individual pieces of academic work is judged. We
believe that applying it to institutions in the
controlled way we have done provides an up-to-date
measure of the dynamism of whole institutions and of
wide groups of subjects in them.
The other half of the rankings scores are made up of
quantitative measures. As with the whole of this
exercise, the problem is to obtain a measure of
university quality that can be calculated on a
consistent basis in widely differing environments. This
means developing questions that can be answered in a
valid and informative way in Norway as well as in
Brazil.
Teaching and research are the main activities that
occur in universities. Measures designed to capture the
quality of these activities account for 40 per cent of
the total score in our rankings.
We measure teaching by the classic criterion of
staff-to-student ratio. This is captured by asking
universities how many staff and students they have, and
dividing one by the other. In practice, things are not
quite so simple. One complication is to decide exactly
who is a student. We ask universities to count people
studying towards degrees or other substantial
qualifications, not those taking short courses. Staff
numbers, too, can be a matter of opinion. We ask
universities to submit a figure based on staff with some
regular contractual relationship with the institution. A
guest lecturer, however distinguished, should not count.
This measure is also prone to subject bias. Teaching
people to be surgeons or musicians is inherently more
person-intensive than transmitting some other forms of
knowledge. But because our analysis deals mainly with
large general universities, this variation should even
itself out.
The measure of staff-to-student ratio is intended to
determine how much attention a student can hope to get
at a specific institution, by seeing how well stocked it
is with academic brainpower relative to the size of its
student body. It accounts for 20 per cent of the
possible score.
Our next measure, relating to research, is intended
to examine how much intellectual power a university has
relative to its size. It is based on citations of
academic papers, since these are regarded as the most
reliable measure of a paper’s impact. The world’s
accepted authority on citations is Thomson Scientific in
Philadelphia, formerly the Institute of Scientific
Information. We use data from Thomson’s Essential
Science Indicators database, processed by Evidence Ltd
in Leeds. The ESI concentrates on the world’s most
highly cited and influential research. Our analysis uses
data covering 2001-06. This is a change from the first
two editions of the World University Rankings, which
used ten years of data. Using five years increases the
dynamism and rate of change of this measure, but still
provides a statistically valid amount — more than 40,000
papers and more than a million citations each for Texas
and Harvard universities, the world’s top two generators
of scholarship on this measure.
To compile our analysis, we divide the number of
citations by staff numbers to correct for institution
size and to give a measure of how densely packed each
university is with the most highly cited and impactful
researchers.
There are well-known problems with citations as a
measure of research. One is the underrepresentation of
papers in languages other than English in citations
data. Thomson is addressing this issue by sampling more
journals in Asian and continental European languages.
But it is also becoming less of a factor as English
becomes the language of choice for academic publishing
across the world.
As our introduction on page 2 makes clear, the
increasingly international nature of higher education is
a key reason for the existence of the World University
Rankings. The final 10 per cent of our score is intended
to determine how global universities are:
5 per cent is awarded on the basis of the percentage
of overseas staff each university has, and a further 5
per cent for its percentage of overseas students. This
measure is intended to help mobile staff and students by
giving them an impression of how international a
university may be. But because this measure counts for
only 10 per cent of the total score, it is not possible
for an institution to do well in the overall table on
this measure without being excellent in other
categories.
There are many measures we do not attempt to capture
in these pages. We gather data on universities that
teach undergraduates only. This eliminates many
high-quality specialist institutions such as Rockefeller
University and the University of California, San
Francisco, both of which are postgraduate medical
institutions.
We have considered a wide range of other criteria,
such as graduate employment and entry standards, as
possible quality measures. But these have all failed the
test of being applicable evenly around the world. For
example, a university in a particular country could show
poor graduate employment figures because of the state of
its national economy, not because it provided a bad
education.
Likewise, universities are under pressure to produce
spin-off companies and other forms of knowledge
transfer. But their success in doing so will depend to a
large extent on the economic system in which they are
embedded. In the same way, it is impossible to devise a
universal measure for entry standards. However, we are
always interested in readers’ suggestions for new
measures we could consider applying.
We regret that there are no data on Royal Holloway,
University of London. We plan to include the institution
in the rankings for 2007.
Acknowledgments
The
World University Rankings are co-ordinated by Martin
Ince (martin@martinince.com),
contributing editor of The Times Higher. He
would like to thank Nunzio Quacquarelli and Ben Sowter
of QS (http://www.qsnetwork.com/),
Jonathan Adams of Evidence Ltd (http://www.evidence.co.uk/)
and their colleagues, as well as the staff of The
Times Higher, for their participation in this
project.
| 
