TheCenter The Top American Research Universities

Deconstructing University RankingsMedicine and Engineering, and Single CampusResearch CompetitivenessFinding Number OneMajor research universities continually monitortheir performance relative to other highly competitiveinstitutions, seeking to understand their own positionwithin the competition for the faculty, students, andresources that produce institutional quality and, byextension, prestige. We have participated in this effortat TheCenter for a number of years, producing tableson research university performance that rely on thebest nationally available data. Many other organizations,mostly commercial, make various comparisonsthat attempt to rank institutional performance, butmost of these efforts are highly controversial and oftenof dubious methodological reliability. The most suspectcomparisons attempt to assign a single numberthat represents university quality in some way andthen rank all institutions by this single number. Eventhough the errors in this process are many and anextensive literature exists critiquing the process, singlevaluerankings remain popular for many reasons.The principal virtue of these spurious rankings isthat they purport to declare a “winner” in the competitionfor institutional quality in a format familiarto followers of college sports teams or the magazinerankings of corporations, hospitals, restaurants, movies,towns, and every other facet of American life. Therankings of colleges and universities offer the possibilityfor an annual, end-of-the-competitive-seasonscore that heralds the season’s winner of the academictournament. Although such notions become popularamong various constituencies, the basic conceptis foolish because university quality rarely changesdramatically from year to year, and the differences inthe performance of similar institutions are not onlytoo small to measure but essentially meaningless. Asa result, institutions interested in understanding theirrelative position in the competition for quality look toother data such as the tables in TheCenter’s Top AmericanResearch Universities (TARU) to provide them witha context for assessing their own performance overtime.Measuring Market ShareAll efforts to categorize academic institutions havelimits defined by the characteristics of the data usedand the methodology that constructs the tables. Intelligentuse of any comparative university data requiresan understanding of the purposes of the tables, anda clear sense of the questions the methodology andthe categories derived from it can reliably answer.TheCenter’s data and methodology speak to a specificset of questions and provide answers within a specificcontext. Although we often make this point invarious conversations, it bears repeating here. TARUdoes not attempt to measure something that might becalled total university quality. It does not display thebest undergraduate or graduate program. It does notfind the highest-quality average faculty performance.These might well be useful outcomes, but the datacurrently available do not yield good answers to suchquestions, even though many commercial organizationsattempt to provide answers using flawed dataand unreliable methodology.TARU asks a different set of questions. We havediscussed this issue before, but the topic warrants continuedattention. TARU is essentially a market sharestudy. It begins with the observation that the pool ofhighly productive research talent is scarce relative tothe number of higher education institutions competingfor that talent. It continues with the observationthat universities compete with each other to acquireas much of this talent as possible. TARU then looksat the national indicators that reflect institutional successin capturing shares of this scarce talent pool. Asa result, in measuring federal research expenditures,for example, TARU reflects that portion of the totalresearch dollars spent in any year attributable to theresearch work on each campus. This, in turn, is areflection of the number of the campus’ high-qualityresearch people and their effectiveness in the competitionfor grants from the fixed pool of federal researchsupport.The Top American Research Universities 2005Page