Culture and Education

Understanding University Rankings

Yasushi Matsunaga
Professor, Center for Research Strategy, Waseda University

Internationalization of the Education Market

Recently, global university rankings have become a topic of discussion in the media. University rankings provided by publishers and preparatory schools have existed in Japan but not on an international scale. Business magazines and investment journals have started to rank international companies and publish the results, and the education market is now following this trend. World university rankings could be considered as an indication of education becoming an international business.

Waseda University has declared to become an international research university, and because the mission of the Center for Research Strategy, where I belong, is to enhance research capabilities, we must inevitably pay attention to global university rankings. Of course, these rankings are just one of the ways to look at universities, and the purpose of research and education activities at the university are not solely for improving the ranks. However, rankings are a good source of information to understand the areas where Waseda excels and where it needs improvement in comparison to other research universities around the world. Rankings are also useful for undergraduate students and teaching staff in and outside Japan who are considering studying abroad for graduate school or seeking employment, as well as for universities to determine which overseas institutions to sign partnership agreements with.

・ARWU (Academic Ranking of World Universities): Started by Shanghai Jiao Tong University with the objective of comparing research capabilities of universities in China with universities around the world. The ranking uses an objective evaluation index to rank the research capabilities of universities with natural science departments.

・QS (Quacquarelli Symons): Uses an objective research result index and focuses on reputation and education environment plus an index for internationalization. There are many derived rankings such as regional, field, MBA and employment rankings. The ranking had been jointly operated with THE until 2009, and then became an independent operation in 2010.

・THE (Times Higher Education): First began in the special edition of the Times, and it is published by TES Global today. Indices are similar to QS with an additional index for industry-university collaboration. Joint operation with Thomson Reuters was dissolved in 2015 and it is now operating independently.

・CWUR (Center for World University Ranking): Disclosed by a Saudi Arabian organization. It is characterized by not using peer review and reputation indices but instead an objective data such as number of awards received by alumni and the number of CEOs to compare with other universities as evaluation indices.

・U.S. News: Ranking by U.S. News & World Report. Evaluation of universities in the U.S. expanded worldwide.

Elsevier: The world's largest scale science and technology publisher based in the Netherlands
Clarivate Analytics: A company that provides science and technology information, which spun off from Thomson Reuters

Number of universities is as of June 2017.

Major World University Rankings

The world university rankings shown in the chart above tend to place universities in English speaking nations, science and engineering-oriented universities, and universities with medical departments higher since they have an advantage in the number of papers and citations. Each ranking has different objectives and evaluation indices, but even if they shared the same objective, they all place heavier weight on different indices and have varying data collection methods, resulting in university rankings looking very differently from each other. Moreover, universities with graduate schools only in the humanities and social sciences do not appear at the top of the rankings, with the exception of universities in English-speaking nations. Liberal arts universities with a focus on education do not rank at the top either.

Evaluation indices used for university rankings whose accuracy can be guaranteed are ones that include information from publications counting the number of citations and number of international awards received. Rankings that use these numbers only are reliable. Criteria that cannot be assessed include education systems, features, social contributions, and prestige. To quantify these criteria, ranking companies collect votes from research and education staff, enterprises, and institutions worldwide to evaluate reputation (peer review).

There are other unmeasurable matters by peer review, including the university’s relationship with and its place in the national government local communities, properties such as libraries and research and academic facilities, campus atmosphere, and extracurricular activities. These aspects only become clear after actually enrolling in or being employed at the university, so they are considered inapplicable as factors in the scoring system comparing universities worldwide.

Status of Japanese Universities in World University Rankings

Japanese universities' rating for general CPF (Citations per Faculty) and internationalization is low. Because the bibliography database is of English-speaking nations, it is not easy to carry out an academic and social evaluation of humanities and social science papers and literature written in Japanese and languages other than English. According to the chart showing the number of universities ranked within the top 500, CWUR > QS > ARWU > News > THE is the order of ranking advantageous for Japanese universities. This order does not change for a number of universities ranked within the top 200. Comparing the number of universities ranked within the top 200 by the three leading companies that include peer review as indices, we see that there is a huge difference with 9 (QS) > 4 (News) > 2 (THE). These rankings have business in mind to win advertising income and consultation service fees, so it is easy to see that companies place heavier weight on different regions and nations. In fact, distribution of peer ranking by QS leans toward developing nations in Asia, Central and South America.

Limitations in University Rankings

Most of the world university rankings make evaluations of about 1,000 universities. This is to ensure reliability by drawing a lower limit line in the published information and poll numbers. Comprehensive world university ranking is a battle of approximately 1,000 universities that have fulfilled the entry qualifications and these universities are competing in different competitions.

There would not be any issue if postgraduate courses were all structured the same, but considering the differences in the actual organization of universities, comparing universities in comprehensive rankings require caution. Universities specializing only in the arts, humanities, social sciences and medicine cannot rank at the top. Taking this fact into account, QS provides information separating comprehensive universities with universities specializing in specific areas of study. In THE, Caltech (California Institute of Technology) has been the top university for several years. Caltech is hands down one of the world’s leading research universities, but it is difficult to understand the conclusions drawn from comparing comprehensive universities that offer an almost complete set of graduate programs such as Oxford University to universities specializing in engineering like Caltech, where the student body is one tenth of the comprehensive universities. This is the limitation to the current ranking methodology. One reason people concerned with Japanese universities are unsatisfied with the world university rankings is that these rankings firms publish their rankings without sufficiently explaining this limitation, and that the Japanese media follows this suit.

How to View University Rankings

So how should we view these world university rankings? To be frank, rankings that are most advantageous to university management should be leveraged. The three rankings by big name companies are for commercial purposes, so there is a problem with objectivity. Yet, having some kind of international standards for comparing universities is necessary because the globalization of the education business cannot be stopped. Though insolence in a way, understanding the features of each ranking and using one that is most convenient at the time would be best.

Referring to rankings by field is recommended for students considering studying abroad to heighten their specialized knowledge and skills. Polls for peer review of research and education will be highly reliable if they reflect their specialties. Evaluation by number of citations will increase accuracy with minimal correction. At such times, it would be good to compare several rankings yourself while considering evaluation indices. Although natural science universities at the top of the rankings weigh heavily in quantitative indices and distribution is relatively small, we need to recognize that humanities and social sciences have a large distribution with a high weighting of reputation. On the other hand, comparison by field will heighten reliance on evaluating research and education but requires caution because indices for internationalization and industry-university collaboration are excluded.

Yasushi Matsunaga
Professor, Center for Research Strategy, Waseda University

Professor Yasushi Matsunaga completed the Graduate School of Science and Engineering, Waseda University in 1990, and earned a Ph.D. in Science in 1997.
Prior to taking his current position in 2013, he served as a researcher for Toshiba Corporation, a research associate at Waseda University, a postdoctoral researcher at the Japan Society for the Promotion of Science and an assistant professor and an associate professor at Waseda University. He specializes in plasma science, nanostructure science, and research strategy/evaluation.
Professor Matsunaga has served as MEXT Science and Technology Policy Bureau’s Deliberation Committee member for the Status of Human Resources Involved in Research Development Evaluation and Research and Analysis on Personnel Development, primary screening and screening member for the Japan Society for the Promotion of Science Research Fund, and controller for the Research Administrator Association.