Rethinking Impact: China’s New Approach to Evaluating Scientific Research

For decades, the journal impact factor has served as the dominant metric for assessing the quality and influence of scientific research. However, growing criticism over its limitations — particularly its susceptibility to manipulation and failure to reflect true societal value — has prompted a global search for better alternatives. In China, this debate has taken on new urgency as the nation seeks to align its scientific evaluation system with broader goals of innovation and public benefit. Recent developments suggest a shift toward more nuanced, multidimensional approaches that prioritize real-world impact over journal prestige.

The Limitations of the Impact Factor

The impact factor, calculated annually by Clarivate Analytics, measures the average number of citations received by articles published in a journal over the preceding two years. While widely used in hiring, promotion, and funding decisions, it has long been criticized by scientists and policymakers alike. Critics argue that it encourages short-term, citation-chasing research rather than long-term, high-risk innovation. The metric can be skewed by a small number of highly cited papers and does not account for differences across disciplines.

As noted in a 2023 commentary published in Nature, reliance on the impact factor risks undermining the remarkably goals of scientific progress by incentivizing quantity over quality and favoring established journals over emerging or interdisciplinary function.

Nature: The problem with impact factors (2023)

China’s Search for a Better Metric

Recognizing these shortcomings, Chinese policymakers and research institutions have begun exploring alternatives that better reflect the nation’s strategic priorities in science and technology. In 2022, the Ministry of Science and Technology launched a pilot program to evaluate research based on contributions to technological innovation, economic development, and social welfare — rather than journal rankings alone.

This initiative builds on earlier reforms, including the 2019 directive from the State Council calling for a “classified evaluation” system that tailors assessment criteria to different types of research, such as basic science, applied technology, and social sciences.

Ministry of Science and Technology of the People’s Republic of China: Pilot Reform of Research Evaluation (2022)

Emerging Alternatives in Practice

Several Chinese universities and research institutes are now experimenting with hybrid evaluation models. For example, Tsinghua University has implemented a system that combines peer review, altmetrics (such as social media mentions and policy citations), and tangible outcomes like patents filed or technologies transferred to industry.

Similarly, the Chinese Academy of Sciences has begun using “contribution-based” assessments in certain divisions, where researchers are evaluated on their role in major national projects — such as lunar exploration or quantum computing — rather than publication counts.

These efforts mirror global trends. The San Francisco Declaration on Research Assessment (DORA), which has been signed by over 2,000 organizations worldwide, explicitly rejects the use of journal-based metrics like the impact factor in research evaluation. In Europe, initiatives such as the Netherlands’ Recognition and Rewards program are reshaping academic careers around teamwork, leadership, and societal impact.

San Francisco Declaration on Research Assessment (DORA)

VSNU: Recognition and Rewards in Dutch Academia

Challenges and the Road Ahead

Despite promising pilots, widespread adoption faces hurdles. Changing deeply entrenched evaluation practices requires not only new metrics but also cultural shifts within academia. There are concerns about subjectivity in assessing societal impact, the administrative burden of collecting diverse data types, and ensuring fairness across disciplines.

Nonetheless, experts agree that the status quo is unsustainable. As China aims to grow a global leader in science and innovation by 2035, aligning research evaluation with national development goals is seen as both necessary and strategic.

“We are moving from asking ‘Where was it published?’ to ‘What did it change?’” said one senior science policy advisor involved in the reform discussions, speaking on condition of anonymity.

Conclusion

The impact factor’s dominance in scientific evaluation is waning, both globally and within China. While no single replacement has yet emerged, the shift toward more comprehensive, context-sensitive assessments reflects a growing consensus: science should be judged not just by where it appears, but by what it achieves. For China, this evolution is not merely academic — it is integral to its vision of becoming an innovation-driven economy. As pilot programs expand and lessons are shared, the future of research evaluation may well be defined not by impact factors, but by real-world impact.

Key Takeaways

• The journal impact factor has significant limitations, including bias toward certain disciplines and encouragement of short-term, citation-focused research.
• China is actively reforming its research evaluation system to prioritize innovation, technological transfer, and societal benefit over journal prestige.
• Pilot programs at institutions like Tsinghua University and the Chinese Academy of Sciences incorporate altmetrics, patents, and contributions to national projects.
• Global initiatives such as DORA and the Netherlands’ Recognition and Rewards program provide models for moving beyond journal-based metrics.
• Challenges remain, including ensuring objectivity, managing administrative complexity, and shifting academic culture — but the momentum for change is growing.

Frequently Asked Questions

What is wrong with using the impact factor to evaluate research?

The impact factor measures only average journal citations and does not reflect the quality, originality, or societal value of individual papers. It can be manipulated and unfairly disadvantages researchers in slower-publishing or interdisciplinary fields.

Is China abandoning the impact factor entirely?

Not yet. While the impact factor is still used in many contexts, China is piloting alternative systems that reduce its weight and supplement it with metrics tied to innovation and real-world impact.

What are some alternatives being tested in China?

Alternatives include contribution-based assessments, altmetrics (e.g., policy citations, social media engagement), patent activity, technology transfer outcomes, and peer review focused on societal relevance.

How does this compare to efforts in other countries?

Similar reforms are underway globally. The San Francisco Declaration on Research Assessment (DORA) has gained broad support, and countries like the Netherlands and the UK are implementing national frameworks that reward collaboration, open science, and public engagement.

Will these changes affect early-career researchers?

Yes — and potentially for the better. By valuing diverse contributions beyond first-author papers in high-impact journals, new systems may create fairer pathways for researchers pursuing applied, interdisciplinary, or public-interest work.