haonan-li/cmmlu

Dataset Overview

Dataset Name

• CMMLU

Description

• CMMLU is a comprehensive benchmark for evaluating large language models (LLMs) on advanced knowledge and reasoning abilities within Chinese language and cultural contexts. It spans 67 topics from elementary to advanced professional levels, covering STEM subjects such as physics and mathematics, as well as humanities and social sciences.

Features

• Multiple‑choice and QA tasks.
• Each question offers four options with only one correct answer.
• Many tasks contain context‑specific nuances and phrasing that are difficult to translate.
• Answers for many tasks are China‑specific and may not be suitable for other regions or languages.

Structure

• Provides development and test sets for each topic.
• Development set contains 5 questions per topic; test set contains over 100 questions per topic.

Usage

• The dataset can be loaded via Python, supporting per‑topic loading or loading all data at once.

License

• The dataset follows the Creative Commons Attribution‑NonCommercial‑ShareAlike 4.0 International License.

Citation

@misc{li2023cmmlu,
      title={CMMLU: Measuring massive multitask language understanding in Chinese}, 
      author={Haonan Li and Yixuan Zhang and Fajri Koto and Yifei Yang and Hai Zhao and Yeyun Gong and Nan Duan and Timothy Baldwin},
      year={2023},
      eprint={2306.09212},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}