C4 Dataset Overview

C4 is a large-scale, unlabeled text corpus constructed by the China Information Processing Laboratory and collaborators. It serves as a primary resource for pre‑training massive language models.

Dataset Contents

• Scale: Approximately 400 million cleaned text passages.
• Sources: Diverse high‑quality unstructured text resources.
• Cleaning Process: Heuristic rules filter well‑structured, valuable content; additional instruction generation and response rewriting improve quality.

Primary Use Cases

• Instruction Tuning: Enhances zero‑shot and few‑shot performance of large language models.
• Research: Supports various NLP tasks requiring extensive textual knowledge.

Annotation Quality

• Expert Involvement: Annotations and quality checks are performed by domain experts to ensure reliability.

Limitations & Risks

• Potential Biases: Inherent biases from source documents may be present; users should be aware when applying the data.

Citation

@article{c4_dataset,
  title   = {C4: A Large-Scale Chinese Unlabeled Text Corpus},
  author  = {China Information Processing Laboratory et al.},
  year    = {2023},
  note    = {Dataset release}
}

Contact

For questions, please contact the China Information Processing Laboratory.