IndustryCorpus_automobile

Dataset Overview

Dataset Description

• Language: Chinese and English
• Data Size: 1 TB Chinese data, 2.4 TB English data
• Task Category: Text Generation
• Industry Categories: 18 industry categories, including Medical, Education, Literature, Finance, Tourism, Law, Sports, Automotive, News, etc.

Data Processing

• Data Sources: Filtered from over 100 TB of open‑source corpora, including WuDaoCorpora, BAAI‑CCI, redpajama, SkyPile‑150B
• Data Processing Operations: Applied 22 industry‑specific operators for cleaning and filtering
• Rule‑based Filtering: Traditional Chinese conversion, email removal, IP address removal, link removal, Unicode repair, etc.
• Model‑based Filtering: Utilized industry classification language models with an accuracy of 80 %
• Deduplication: Document‑level deduplication using MinHash

Data Annotation

• Chinese Data Labels: 12 label types such as alphanumeric ratio, average line length, language confidence score, maximum line length, perplexity, toxic character ratio, etc.

Dataset Performance Validation

• Model Training: Conducted continual pre‑training, SFT, and DPO training
• Performance Gains: Target performance increased by 20 %, subjective win rate 82 %

Industry Data Size

Dataset Split

• Splitting Method: The large dataset is divided into 18 industry‑specific subsets; the current description refers to the automotive subset.