The WoW: A Large-Scale World Knowledge Corpus for Full-Lifecycle LLM Training
Organizations: Harbin Institute of Technology · OpenCSG
Abstract
Existing Chinese educational corpora often suffer from fragmented coverage and a lack of systematic organization. In this paper, we propose WoW (World Knowledge), a large-scale dataset designed for the full-cycle training of foundation models. The construction of this dataset is driven by an Educational Knowledge Graph, which serves as a taxonomic backbone for the systematic collection and completion of missing educational modules across all levels. Compared to previous versions, this iteration provides enhanced systematicity, superior quality, and a larger scale, while incorporating multimodal signals and supporting every stage of model development. Experiments demonstrate that WoW significantly improves model performance across diverse benchmarks. To facilitate community research, we further provide an interactive platform for graph-based data selection and retrieval.
Figures & tables
| Discipline | Subject | Type | Format | Size | Source (KG-driven) |
|---|---|---|---|---|---|
| Computer Sci. | Software; Applications | Web article | JSONL | 45GB / 119.8K | CSDN technical community |
| Computer Sci. | Software; Applications | Curated notes | Markdown | 1.6GB / 6.5K | Disciplinary Knowledge Repository (CS) ⋆ |
| Computer Sci. | Software; Architecture | Book/Textbook | PDF MD+Img | 1.5TB / 8.1K | High-Quality Structured Documents (CS) |
| Computer Sci. | Software; Applications | Book | 19GB / 4.1K | Publishing House of Electronics Industry | |
| Computer Sci. | Software; Applications | Book | 3.5GB / 0.2K | Posts & Telecom Press | |
| Engineering | Mechanical; Electrical & Electronics | Book | 196GB / 1.7K | China Machine Press |
| Domain | KPs | Covered KPs | Instances |
|---|---|---|---|
| Computer Science & Technology | 3,941 | 3,492 (88.6%) | 135,139 (71.6%) |
| Law & Institutional Studies | 10,137 | 2,882 (28.4%) | 14,088 (7.5%) |
| Natural Sciences & Technology | 14,305 | 4,126 (28.8%) | 10,036 (5.3%) |
| Humanities & Social Sciences | 11,523 | 3,636 (31.6%) | 9,858 (5.2%) |
| Engineering & Technology | 7,162 | 2,320 (32.4%) | 8,230 (4.4%) |
| Social Edu. & Literacy | 14,409 | 3,031 (21.0%) | 6,298 (3.3%) |
| Broad-coverage exams | Science | Control | ||||||||
| Model | MMLU | CMMLU | C-Eval | ChemBench | ScienceQA | SciEval | MoleculeQA | MedMCQA | Mean | |
| Llama-3.2-3B | Base | 56.5 | 44.3 | 44.3 | 49.2 | 76.8 | 53.4 | 27.4 | 50.4 | 50.3 |
| + WoW | 55.2 | 43.2 | 41.7 | 50.3 | 78.9 | 58.8 | 28.8 | 48.1 | 50.6 | |
| Llama-3.1-8B | Base | 65.3 | 52.9 | 52.5 | 53.9 | 82.7 | 64.6 | 29.3 | 58.0 | 57.4 |
| + WoW | 65.0 | 52.2 | 52.7 | 54.2 | 82.9 | 65.1 | 31.0 | 57.4 | 57.6 | |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Fine-tuning | full-parameter SFT |
|---|---|
| Epochs | 1 ( 1,430 steps) |
| Effective batch size | 128 sequences |
| Per-device batch accumulation | (8B: ) |
| Max sequence length | 4,096 tokens |
| Optimizer | AdamW ( , ) |
| Peak learning rate |
| Discipline | MMLU | CMMLU | C-Eval |
|---|---|---|---|
| Mathematics | 1,064 (7.6%) | 711 (6.1%) | 90 (6.7%) |
| Physics | 640 (4.6%) | 422 (3.6%) | 57 (4.2%) |
| Chemistry | 303 (2.2%) | 132 (1.1%) | 63 (4.7%) |
| Biology | 855 (6.1%) | 662 (5.7%) | 62 (4.6%) |
| Computer science | 557 (4.0%) | 669 (5.8%) | 133 (9.9%) |
| Law | 1,763 (12.6%) | 1,129 (9.7%) | 47 (3.5%) |
| Benchmark | Items | Composition |
|---|---|---|
| ChemBench | 2,147 | Single-answer items. Chemical preference 1,000 (pairwise choice between candidate molecules, hence two options), organic 384, toxicity and safety 341, analytical 99, general 98, physical 83, materials science 57, inorganic 49, technical chemistry 36. |
| ScienceQA | 2,224 | Text-only test items (K–12). Natural science 1,043 (biology 397, units and measurement 207, physics 192, chemistry 119, earth science 65), language science 1,056 (writing strategies, figurative language, reference skills, punctuation, grammar), social science 125 (mainly economics and civics). 2–4 options. |
| SciEval | 10,704 | Multiple-choice items of the test set. Chemistry 5,796, biology 2,288, and physics 1,478 (SocraticQA), and medicine 1,142 (MedQA). |
| MoleculeQA | 5,786 | Questions about a molecule given as SMILES, covering its structure, source, properties, and applications; four options. |
| MedMCQA | 4,183 | Validation split of Indian medical entrance exams: dental 1,318, surgery 369, pathology 337, medicine 295, pharmacology 243, anatomy 234, pediatrics 234, and 14 further clinical and basic-science subjects. Our sources contain almost no medicine. |
| Relation Type | Source Target | Semantic Meaning | Level |
|---|---|---|---|
| HAS_SUBJECT | Domain Subject | A domain contains a subject | Mandatory |
| HAS_KP | Subject KnowledgePoint | A subject contains a knowledge point | Mandatory |
| PRECURSOR_TO | KnowledgePoint KnowledgePoint | A knowledge point is a prerequisite for another | Mandatory |
| SUB_KP_OF | KnowledgePoint KnowledgePoint | A knowledge point is a subpoint of another | Mandatory |
| RELATED_TO | KnowledgePoint KnowledgePoint | Two knowledge points are laterally related | Recommended |
| SAME_AS | KnowledgePoint KnowledgePoint | Two knowledge points are equivalent or aliases | Recommended |
| Domain | Subjects | Nodes ( ) | Edges ( ) | Chunks ( ) |
| Formal Disciplinary Education (Narrow Sense) | ||||
| Computer Science & Technology | Theoretical Computer Science | 102 | 534 | 3,166 |
| Computer Architecture | 420 | 2,120 | 12,203 | |
| Computer Software | 1,605 | 7,149 | 66,923 | |
| Computer Networks | 835 | 3,811 | 12,791 | |
| Artificial Intelligence | 114 | 539 | 7,683 | |
LLM Training
Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.
See all 377 papers