Organizations: School of Mechanics and Engineering Science, Peking University, Beijing, 100871, PRChina · AI for Science Institute, Beijing, 100080, China
To advance foundation Large Language Models (LLMs) for combustion science, this study presents the first end-to-end framework for developing domain-specialized models for the combustion community. The framework comprises an AI-ready multimodal knowledge base at the 3.5 billion-token scale, extracted from over 200,000 peer-reviewed articles, 8,000 theses and dissertations, and approximately 400,000 lines of combustion CFD code; a rigorous and largely automated evaluation benchmark (CombustionQA, 436 questions across eight subfields); and a three-stage knowledge-injection pathway that progresses from lightweight retrieval-augmented generation (RAG) to knowledge-graph-enhanced retrieval and continued pretraining. We first quantitatively validate Stage 1 (naive RAG) and find a hard ceiling: standard RAG accuracy peaks at 60%, far surpassing zero-shot performance (23%) yet well below the theoretical upper bound (87%). We further demonstrate that this stage's performance is severely constrained by context contamination. Consequently, building a domain foundation model requires structured knowledge graphs and continued pretraining (Stages 2 and 3).
Figures & tables
Fig. 1: Our proposed three-stage domain knowledge injection framework. It integrates (Top) a knowledge base, (Bottom) an evaluation benchmark, and (Middle) an evolution roadmap (Validated).
Fig. 2: Knowledge landscape of combustion science derived from the proposed knowledge base.
Fig. 3: The multi-stage construction process for CombustionQA (436 questions).
Scenario
Provides Context
Contains Answer
Contains Noise*
Zero-Shot
No
N/A
N/A
Theoretical Limit
Yes
Yes
No
Optimal RAG
Yes
Yes
Yes
Noise RAG
Yes
No
Yes
Table 1: Feature matrix of the four evaluation scenarios. *“Noise” is defined as high-quality distracting information irrelevant to the answer.
Fig. 4: Accuracy of the four core scenarios across different LLMs.
Fig. 5: Performance comparison under “Optimal RAG” scenario, categorized by whether the exact source segment was retrieved (Hit vs. Miss).
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University · IDEA · Zhongguancun Academy +5