Kongzi: A Historical Large Language Model with Fact Enhancement

Jiashu Yang; Ningning Wang; Yian Zhao; Chaoran Feng; Junjia Du; Hao Pang; Miao Yu; Xuxin Cheng

cs.CLApr 13, 2025

Kongzi: A Historical Large Language Model with Fact Enhancement

Authors: Jiashu Yang, Ningning Wang, Yian Zhao, Chaoran Feng, Junjia Du, Hao Pang, Miao Yu, Xuxin Cheng

Organizations: Dalian University of Technology · Peking University · Nanyang Technological University

Abstract

Large Language Models (LLMs) can exhibit factual inaccuracies during long-chain reasoning or under misleading prompts, limiting their reliability in historical analysis. We propose Kongzi, a language model for historical understanding and reasoning that combines continued pre-training on historical corpora, supervised fine-tuning, and fact-enhanced reinforcement learning. Its factuality reward compares entities and relations extracted from generated responses with historical references through knowledge-graph similarity. We evaluate 0.5B and 7B models on historical question answering and counterfactual reasoning, using reported ratings from three LLM judges, and on CMMLU-Chinese-History. Kongzi-7B receives the highest historical-accuracy ratings among the compared models under all three judges. Reinforcement learning improves the mean answer scores of both model sizes and their CMMLU-Chinese-History accuracy, although Kongzi-7B remains below the strongest reported baselines on that benchmark. These descriptive results support further study of factuality rewards for historical language models.

Figures & tables

Figure 1: Comparative analysis of different LLM’s responses to misleading historical fact-checking queries.

Figure 2: Overview of our pipeline. In terms of data processing, we clean the original corpus and separate it into factual data and inference text, where factual data undergoes augmentation while inference text is chunked for generating high-quality CoT data. For model training, we first conduct CPT with basic data, followed by a two-stage SFT to develop both fundamental QA capabilities and CoT reasoning abilities. Finally, Content RL is employed to optimize output quality through reinforcement learning.

Figure 3: The HFRM module constructs a knowledge graph by extracting entities and relations from the model outputs, which serves as the foundation for information representation and computation. The semantic similarity between this graph and the ground truth is used as the reward signal.

Figure 4: Composite scores recomputed from the reported aggregate data. For models with all required components, the score is

0.8A+0.1T+0.1W

, where

A

is the arithmetic mean of the three displayed answer-dimension scores,

T

is Think Score, and

W

is the reported ratio favoring the model over Deepseek-r1 on the 0–100 scale. Means are used before rounding. Deepseek-r1 and QwQ are omitted from this composite plot because their reported Ratio and Think Score, respectively, are unavailable; their available component scores remain in Table 3 . These derived composites do not represent a new evaluation run.

Model	Think Score	Mean Answer Score	Historical Accuracy	Logical Reasoning	Problem Solving	$>$ Deepseek-r1 Ratio
GPT-4o
Kongzi-0.5B-SFT	50.42	45.68	33.87	56.59	46.59	52.18%
Kongzi-7B-SFT	60.46	64.27	50.97	76.96	64.88	85.29%
Kongzi-0.5B-RL	60.42 (+10.0)	56.39 (+10.7)	53.34 (+19.5)	62.41 (+5.8)	53.43 (+6.8)	62.28 % (+10.1)
Kongzi-7B-RL	72.34 (+11.9)	70.64 (+6.4)	58.45 (+7.5)	82.13 (+5.2)	71.34 (+6.5)	94.32 % (+9.0)
Gemini-2.5

Table 1: Improvement of Kongzi model performance in various metrics by RL training. Mean Answer Score and its changes are recomputed from the displayed Historical Accuracy, Logical Reasoning, and Problem Solving scores. Subscripts give RL-minus-SFT changes; ratio changes are in percentage points. Bold identifies RL rows. Other scores and ratios are retained as reported.

Base Model Table
Category	Model	w/o	Cot
Dense Model	Qwen2.5-72B	91.02%	—
Dense Model	Qwen2.5-7B few-shot	87.31%	69.88%
Dense Model	Qwen2.5-0.5B few-shot	35.29%	29.41%
Mixed Reasoning Model Table
Category	Model	w/o	Thinking

Table 2: Results on CMMLU-Chinese-History. Best scores in each metric are highlighted in bold. ‘w/o‘ refers to accuracy without enhancement prompts (e.g., Cot or Thinking). ‘—‘ indicates the model was not tested or data is unavailable under the corresponding condition.

Model	Think Score	Mean Answer Score	Historical Accuracy	Logical Reasoning	Problem Solving	$>$ Deepseek-r1 Ratio
GPT-4o
Deepseek-r1-distill-7B	45.66	37.82	24.29	49.54	39.63	24.49%
Deepseek-r1-distill-14B	62.41	60.86	43.92	74.65	64.01	68.48%
Deepseek-r1-distill-32B	60.13	60.83	44.70	74.88	62.90	70.07%
Deepseek-r1	67.81	59.94	46.31	70.14	63.36	–
O3-mini	32.77	61.05	45.90	74.06	63.18	53.00%

Table 3: Comparison of Kongzi with other models on various metrics. Think Score is the rating given by the LLM for the thinking process. Mean Answer Score is recomputed as the arithmetic mean of the displayed Historical Accuracy, Logical Reasoning, and Problem Solving scores, then rounded to two decimals. It is a derived summary, not an additional evaluation run.

>

Deepseek-r1 Ratio retains the reported percentage of cases favoring the model over Deepseek-r1; per-item win/tie counts and aggregation details are unavailable in the current materials. The best scores in each metric are in bold .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	LR	Steps / Epochs	Batch Size	Seq Length
1. Continue Pretrain Stage
Kongzi-0.5B	2e-5	1500	128	8192
Kongzi-7B	5e-5	750	256	8192
Optimizer	AdamW		Hardware	NVIDIA A6000
2. SFT Stage
Kongzi-0.5B	2e-5	4000	128	4096

Appendix

Table 4: Training configuration across different stages.

Figure 5: Spatial-temporal relationship comprehension test, the results of Kongzi.

Figure 6: Spatial-temporal relationship comprehension test, the results of Deepseek.

Figure 7: Spatial-temporal relationship comprehension test, the results of QWQ.

Figure 8: Historical Fact Information Test, the results of Kongzi.

Figure 9: Historical Fact Information Test, the results of Deepseek.

Figure 10: Historical Fact Information Test, the results of QWQ.

Figure 11: Open-ended Situational Reasoning Q&A, the results of Kongzi.

Figure 12: Open-ended Situational Reasoning Q&A, the results of Kongzi.

Figure 13: Open-ended Situational Reasoning Q&A, the results of Deepseek.

Figure 14: Open-ended Situational Reasoning Q&A, the results of QWQ.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination
Apr 27, 2026Lirong Gao, Zeqing Wang, Yuyan Cai +6LLM Evaluation Question Answering
State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group · Zhejiang University
PROOF: Profiling Reliability of Object-Level Facts in Large Language Models
Aug 24, 2026Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha +4Factual Knowledge in Language Models LLM Evaluation
Pretraining Language Models on Historical Text
Jun 2, 2026Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber +5Language Model Pretraining LLM Evaluation
University of Waterloo · Vector Institute · AIML, Adelaide University +3

Apr 27, 2026J/K move · Enter open · S save

Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination

Lirong Gao, Zeqing Wang, Yuyan Cai, Jiayi Deng+5

State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group · Zhejiang University

LLM Evaluation Question Answering

Open paper AskPDF arXiv

While Large Language Models (LLMs) have increasingly assisted in historical tasks such as text processing, their capacity for professional-level historical reasoning remains underexplored. Existing benchmarks primarily assess basic knowledge breadth or lexical understanding, failing to capture the higher-order skills, such as evidentiary reasoning,that are central to historical research. To fill this gap, we introduce ProHist-Bench, a novel benchmark anchored in the Chinese Imperial Examination (Keju) system, a comprehensive microcosm of East Asian political, social, and intellectual history spanning over 1,300 years. Developed through deep interdisciplinary collaboration, ProHist-Bench features 400 challenging, expert-curated questions across eight dynasties, accompanied by 10,891 fine-grained evaluation rubrics. Through a rigorous evaluation of 18 LLMs, we reveal a significant proficiency gap: even state-of-the-art LLMs struggle with complex historical research questions. We hope ProHist-Bench will facilitate the development of domain-specific reasoning LLMs, advance computational historical research, and further uncover the untapped potential of LLMs. We release ProHist-Bench at https://github.com/inclusionAI/ABench/tree/main/ProHist-Bench.

Abstract

Language Modeling

Explore similar work

Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

Pretraining Language Models on Historical Text