Large Language Models (LLMs) can exhibit factual inaccuracies during long-chain reasoning or under misleading prompts, limiting their reliability in historical analysis. We propose Kongzi, a language model for historical understanding and reasoning that combines continued pre-training on historical corpora, supervised fine-tuning, and fact-enhanced reinforcement learning. Its factuality reward compares entities and relations extracted from generated responses with historical references through knowledge-graph similarity. We evaluate 0.5B and 7B models on historical question answering and counterfactual reasoning, using reported ratings from three LLM judges, and on CMMLU-Chinese-History. Kongzi-7B receives the highest historical-accuracy ratings among the compared models under all three judges. Reinforcement learning improves the mean answer scores of both model sizes and their CMMLU-Chinese-History accuracy, although Kongzi-7B remains below the strongest reported baselines on that benchmark. These descriptive results support further study of factuality rewards for historical language models.
Figures & tables
Figure 1: Comparative analysis of different LLM’s responses to misleading historical fact-checking queries.
Figure 2: Overview of our pipeline. In terms of data processing, we clean the original corpus and separate it into factual data and inference text, where factual data undergoes augmentation while inference text is chunked for generating high-quality CoT data. For model training, we first conduct CPT with basic data, followed by a two-stage SFT to develop both fundamental QA capabilities and CoT reasoning abilities. Finally, Content RL is employed to optimize output quality through reinforcement learning.
Figure 3: The HFRM module constructs a knowledge graph by extracting entities and relations from the model outputs, which serves as the foundation for information representation and computation. The semantic similarity between this graph and the ground truth is used as the reward signal.
Figure 4: Composite scores recomputed from the reported aggregate data. For models with all required components, the score is 0.8A+0.1T+0.1W , where A is the arithmetic mean of the three displayed answer-dimension scores, T is Think Score, and W is the reported ratio favoring the model over Deepseek-r1 on the 0–100 scale. Means are used before rounding. Deepseek-r1 and QwQ are omitted from this composite plot because their reported Ratio and Think Score, respectively, are unavailable; their available component scores remain in Table 3 . These derived composites do not represent a new evaluation run.
Model
Think Score
Mean Answer Score
Historical Accuracy
Logical Reasoning
Problem Solving
> Deepseek-r1 Ratio
GPT-4o
Kongzi-0.5B-SFT
50.42
45.68
33.87
56.59
46.59
52.18%
Kongzi-7B-SFT
60.46
64.27
50.97
76.96
64.88
85.29%
Kongzi-0.5B-RL
60.42 (+10.0)
56.39 (+10.7)
53.34 (+19.5)
62.41 (+5.8)
53.43 (+6.8)
62.28 % (+10.1)
Kongzi-7B-RL
72.34 (+11.9)
70.64 (+6.4)
58.45 (+7.5)
82.13 (+5.2)
71.34 (+6.5)
94.32 % (+9.0)
Gemini-2.5
Table 1: Improvement of Kongzi model performance in various metrics by RL training. Mean Answer Score and its changes are recomputed from the displayed Historical Accuracy, Logical Reasoning, and Problem Solving scores. Subscripts give RL-minus-SFT changes; ratio changes are in percentage points. Bold identifies RL rows. Other scores and ratios are retained as reported.
Base Model Table
Category
Model
w/o
Cot
Dense Model
Qwen2.5-72B
91.02%
—
Dense Model
Qwen2.5-7B few-shot
87.31%
69.88%
Dense Model
Qwen2.5-0.5B few-shot
35.29%
29.41%
Mixed Reasoning Model Table
Category
Model
w/o
Thinking
Table 2: Results on CMMLU-Chinese-History. Best scores in each metric are highlighted in bold. ‘w/o‘ refers to accuracy without enhancement prompts (e.g., Cot or Thinking). ‘—‘ indicates the model was not tested or data is unavailable under the corresponding condition.
Model
Think Score
Mean Answer Score
Historical Accuracy
Logical Reasoning
Problem Solving
> Deepseek-r1 Ratio
GPT-4o
Deepseek-r1-distill-7B
45.66
37.82
24.29
49.54
39.63
24.49%
Deepseek-r1-distill-14B
62.41
60.86
43.92
74.65
64.01
68.48%
Deepseek-r1-distill-32B
60.13
60.83
44.70
74.88
62.90
70.07%
Deepseek-r1
67.81
59.94
46.31
70.14
63.36
–
O3-mini
32.77
61.05
45.90
74.06
63.18
53.00%
Table 3: Comparison of Kongzi with other models on various metrics. Think Score is the rating given by the LLM for the thinking process. Mean Answer Score is recomputed as the arithmetic mean of the displayed Historical Accuracy, Logical Reasoning, and Problem Solving scores, then rounded to two decimals. It is a derived summary, not an additional evaluation run. > Deepseek-r1 Ratio retains the reported percentage of cases favoring the model over Deepseek-r1; per-item win/tie counts and aggregation details are unavailable in the current materials. The best scores in each metric are in bold .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
LR
Steps / Epochs
Batch Size
Seq Length
1. Continue Pretrain Stage
Kongzi-0.5B
2e-5
1500
128
8192
Kongzi-7B
5e-5
750
256
8192
Optimizer
AdamW
Hardware
NVIDIA A6000
2. SFT Stage
Kongzi-0.5B
2e-5
4000
128
4096
Appendix
Table 4: Training configuration across different stages.
Figure 5: Spatial-temporal relationship comprehension test, the results of Kongzi.
Figure 6: Spatial-temporal relationship comprehension test, the results of Deepseek.
Figure 7: Spatial-temporal relationship comprehension test, the results of QWQ.
Figure 8: Historical Fact Information Test, the results of Kongzi.
Figure 9: Historical Fact Information Test, the results of Deepseek.
Figure 10: Historical Fact Information Test, the results of QWQ.
Figure 11: Open-ended Situational Reasoning Q&A, the results of Kongzi.
Figure 12: Open-ended Situational Reasoning Q&A, the results of Kongzi.
Figure 13: Open-ended Situational Reasoning Q&A, the results of Deepseek.
Figure 14: Open-ended Situational Reasoning Q&A, the results of QWQ.
While Large Language Models (LLMs) have increasingly assisted in historical tasks such as text processing, their capacity for professional-level historical reasoning remains underexplored. Existing benchmarks primarily assess basic knowledge breadth or lexical understanding, failing to capture the higher-order skills, such as evidentiary reasoning,that are central to historical research. To fill this gap, we introduce ProHist-Bench, a novel benchmark anchored in the Chinese Imperial Examination (Keju) system, a comprehensive microcosm of East Asian political, social, and intellectual history spanning over 1,300 years. Developed through deep interdisciplinary collaboration, ProHist-Bench features 400 challenging, expert-curated questions across eight dynasties, accompanied by 10,891 fine-grained evaluation rubrics. Through a rigorous evaluation of 18 LLMs, we reveal a significant proficiency gap: even state-of-the-art LLMs struggle with complex historical research questions. We hope ProHist-Bench will facilitate the development of domain-specific reasoning LLMs, advance computational historical research, and further uncover the untapped potential of LLMs. We release ProHist-Bench at https://github.com/inclusionAI/ABench/tree/main/ProHist-Bench.
Lirong Gao, Zeqing Wang, Yuyan Cai +6
State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group · Zhejiang University
Aggregate factuality scores hide where a language model succeeds, which relations it confuses, and whether an answer survives innocuous changes to the question or decoder. We introduce PROOF, a profile-oriented benchmark for factual coverage in instruction-tuned language models. PROOF converts a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts, 101 classes, 392 properties, and 14 domains. Each question has an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations; 1,849 questions are no-correct-option traps. We evaluate 18 open-weight model deployments on 166,374 prompts each and separately perturb decoding on a fixed 10% subset. Base factual accuracy ranges from 6.58% to 57.59% (chance: 8.64%), yet every model has a 19.3-36.4 percentage-point spread across domains. Paired facts reveal direction-dependent retrieval, usually favoring subject-to-object queries, with the pattern reversing for one model. We find no consistent temporal penalty after exact-stratum adjustment. Neutral wording changes accuracy by as much as 26.5 percentage points, while adversarial formulations break up to 79.4% of answers that were initially correct. Direct switching to an injected false label varies from 0.04% to 27.5%, showing that accuracy loss and hint following are distinct. Selected-token confidence often indicates severe overconfidence, and decoder perturbations move accuracy by up to 15.7 percentage points and domain profiles by 16.8 points. PROOF therefore measures factual coverage as a structured, intervention-aware profile rather than a single claim about what a model "believes."
Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha +4
We introduce TypewriterLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913. Developing History LMs requires addressing challenges in data quality and availability, preventing temporal leakage, designing temporally consistent post-training pipelines, and constructing reliable evaluations. To address these issues, we construct TypewriterCorpus, a 54B-token historical corpus collected from diverse archival and linguistically annotated sources with extensive data cleaning and leakage mitigation procedures. Furthermore, we introduce lexically grounded instructing tuning, a post-training framework that constraints responses to remain directly grounded in historical source documents. Using this framework we construct two historical instruction tuning datasets: History-LIMA and History-SelfInstruct. To evaluate capability and temporal consistency, we introduce History-Event, a benchmark suite for evaluating competence, temporal grounding and data leakage. We release TypewriterLM and all associated resources to support future research on historical language models.