Lüshi-1840: Building and Probing a Knowledge Cutoff for a Classical Chinese Language Model

Alex Liu

cs.CLOct 12, 2026

Lüshi-1840: Building and Probing a Knowledge Cutoff for a Classical Chinese Language Model

Authors: Alex Liu

Organizations: Ybuild LLC

Abstract

Language models trained only on text from before a given year are useful for studying hindsight and historical language, but the open collections of pre-modern Chinese carry no publication dates and unreliable dynasty labels. We present Lüshi-1840, a 983M-parameter model trained from scratch on 12,262 books (1.28B characters) selected as composed before 1840. A first corpus, dated by a common heuristic, in effect ended at 1644 while containing twentieth-century material. We describe the re-dating and cleaning pipeline that replaced it and an audit with markers the pipeline did not use, which still finds later material. On the model side we measure surprisal, relative to a modern reference, on annals compiled after the cutoff. For events after 1840 against events up to 1820 the ratio is 1.38 (95% interval 1.33-1.44), against about 1.00 for modern general-purpose models. In a series of 294M models, adding the later court records lowers the ratio from 1.33 to 1.16, and the two models diverge from the 1820s, where the added records begin. The probe tracks the coverage of sources and does not establish a single cutoff year; none of our probes exposes the audited contamination. Model, book list and scripts will be released.

Figures & tables

Prompt: 順治、康熙、雍正、乾隆、嘉慶、道光、

(the Qing reign names in order, up to Daoguang, 1821–1850)

Lüshi-old: 咸豐、光緒、崇禎、同治、光緒…

Xianfeng (1851–61), Guangxu (1875–1908), …, Tongzhi (1862–74)

Lüshi-1840: 天啓、崇禎、皇清、雍正、乾隆…

two late-Ming reign names, then back to the early Qing

Prompt: 火輪車者， (“The fire-wheel carriage is”; a mid-nineteenth-century word for the train)

Figure 1: Greedy continuations from the model trained on the heuristically dated corpus (Lüshi-old) and on the re-dated corpus (Lüshi-1840); two of fourteen prompts, chosen for illustration. Both models were intended to have an 1840 cutoff.

Marker		Old	New
光緒	reign, 1875–1908	2,633	0
宣統	reign, 1909–1911	246	0
李鴻章	statesman, d. 1901	707	0
王先謙	scholar, d. 1917	288	0
王國維	scholar, d. 1927	135	0
梁啓超	writer, d. 1929	86	0

Table 1: Occurrences of late markers in the training file of Lüshi-old (1.58B tokens) and Lüshi-1840 (1.24B). These are the markers the cleaning rules use, so the New column shows that the rules were applied, not that the corpus is clean (§ 3.3 ). Names of persons are candidate indicators of late material, not proof of it. ∗ Also ordinary phrases (e.g. 諸味而同治); we inspected a sample of the remaining hits, not all. † Genuine residue: modern collation notes citing Yu Yue.

Decision	Books	M chars
Canon, kept whole	5,291	177
A: before 1840, kept	8,359	1,173
B: composed after 1840	164	27
C: modern critical edition	245	42
D: undeterminable	825	43
A, dropped by backstop rules	50	51

Table 2: Outcome of re-dating for the 14,934 candidate books.

Figure 2: Characters by period in the final corpus, by the collections’ dynasty labels (not revised inside the canons).

Figure 3: Bits per character relative to Qwen2.5-7B by the year of the events narrated (10-year bins; bands are 95% intervals from a bootstrap over years). The test text changes at 1644, marked by the first dotted line and the shading: Ming tongjian (printed 1873) before, annals of Qingshi gao (compiled 1914–27) after.

Lüshi-1840 divided by reference:
	$\leq$ 1820	$>$ 1840	post/pre [95% CI]
Qwen2.5-7B	0.958	1.326	1.384 [1.359, 1.411]
Qwen2.5-1.5B	0.831	1.158	1.394 [1.368, 1.420]
Gemma-4-E4B	0.838	1.156	1.380 [1.353, 1.407]
Yi-1.5-6B	1.030	1.428	1.386 [1.359, 1.412]
Yi-1.5-9B	1.057	1.457	1.378 [1.350, 1.406]

Table 3: Relative bits per character within the annals of Qingshi gao , for events of 1645–1820 and 1841–1911. Goldfish is scored on the unmapped text. GPT-2-DZG is the UER GPT-2 trained on Daizhige, which includes Qingshi gao .

Figure 4: The cutoff series: three 294M models differing in corpus. Top: relative to Qwen2.5-7B. Bottom: the to-1840 model relative to the extended model, with no modern reference involved. Bins and bands as in Figure 3 .

Persons	$n$	Ours	Old	Q-1.5B	Q-7B
died before 1800	1,172	.824	.709	.506	.734
born 1800–1839	86	.442	.337	.360	.581
born 1840 or later	184	.348	.364	.435	.663
drop, first to last		.476	.345	.071	.071

Table 4: Top-1 accuracy on four-way courtesy-name items (Q: Qwen2.5). Bootstrap 95% intervals over sets of four, first and last rows: Lüshi-1840 [.799, .849], [.288, .408]; Qwen2.5-7B [.708, .758], [.582, .739].

Prompt (ending)	Lüshi-old	Lüshi-1840
…嘉慶、道光、	1 / $-$ 1.99	203 / $-$ 7.77
…宣宗道光，文宗	77 / $-$ 6.64	337 / $-$ 10.49
…道光三十年，	5 / $-$ 3.16	357 / $-$ 9.51

Table 5: Rank and log-probability of 咸 (first character of the reign name Xianfeng, 1851–61) as the next character. Qwen2.5-1.5B ranks it first under all three prompts.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	MTJ	Qingshi gao annals
	1368–	1645–	1736–	1821–	1841–
Model	1644	1735	1820	1840	1911
Lüshi-1840	4.59	4.56	4.06	4.67	6.11
Lüshi-old	5.05	5.77	5.64	5.98	6.42
Qwen2.5-1.5B	5.62	5.28	5.06	5.04	5.27
Gemma-4-E4B	5.74	5.28	4.98	4.99	5.28

Appendix

Table 6: Bits per character by period of the events narrated ( MTJ : Ming tongjian ). Paragraphs per column: 3,317 / 1,010 / 925 / 227 / 863. † UER GPT-2 trained on Daizhige, which includes Qingshi gao .

Task	$n$	Ours	Old	Q-0.5B	Q-1.5B
Segmentation	500	69.2	52.6	92.8	93.8
Poetry quality	406	52.0	61.6	44.1	50.5
Couplets	500	38.6	32.2	38.0	42.4
Medical texts	211	33.2	33.6	24.2	29.4
Sinology basics	136	32.4	32.4	30.1	31.6
Poetry apprec.	103	31.1	25.2	16.5	15.5

Appendix

Table 7: ACLUE accuracy (%), zero-shot by length-normalised option log-likelihood. Q: Qwen2.5. Task names are our translations.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 521 papers

Pretraining Language Models on Historical Text
Jun 2, 2026Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber +5Language Model Pretraining LLM Evaluation
University of Waterloo · Vector Institute · AIML, Adelaide University +3
Chronologic: Measuring Language Models' Ability to Represent the Past
Sep 19, 2026Ted Underwood, Ziliang Qiu, Sarah Griebel +4LLM Evaluation Temporal Reasoning in Language Models
University of Illinois Urbana-Champaign · University of British Columbia · University of Missouri +1
Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model
Mar 31, 2026Jiuting Chen, Yuan Lian, Hao Wu +4Metacognition in Language Models Factual Knowledge in Language Models
Eaglewood Japan Co., Ltd.

Jun 2, 2026J/K move · Enter open · S save

Pretraining Language Models on Historical Text

Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber, Yixuan Wang+4

University of Waterloo · Vector Institute · AIML, Adelaide University+3

Language Model Pretraining LLM Evaluation LLM Grounding Language Modeling

Open paper AskPDF arXiv

We introduce TypewriterLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913. Developing History LMs requires addressing challenges in data quality and availability, preventing temporal leakage, designing temporally consistent post-training pipelines, and constructing reliable evaluations. To address these issues, we construct TypewriterCorpus, a 54B-token historical corpus collected from diverse archival and linguistically annotated sources with extensive data cleaning and leakage mitigation procedures. Furthermore, we introduce lexically grounded instructing tuning, a post-training framework that constraints responses to remain directly grounded in historical source documents. Using this framework we construct two historical instruction tuning datasets: History-LIMA and History-SelfInstruct. To evaluate capability and temporal consistency, we introduce History-Event, a benchmark suite for evaluating competence, temporal grounding and data leakage. We release TypewriterLM and all associated resources to support future research on historical language models.

Abstract

Language Modeling

Explore similar work

Pretraining Language Models on Historical Text

Chronologic: Measuring Language Models' Ability to Represent the Past

Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model