cs.CLSep 19, 2026
SaveChronologic: Measuring Language Models' Ability to Represent the Past
Organizations: University of Illinois Urbana-Champaign · University of British Columbia · University of Missouri · Cornell University
Abstract
Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its responses fit the period represented. Validation is challenging, because this is not a task living people ordinarily perform, and because many questions have multiple correct answers. We use historical texts to develop a benchmark for a model's representation of English-language contexts 1831-1930, relying on pairwise comparisons to multiple ground truths and strong distractors to score the hardest questions in an appropriately graduated way. We find that generative tasks are harder than discriminative ones; in fact, reasoning models can typically discern the weakness of their own generated answers. While models pretrained exclusively on historical text lead the pack when evaluated by answer likelihood, they cannot compete with commercial models in free generation. None of the models we tested represent historical contexts in a fully persuasive way yet, but progress toward that goal is evident.
Figures & tables
| Reasoning type | % | |
| Cloze tasks | ||
| structured cloze – sentence | 160 | 18.5 |
| structured cloze – phrase | 155 | 17.9 |
| topic sentence | 44 | 5.1 |
| Generation tasks | ||
| constrained generation | 158 | 18.2 |
Table 1: Distribution of reasoning types in Chronologic-EN-1.0.
Table 2: An example of the relationship between metadata frame and main question in a “book context” question. This sample distractor comes from Report of the Indian Education Commission (Hunter Commission), issued by the Government of India under British colonial rule, 1882. Other distractors came from 21c sources.
Figure 1: Scoring paths for a freely-generated answer. Substantive scoring uses either binary pass/fail evaluation (for 520 questions) or Bradley–Terry partial-credit scoring (for 346); the two paths produce a common substantive score, which can be aggregated by question type. Style evaluation applies both authenticity and date-fidelity measures to all answers of sufficient length.
Figure 2: Initial generative success rates are on the x axis; discriminative accuracy in rejecting the model’s own answers in MCQ format is on the y axis.
| Model | Brier | Cloze | Gen. | Know. |
| Talkie 1930 13B base | 0.1574 | 39.5 | 24.9 | 44.0 |
| Talkie 1930 13B it | 0.1593 | 37.4 | 23.8 | 25.1 |
| Qwen 2.5 7B finetune | 0.1601 | 36.2 | 14.9 | 34.4 |
| Qwen 2.5 72B | 0.1607 | 10.8 | 8.8 | 38.0 |
| Qwen 3.5 35B-A3B base | 0.1623 | 12.6 | 8.0 | 37.1 |
| Talkie web 13B base | 0.1627 | 18.2 | 13.6 | 34.1 |
Table 3: Open-weight models scored by per-token likelihoods assigned to answers. Overall Brier score (lower is better) and per-category accuracy (%, higher is better) on cloze, constrained generation, and knowledge/inference questions.
| Model | Effort | Cloze | Generation | Knowledge | Date fidelity | Error in years | Human authenticity |
|---|---|---|---|---|---|---|---|
| Talkie 1930 13B base | 1.5 | 12.6 | 20.3 | 54.4 | 30.2 | 32.6 | |
| Talkie 1930 13B it | 6.3 | 32.9 | 22.6 | 87.7 | 12.6 | 32.1 | |
| Qwen2.5 7B | 7.3 | 13.2 | 25.5 | 4.8 | 105.2 | 3.7 | |
| Qwen2.5 72B it | 19.1 | 20.2 | 50.2 | 13.8 | 84.2 | 5.4 | |
| Gemini 3.7 flash | high | 85.8 | 60.9 | 98.2 | 76.7 | 14.9 | 27.4 |
| Kimi K3 | max | 85.7 | 66.6 | 94.4 | 94.8 | 6.6 | 24.0 |
Table 4: Free-generation results (higher is better for all columns except error in years, a signed quantity where lower absolute magnitude is better). For scoring method, see Figure 1 ; substantive score is broken down here by the reasoning type of questions. For confidence intervals, see Appendix D . For fine-tuning of GPT-4.1, Appendix G .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Observed directional agreement between six human readers and our Bradley-Terry scoring model. Gaps between the two answers are displayed here on the [0,1] scale of , although the logistic model is actually fit to , an unbounded quantity.
| Distractor class | Likelihood | Free generation |
| Partially valid | 0.64 | 0.69 |
| Negation | 0.17 | 0.44 |
| LLM-generated | 0.55 | 0.35 |
| Same character | 0.23 | 0.33 |
| Same book | 0.28 | 0.24 |
| Hand-written | 0.39 | 0.24 |
Table 5: Rate at which each class of distractor defeats the model. Rates are conditional on the class being present in the question, and weighted equally per question. Distractors the evaluated model itself generated are excluded.
| Item | Answer source | Prob. | Answer | Reject reason | |
| gt1 | ground truth | 1.00 | +4.33 | The rifled gun, from its great accuracy and penetrating power, and more especially for its power of throwing a loaded shell with an accuracy and range beyond what have heretofore been considered to belong to solid-shot guns, is a very important gun to sea-coast batteries. Its most important use, however, will be as an incendiary and explosive agent against wooden vessels at long … | — accepted |
| gt0 | ground truth | 1.00 | +1.40 | In reference to rifle guns, it is doubtless true that they are destined to play an important part hereafter; that they possess qualities of range and accuracy which must make them, where these … | — accepted |
| d3 | claude-opus-5 | 0.50 | +1.34 | It is not to be concluded from what has been said above of the range and precision of the rifled gun that the spherical projectile has served its turn and may be laid aside; for the problem which the channels of our harbors impose upon the engineer is not that of striking a distant object with nicety, but of disabling, at ranges of a few hundred yards, a vessel cased in four inches and a half of wrought iron laid upon a solid backing of … | is actually quite close to the perspective of an American engineer in 1861, and gives itself away only, perhaps, in its condensation: it reads suspiciously like someone has understood the whole argument and reverse-engineered an ideal paragraph expressing Barnard’s dialectical perspective on rifled and smooth-bore artillery |
| d1 | gemini-3.6-flash | 0.00 | -1.61 | While the superior accuracy, range, and armor-penetrating power of elongated projectiles make rifled artillery destined to eventually supersede smooth-bore ordnance, heavy smooth-bores such as the 15-inch Rodman gun still serve a vital temporary role in coast defense. Their immense caliber … | knows too much about the future, and in particular is too confident that rifled guns will eventually triumph; that, combined with the relatively modern syntax, suggests that this passage may have been composed by a 21c LLM not an 1861 military engineer |
| d2 | 1873 text | 0.00 | -2.33 | In batteries liable to attack by iron-plated ships, the heaviest rifled guns, ML. 7 to 12-in., should be mounted; but against wooden vessels the BL. R. 7-in. and 64-pr., the ML. R. 80 and 64-prs., … | uses British terminology, referring to guns by the weight of the projectiles they throw, and also seems to be written later than 1861; it can already take for granted the existence of very large rifled artillery pieces whose viability was still doubtful in 1861 |
| d0 | 1864 text | 0.00 | -3.12 | The experiments were concluded by firing a few rounds from the 110-pr. at the field splinter proof which had hitherto sheltered the range party during the practice: it was struck three times in six … | is clearly written from the context of practical siege warfare and field engineering, where the problem is breaching earthen fortifications, rather than from Barnard’s seacoast-defense context, where the central problem is stopping ships and armored vessels |
Table 6: Answer options for question 888 (Based on Notes on Sea-coast Defence, J. G. Barnard, 1861). is the posterior mean latent strength from the frozen Bradley–Terry anchor fit, so rows are ordered by the judge’s own ranking; probability is the benchmark’s answer probability, where marks a distractor human editors judged partly valid.
| Model | Effort | Cloze | Generation | Knowledge | Date fidelity |
| Talkie 1930 13B base | 1.5 [0.8, 3.3] | 12.6 [11.1, 20.6] | 20.3 [15.5, 25.3] | 54.4 [50.0, 59.3] | |
| Talkie 1930 13B it | 6.3 [4.8, 9.5] | 32.9 [29.6, 41.8] | 22.6 [17.5, 27.9] | 87.7 [82.4, 92.4] | |
| Qwen2.5 7B | 7.3 [5.4, 10.7] | 13.2 [12.3, 22.4] | 25.5 [19.7, 31.0] | 4.8 [3.5, 6.2] | |
| Qwen2.5 72B it | 19.1 [15.6, 24.0] | 20.2 [18.3, 29.9] | 50.2 [43.6, 56.6] | 13.8 [11.4, 16.5] | |
| Gemini 3.7 flash | high | 85.8 [82.0, 88.9] | 60.9 [54.9, 66.4] | 98.2 [96.6, 99.6] | 76.7 [71.1, 82.8] |
| Kimi K3 | max | 85.7 [81.9, 88.8] | 66.6 [59.9, 71.2] | 94.4 [91.5, 97.1] | 94.8 [89.1, 98.7] |
Table 7: 95% confidence intervals for the point estimates in Table 4 , on the same rows and in the same order. All four measures run 0–100, higher is better. Intervals are percentile intervals from the bootstraps described above; because the point estimate is a plug-in estimate and the interval a percentile of bootstrap replicates, the two are not concentric on the partial-credit channel (the Generation column), and the date-fidelity interval is one-sided where it reaches 100.0.
| Candidate answer | Label | Diagnosis by Claude Opus 5 |
| The sleeves of the new walking-dress are made moderately close at the shoulder, widening toward the elbow, and finished at the wrist with a narrow frill of lace, which gives a light and graceful effect. | (a) | Language model. The tell isn’t any single anachronism; it’s the absence of nouns a real fashion writer couldn’t have avoided. Three clauses about a sleeve and not one fabric, not one trade name, not one named sleeve shape. Period copy names things—coat sleeve, demi-long, bishop, pagoda—because the reader is being told what to ask for. This sentence instead describes the silhouette from scratch, which is what you do when you don’t have the vocabulary and are working from a mental image. And the shape it describes is incoherent: narrow at the shoulder, widening toward the elbow, then somehow a wrist frill—that’s a pagoda sleeve grafted onto a cuffed one, i.e., the average of two decades rather than either. The closer, “which gives a light and graceful effect,” is the giveaway you’d expect: an evaluative summary clause with no referent, appended because the model has learned that period prose sounds appreciative. Real columns do say the effect is graceful, but they say it about a named thing. |
| His shirt, which looked as if it never could have been whole and white, had more than half the sleeves torn away, and fell open in front for want of a collar, to say nothing of a button and buttonhole. | (b) | I’m keeping, though it’s the one I’d argue about. It’s fiction, not fashion copy, so the genre heterogeneity is jarring—but ladies’ magazines of the 1870s were mostly fiction by page count, and reprinting English serial matter was standard. It has the Dickensian comic apposition (“to say nothing of a button and buttonhole”), so if it’s an impostor it’s an 1850s–60s British novel rather than a model output. What holds me is “looked as if it never could have been whole and white”—the counterfactual pushed back past the garment’s own history is a strange enough construction that I don’t think it’s synthetic. Models write “looked as though it had never been white.” That extra modal is a human doing something odd on purpose. |
| Dress of blue pique, made with a plain gored skirt, scalloped on the bottom, and trimmed with white linen braid and pearl buttons, and a plain square-necked waist without sleeves, worn over a guimpe of white nainsook, finished at the neck and wrists with Valenciennes lace. | (c) | Is the anchor. Pique, gored skirt, linen braid, guimpe over a sleeveless square-necked bodice, nainsook, Valenciennes—this is a misses’ or child’s summer dress plate, mid-1870s, and American. “Waist” for bodice is the decisive nationality marker; a British book would say bodice or corsage. The description is also usefully boring: it enumerates and stops, with no aesthetic verdict at the end. |
| Pelisse of cotton-blossom-yellow gros de Naples, trimmed round the border, and on each side of the front, with vandykes in satin, and fastened down with mother-of-pearl buttons: plain sleeves, moderately full, and a pelerine cape, with an antique collar of pointed lace falling over. | (d) | Real, but roughly 1825–1835, and British. Four independent markers, all pointing the same way: pelisse as a current outer-garment name, gros de Naples as a current silk, vandykes as a trimming, pelerine with an antique collar. By the 1870s the pelisse is a costume-history word, gros de Naples has been displaced by gros grain and faille, and the vandyke/pelerine combination is grandmother’s wardrobe. “Cotton-blossom-yellow” belongs to the Ackermann’s / La Belle Assemblée register of invented poetic colorways. The syntax is right—the verbless nominal opening, the colon splicing the second half—which is exactly why it’s a good distractor: it fails on lexical dating, not on style. |
Table 8: Responses to the assignment: “Two authentic sentences referencing ‘sleeves’ from a ladies’ home magazine published in the United States in the 1870s, and two impostors, either generated by language models or taken from books in other periods. Identify the impostors, and explain your rationale.” The model’s diagnoses are all correct.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,408 papers