Chronologic: Measuring Language Models' Ability to Represent the Past
Organizations: University of Illinois Urbana-Champaign · University of British Columbia · University of Missouri · Cornell University
Abstract
Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its responses fit the period represented. Validation is challenging, because this is not a task living people ordinarily perform, and because many questions have multiple correct answers. We use historical texts to develop a benchmark for a model's representation of English-language contexts 1831-1930, relying on pairwise comparisons to multiple ground truths and strong distractors to score the hardest questions in an appropriately graduated way. We find that generative tasks are harder than discriminative ones; in fact, reasoning models can typically discern the weakness of their own generated answers. While models pretrained exclusively on historical text lead the pack when evaluated by answer likelihood, they cannot compete with commercial models in free generation. None of the models we tested represent historical contexts in a fully persuasive way yet, but progress toward that goal is evident.
Figures & tables
| Reasoning type | % | |
| Cloze tasks | ||
| structured cloze – sentence | 160 | 18.5 |
| structured cloze – phrase | 155 | 17.9 |
| topic sentence | 44 | 5.1 |
| Generation tasks | ||
| constrained generation | 158 | 18.2 |
| Model | Brier | Cloze | Gen. | Know. |
| Talkie 1930 13B base | 0.1574 | 39.5 | 24.9 | 44.0 |
| Talkie 1930 13B it | 0.1593 | 37.4 | 23.8 | 25.1 |
| Qwen 2.5 7B finetune | 0.1601 | 36.2 | 14.9 | 34.4 |
| Qwen 2.5 72B | 0.1607 | 10.8 | 8.8 | 38.0 |
| Qwen 3.5 35B-A3B base | 0.1623 | 12.6 | 8.0 | 37.1 |
| Talkie web 13B base | 0.1627 | 18.2 | 13.6 | 34.1 |
| Model | Effort | Cloze | Generation | Knowledge | Date fidelity | Error in years | Human authenticity |
|---|---|---|---|---|---|---|---|
| Talkie 1930 13B base | 1.5 | 12.6 | 20.3 | 54.4 | 30.2 | 32.6 | |
| Talkie 1930 13B it | 6.3 | 32.9 | 22.6 | 87.7 | 12.6 | 32.1 | |
| Qwen2.5 7B | 7.3 | 13.2 | 25.5 | 4.8 | 105.2 | 3.7 | |
| Qwen2.5 72B it | 19.1 | 20.2 | 50.2 | 13.8 | 84.2 | 5.4 | |
| Gemini 3.7 flash | high | 85.8 | 60.9 | 98.2 | 76.7 | 14.9 | 27.4 |
| Kimi K3 | max | 85.7 | 66.6 | 94.4 | 94.8 | 6.6 | 24.0 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Distractor class | Likelihood | Free generation |
| Partially valid | 0.64 | 0.69 |
| Negation | 0.17 | 0.44 |
| LLM-generated | 0.55 | 0.35 |
| Same character | 0.23 | 0.33 |
| Same book | 0.28 | 0.24 |
| Hand-written | 0.39 | 0.24 |
| Item | Answer source | Prob. | Answer | Reject reason | |
| gt1 | ground truth | 1.00 | +4.33 | The rifled gun, from its great accuracy and penetrating power, and more especially for its power of throwing a loaded shell with an accuracy and range beyond what have heretofore been considered to belong to solid-shot guns, is a very important gun to sea-coast batteries. Its most important use, however, will be as an incendiary and explosive agent against wooden vessels at long … | — accepted |
| gt0 | ground truth | 1.00 | +1.40 | In reference to rifle guns, it is doubtless true that they are destined to play an important part hereafter; that they possess qualities of range and accuracy which must make them, where these … | — accepted |
| d3 | claude-opus-5 | 0.50 | +1.34 | It is not to be concluded from what has been said above of the range and precision of the rifled gun that the spherical projectile has served its turn and may be laid aside; for the problem which the channels of our harbors impose upon the engineer is not that of striking a distant object with nicety, but of disabling, at ranges of a few hundred yards, a vessel cased in four inches and a half of wrought iron laid upon a solid backing of … | is actually quite close to the perspective of an American engineer in 1861, and gives itself away only, perhaps, in its condensation: it reads suspiciously like someone has understood the whole argument and reverse-engineered an ideal paragraph expressing Barnard’s dialectical perspective on rifled and smooth-bore artillery |
| d1 | gemini-3.6-flash | 0.00 | -1.61 | While the superior accuracy, range, and armor-penetrating power of elongated projectiles make rifled artillery destined to eventually supersede smooth-bore ordnance, heavy smooth-bores such as the 15-inch Rodman gun still serve a vital temporary role in coast defense. Their immense caliber … | knows too much about the future, and in particular is too confident that rifled guns will eventually triumph; that, combined with the relatively modern syntax, suggests that this passage may have been composed by a 21c LLM not an 1861 military engineer |
| d2 | 1873 text | 0.00 | -2.33 | In batteries liable to attack by iron-plated ships, the heaviest rifled guns, ML. 7 to 12-in., should be mounted; but against wooden vessels the BL. R. 7-in. and 64-pr., the ML. R. 80 and 64-prs., … | uses British terminology, referring to guns by the weight of the projectiles they throw, and also seems to be written later than 1861; it can already take for granted the existence of very large rifled artillery pieces whose viability was still doubtful in 1861 |
| d0 | 1864 text | 0.00 | -3.12 | The experiments were concluded by firing a few rounds from the 110-pr. at the field splinter proof which had hitherto sheltered the range party during the practice: it was struck three times in six … | is clearly written from the context of practical siege warfare and field engineering, where the problem is breaching earthen fortifications, rather than from Barnard’s seacoast-defense context, where the central problem is stopping ships and armored vessels |
| Model | Effort | Cloze | Generation | Knowledge | Date fidelity |
| Talkie 1930 13B base | 1.5 [0.8, 3.3] | 12.6 [11.1, 20.6] | 20.3 [15.5, 25.3] | 54.4 [50.0, 59.3] | |
| Talkie 1930 13B it | 6.3 [4.8, 9.5] | 32.9 [29.6, 41.8] | 22.6 [17.5, 27.9] | 87.7 [82.4, 92.4] | |
| Qwen2.5 7B | 7.3 [5.4, 10.7] | 13.2 [12.3, 22.4] | 25.5 [19.7, 31.0] | 4.8 [3.5, 6.2] | |
| Qwen2.5 72B it | 19.1 [15.6, 24.0] | 20.2 [18.3, 29.9] | 50.2 [43.6, 56.6] | 13.8 [11.4, 16.5] | |
| Gemini 3.7 flash | high | 85.8 [82.0, 88.9] | 60.9 [54.9, 66.4] | 98.2 [96.6, 99.6] | 76.7 [71.1, 82.8] |
| Kimi K3 | max | 85.7 [81.9, 88.8] | 66.6 [59.9, 71.2] | 94.4 [91.5, 97.1] | 94.8 [89.1, 98.7] |
| Candidate answer | Label | Diagnosis by Claude Opus 5 |
| The sleeves of the new walking-dress are made moderately close at the shoulder, widening toward the elbow, and finished at the wrist with a narrow frill of lace, which gives a light and graceful effect. | (a) | Language model. The tell isn’t any single anachronism; it’s the absence of nouns a real fashion writer couldn’t have avoided. Three clauses about a sleeve and not one fabric, not one trade name, not one named sleeve shape. Period copy names things—coat sleeve, demi-long, bishop, pagoda—because the reader is being told what to ask for. This sentence instead describes the silhouette from scratch, which is what you do when you don’t have the vocabulary and are working from a mental image. And the shape it describes is incoherent: narrow at the shoulder, widening toward the elbow, then somehow a wrist frill—that’s a pagoda sleeve grafted onto a cuffed one, i.e., the average of two decades rather than either. The closer, “which gives a light and graceful effect,” is the giveaway you’d expect: an evaluative summary clause with no referent, appended because the model has learned that period prose sounds appreciative. Real columns do say the effect is graceful, but they say it about a named thing. |
| His shirt, which looked as if it never could have been whole and white, had more than half the sleeves torn away, and fell open in front for want of a collar, to say nothing of a button and buttonhole. | (b) | I’m keeping, though it’s the one I’d argue about. It’s fiction, not fashion copy, so the genre heterogeneity is jarring—but ladies’ magazines of the 1870s were mostly fiction by page count, and reprinting English serial matter was standard. It has the Dickensian comic apposition (“to say nothing of a button and buttonhole”), so if it’s an impostor it’s an 1850s–60s British novel rather than a model output. What holds me is “looked as if it never could have been whole and white”—the counterfactual pushed back past the garment’s own history is a strange enough construction that I don’t think it’s synthetic. Models write “looked as though it had never been white.” That extra modal is a human doing something odd on purpose. |
| Dress of blue pique, made with a plain gored skirt, scalloped on the bottom, and trimmed with white linen braid and pearl buttons, and a plain square-necked waist without sleeves, worn over a guimpe of white nainsook, finished at the neck and wrists with Valenciennes lace. | (c) | Is the anchor. Pique, gored skirt, linen braid, guimpe over a sleeveless square-necked bodice, nainsook, Valenciennes—this is a misses’ or child’s summer dress plate, mid-1870s, and American. “Waist” for bodice is the decisive nationality marker; a British book would say bodice or corsage. The description is also usefully boring: it enumerates and stops, with no aesthetic verdict at the end. |
| Pelisse of cotton-blossom-yellow gros de Naples, trimmed round the border, and on each side of the front, with vandykes in satin, and fastened down with mother-of-pearl buttons: plain sleeves, moderately full, and a pelerine cape, with an antique collar of pointed lace falling over. | (d) | Real, but roughly 1825–1835, and British. Four independent markers, all pointing the same way: pelisse as a current outer-garment name, gros de Naples as a current silk, vandykes as a trimming, pelerine with an antique collar. By the 1870s the pelisse is a costume-history word, gros de Naples has been displaced by gros grain and faille, and the vandyke/pelerine combination is grandmother’s wardrobe. “Cotton-blossom-yellow” belongs to the Ackermann’s / La Belle Assemblée register of invented poetic colorways. The syntax is right—the verbless nominal opening, the colon splicing the second half—which is exactly why it’s a good distractor: it fails on lexical dating, not on style. |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,408 papers