Temporal Text Classification with Large Language Models

Nishat Raihan; Marcos Zampieri

cs.CLMar 11, 2026

Temporal Text Classification with Large Language Models

Authors: Nishat Raihan, Marcos Zampieri

Organizations: George Mason University, Fairfax, VA, USA

Abstract

Languages change over time. Computational models can be trained to recognize such changes enabling them to estimate the publication date of texts. Despite recent advancements in Large Language Models (LLMs), their performance on automatic dating of texts, also known as Temporal Text Classification (TTC), has not been explored. This study provides the first systematic evaluation of leading proprietary (Claude 3.5, GPT-4o, Gemini 1.5) and open-source (LLaMA 3.2, Gemma 2, Mistral, Nemotron 4) LLMs on TTC using three historical corpora, two in English and one in Portuguese. We test zero-shot and few-shot prompting, and fine-tuning settings. Our results indicate that proprietary models perform well, especially with few-shot prompting. They also indicate that fine-tuning substantially improves open-source models but that they still fail to match the performance delivered by proprietary LLMs.

Figures & tables

Dataset	Lang	Time Span	Size	Classes
Colonia	PT	16th–20th C.	>5M Tokens	5 (100 year)
CLMET	EN	1710–1920	34M Words	3 (70 year)
DTE	EN	1700–2010	Varies by Split	$\sim$ 4 Intervals*

Table 1: Overview of datasets. Size indicates approximate total corpus size. *Temporal classes for SemEval DTE depend on the specific sub-task configuration (e.g., 4 intervals like pre-1700, 18th C., 19th C., post-1900 were common).

Model	Type	Param.
Claude 3.5 Sonnet	Proprietary	-
GPT-4o	Proprietary	-
Gemini 1.5 Pro	Proprietary	-
LLaMA 3.1	Open-Source	8B
Gemma 2	Open-Source	9B
Mistral	Open-Source	7B

Table 2: List of LLMs evaluated.

Dataset	SVM	Length	Zero-Shot/Few-Shot (5-Shot)
Dataset	SVM	Length	Claude	GPT4o	Gemini	LLaMA	Gemma	Mistral	Nemotron
Colonia	.51	10	.35/.45	.50/ .61	.39/.50	.12/.11	.18/.25	.17/.24	.16/.23
	.65	50	.72 / .88	.68 / .72	.59/.64	.20/.35	.28/.38	.27/.37	.25/.34
	.69	100	.68/ .91	.71 / .96	.77 / .82	.26/.22	.36/.51	.34/.49	.33/.45
CLMET	.70	10	.48/.58	.52/.48	.41/.40	.14/.18	.17/.26	.18/.23	.16/.25
	.77	50	.60/.62	.73/ .85	.63/.67	.21/.27	.26/.39	.27/.34	.23/.37

Table 3: Combined Zero-Shot and Few-Shot (5-Shot) F1 score comparison for seven models on Temporal Text Classification tasks. For the SVM column, TF-IDF vectors are used and the best results are reported.

Model	Colonia	CLMET	DTE	$\Delta$
LLaMA 3.2	0.61	0.65	0.66	+0.29
Gemma 2	0.73	0.75	0.78	+0.23
Mistral	0.72	0.68	0.76	+0.24
Nemotron 4B	0.70	0.67	0.71	+0.23

Table 4: Macro-F 1 for finetuned models on authentic data.

\Delta

is the mean gain over the non-finetuned 5-shot setting across the three datasets (fixed token length of 100).

Dataset	LLaMA	$\Delta$	Gemma	$\Delta$	Mistral	$\Delta$	Nemotron	$\Delta$
Colonia	.21/.31/.70	$-.05/+.09/+.09$	.47/.46/.83	$+.11/-.05/+.10$	.29/.58/.67	$-.05/+.09/-.05$	.24/.52/.62	$-.09/+.07/-.08$
CLMET	.35/.29/.74	$+.07/-.06/+.09$	.25/.62/.70	$-.09/+.09/-.05$	.39/.35/.79	$+.04/-.11/+.11$	.38/.44/.60	$+.07/-.05/-.07$
DTE	.27/.57/.55	$-.05/+.08/-.11$	.42/.41/.84	$+.11/-.11/+.06$	.30/.59/.70	$+.04/+.11/-.06$	.18/.53/.83	$-.11/+.09/+.12$

Table 5: Observed Macro-F 1 scores ( Zero-Shot/Few-Shot(5)/Finetuned ) for open-source models on synthetic test datasets. Adjacent

\Delta

columns show the change from the original scores (same order).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Field	Content
Instruction	Classify the following text into one of the given temporal classes: {class_list}. Respond with only the class label.
Input (Colonia)	“O aggregado foi despedindo os escravos e a fazenda; vendeo tres molecoens e huma negrinha; desfez-se das melhores joias da mulher; comeu os alugueis de tres predios; empenhou a prata”
Output	1800s
Input (CLMET)	“The village lay in a hollow, which was, perhaps, one reason for its dullness; and the church, standing on higher ground, overlooked it with a certain air of superiority”
Output	1850-1920
Input (DTE)	“We have cabled the English house from which we get it and expect a reply to-morrow.”

Appendix

Table 6: Fine-tuning input format used for training open-source models via LoRA. Each training instance follows the instruction-input-output template. The {class_list} placeholder is replaced with the appropriate temporal classes for each dataset: 5 century-level classes for Colonia, 3 period classes for CLMET, and interval-based classes for DTE.

Text Classification

Text classification assigns text to categories such as topic, sentiment, intent, stance, or risk, supporting search, moderation, and decisions.

See all 224 papers

Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Sep 29, 2026Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1LLM Evaluation Language Model Generation Evaluation
Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA
Temporal Flattening in LLM-Generated Text: Comparing Human and LLM Writing Trajectories
Apr 13, 2026Zhanwei Cao, YeoJin Go, Yifan Hu +1Language Model Generation Evaluation AI-Generated Text Detection
Northeastern University, Seattle, WA, USA
Using LLMs to Detect LLM-Generated Texts: A Cross-Generation Analysis
Sep 28, 2026Haiyue Yuan, Jie Guo, Weidong Qiu +3LLM Evaluation AI-Generated Text Detection
Institute of Cyber Security for Society (iCSS) & School of Computing, University of Kent, UK · School of Cyber Science and Engineering, Shanghai Jiao Tong University, China · School of Computer Science, University of Birmingham, UK

Sep 29, 2026J/K move · Enter open · S save

Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, Katharina von der Wense

Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA

LLM Evaluation Language Model Generation Evaluation LLM Prompting LLM Reliability ML Reproducibility Benchmark Reproducibility

Open paper AskPDF arXiv

Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spanning multiple-choice QA (MCQA), math reasoning, code generation, and machine translation, performance varies solely with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation. Model rankings also shift, affecting leaderboards. This date effect exceeds other sources of non-determinism, such as batch size and numerical precision. Standard prompting techniques -- chain-of-thought and few-shot prompting -- do not reduce the sensitivity; chain-of-thought even amplifies it. Our findings underscore the need for careful evaluation protocols to ensure reproducibility and fair comparisons in LLM research.

Abstract

Text Classification

Explore similar work

Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

Temporal Flattening in LLM-Generated Text: Comparing Human and LLM Writing Trajectories

Using LLMs to Detect LLM-Generated Texts: A Cross-Generation Analysis