Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

Mario Sanz-Guerrero; Minh Duc Bui; Manuel Mager; Katharina von der Wense

cs.CLSep 29, 2026

Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

Authors: Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, Katharina von der Wense

Organizations: Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA

Abstract

Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spanning multiple-choice QA (MCQA), math reasoning, code generation, and machine translation, performance varies solely with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation. Model rankings also shift, affecting leaderboards. This date effect exceeds other sources of non-determinism, such as batch size and numerical precision. Standard prompting techniques -- chain-of-thought and few-shot prompting -- do not reduce the sensitivity; chain-of-thought even amplifies it. Our findings underscore the need for careful evaluation protocols to ensure reproducibility and fair comparisons in LLM research.

Figures & tables

Panel of Figure 1 — Figure 1: The performance of LLMs on standard benchmarks varies solely by changing the current date, which is usually hidden from users in the system prompt.

Model	MMLU	GPQA	ARC-C	Avg.
Llama 3.1 (8B)	$2.11$	$4.53$	$2.01$	$2.88$
Llama 3.1 (70B)	$2.46$	$3.54$	$0.67$	$2.22$
Gemma 3 (4B)	$1.75$	$3.03$	$1.87$	$2.22$
Gemma 3 (27B)	$1.38$	$4.04$	$1.01$	$2.14$
Qwen3 (4B)	$2.46$	$2.53$	$1.34$	$2.11$
Qwen3-Next (80B)	$2.11$	$2.43$	$1.01$	$1.85$

Table 1: Difference in accuracy (delta) from the worst to the best date in 2024 across models and datasets.

Figure 2: Accuracy (top) and ECE (bottom) across different dates in 2024 for the top-5 models on MMLU.

	GSM8K	HumanEval	WMT (en $\to$ de)		WMT (en $\to$ fi)		WMT (en $\to$ cs)
Model	$\Delta$ Acc.	$\Delta$ pass@1	$\Delta$ BLEU	$\Delta$ chrF	$\Delta$ BLEU	$\Delta$ chrF	$\Delta$ BLEU	$\Delta$ chrF
Llama 3.1 (8B)	$9.85$	$7.32$	$1.66$	$1.13$	$1.79$	$1.52$	$1.82$	$1.42$
Llama 3.1 (70B)	$7.58$	$5.49$	$1.58$	$1.07$	$1.71$	$1.46$	$1.76$	$1.36$
Gemma 3 (4B)	$8.33$	$6.71$	$1.57$	$1.15$	$1.64$	$1.11$	$1.68$	$1.22$
Gemma 3 (27B)	$3.03$	$3.66$	$1.58$	$0.96$	$1.38$	$0.91$	$1.90$	$1.06$
Qwen3 (4B)	$6.06$	$2.44$	$1.34$	$0.95$	$1.39$	$1.38$	$1.88$	$1.30$

Table 2: Worst-to-best date delta in 2024 across models on open-ended generation tasks: accuracy on GSM8K, pass@1 on HumanEval, and BLEU and chrF on 3 WMT language pairs (English to German, Finnish, and Czech).

Pattern	N	Median	Range	Same dir.
Trend over the year	27	0.02	[ $-$ 0.40, 0.53]	–
Across datasets	27	$-$ 0.03	[ $-$ 0.16, 0.15]	51%
Across models	108	0.01	[ $-$ 0.17, 0.25]	51%

Table 3: Correlation between the date and accuracy (trend over the year; Spearman), and between the accuracies across dates of the same model on two datasets or of two models on the same dataset (Pearson), on MCQA. N : number of model–dataset combinations (trend) or compared pairs. Same dir. : share of dates where both accuracies are above or both below their yearly average (50% = chance).

Figure 3: Accuracy deviation across different dates for GPT-5.1. The 0.0 line represents the average accuracy over the week for each dataset.

Figure 4: Comparison of accuracy across different dates in 2024 for Llama 3.1 (8B) on MMLU using zero-shot prompting vs. chain-of-thought prompting.

Model	MMLU	GPQA	ARC-C	Avg.
Llama 3.1 (8B)	$2.81$	$5.05$	$2.35$	$3.40$
Llama 3.1 (70B)	$1.40$	$5.56$	$1.01$	$2.66$
Gemma 3 (4B)	$2.46$	$3.54$	$1.34$	$2.45$
Gemma 3 (27B)	$0.70$	$1.52$	$0.67$	$0.96$
Qwen3 (4B)	$1.75$	$3.03$	$1.43$	$2.07$
Qwen3-Next (80B)	$1.75$	$3.54$	$1.34$	$2.21$

Table 4: Difference in accuracy (delta) from the worst to the best date in 2024 across models and datasets using 5-shot prompting.

Source of Variation	Acc.	ECE
Current Date [Jan 1 – Dec 31]	$0.78$ %	$2.56$ %
Batch Size [1, 2, 4, 8, …, 128]	$0.61$ %	$2.00$ %
Num. Precision [BF16, FP16, FP32]	$0.52$ %	$2.54$ %
GPU Model [A100, A40, RTX4090]	$0.48$ %	$1.60$ %
Option Order [5 random permutations]	$0.75$ %	$3.12$ %
System Instruction [6 different wordings]	$0.78$ %	$3.54$ %

Table 5: Coefficient of variation (CV) of accuracy and ECE from different sources of non-determinism using Llama 3.1 (8B) on MMLU.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Response of Qwen3 called through OpenRouter, with an empty system prompt, no tool use, and no internet access. The model reports the current date, showing that the provider injects it server-side.

Panel of Figure 6 — Figure 6: Prompt used for multiple-choice questions. We extract the probabilities of the tokens after the arrow ( $\rightarrow$ ). “ X ” denotes the option label ( A / B / C / D ).

Panel of Figure 7 — Figure 7: Prompt used for chain-of-thought MCQA. The model generates a reasoning chain ending in “ The answer is X. ” (where X = A / B / C / D ); we parse the final option label X from this string.

Panel of Figure 8 — Figure 8: Prompt used for GSM8K. The model generates a step-by-step solution ending with “ Answer: {number} ”, from which we parse the final number.

Panel of Figure 9 — Figure 9: Prompt used for HumanEval. The model completes the given Python function, and we run it against the unit tests to compute pass@1.

Panel of Figure 10 — Figure 10: Prompt used for machine translation. {target_language} is German, Finnish, or Czech, depending on the language pair.

Dataset	# Questions	% Time-Dependent
MMLU	$14\,042$	$0.0$ %
GPQA	$198$	$0.0$ %
ARC-Challenge	$1172$	$0.0$ %
GSM8K	$1319$	$0.0$ %
HumanEval	$164$	$0.0$ %
WMT en $\to$ de	$2999$	$0.0$ %

Appendix

Table 6: Number and percentage of time-dependent questions in each dataset, as determined by GPT-OSS (120B).

Model	Consistent	Inconsistent
Llama 3.1 (8B)	$83.71$ %	$41.85$ %
Llama 3.1 (70B)	$92.13$ %	$47.59$ %
Gemma 3 (4B)	$99.49$ %	$88.86$ %
Gemma 3 (27B)	$99.92$ %	$80.48$ %
Qwen3 (4B)	$98.96$ %	$66.69$ %
Qwen3-Next (80B)	$98.82$ %	$64.38$ %

Appendix

Table 7: Average confidence (token probability of the selected answer) for consistent and inconsistent predictions across all dates in 2024 on MMLU.

Model	MMLU	GPQA	ARC-C	Avg.
Llama 3.1 (8B)	$2.38$	$4.17$	$1.95$	$2.83$
Llama 3.1 (70B)	$3.35$	$4.61$	$1.24$	$3.07$
Gemma 3 (4B)	$2.05$	$2.98$	$1.78$	$2.27$
Gemma 3 (27B)	$1.49$	$3.23$	$0.79$	$1.84$
Qwen3 (4B)	$2.41$	$2.93$	$1.52$	$2.29$
Qwen3-Next (80B)	$2.47$	$3.39$	$0.75$	$2.20$

Appendix

Table 8: Difference in ECE (delta) from the worst to the best date in 2024 across models and datasets.

	MMLU		GPQA		ARC-C
Model	Acc.	ECE	Acc.	ECE	Acc.	ECE
Llama 3.1 (8B)	$67.0$	$14.5$	$29.3$	$28.4$	$81.9$	$9.4$
Llama 3.1 (70B)	$79.4$	$11.2$	$40.1$	$33.6$	$92.6$	$4.8$
Gemma 3 (4B)	$54.4$	$44.4$	$29.3$	$67.1$	$78.6$	$21.1$
Gemma 3 (27B)	$75.0$	$24.2$	$33.8$	$62.1$	$93.9$	$5.7$
Qwen3 (4B)	$73.7$	$24.3$	$45.0$	$46.1$	$89.0$	$10.5$

Appendix

Table 9: Accuracy and ECE for all models and datasets when the current date is not injected in the system prompt.

Figure 11: Accuracy and ECE across different dates in 2024 for all models on MMLU.

Figure 12: Accuracy and ECE across different dates in 2024 for all models on GPQA.

Figure 13: Accuracy and ECE across different dates in 2024 for all models on ARC-Challenge.

Panel of Figure 14 — Figure 14: Impact of different user locations in the system prompt on Llama 3.1 (8B) performance on MMLU.

Explore similar work

How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
Sep 24, 2026Dipankar SarkarLarge Language Model Evaluation Reproducibility
Skelf Research
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
May 19, 2026David Pape, Jonathan Evertz, Lea SchönherrLarge Language Model Benchmarks LLM Inference Optimization
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading
Apr 30, 2026Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2Large Language Model Evaluation Prompt Engineering
SAP · Stanford University

Sep 24, 2026, cs.CLJ/K move · Enter open · S save

How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

Dipankar Sarkar

Skelf Research

Large Language Model Evaluation Reproducibility Prompt Engineering Model Auditing Virtual Cell Model-Based Bootstrap

Open paper AskPDF arXiv

Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much confidence such a table deserves, using LLM-based prompt-structure inference as the case study: eight open model variants across five families and 8B to 675B parameters, caching disabled, 293 raw intermediate representations persisted. The measured phenomenon is unstable to begin with. Identical calls do not reliably recover identical structure, with mean node-set Jaccard from 0.39 to 0.96 and 72% of prompt-model cells never node-set-perfect. Auditing the evaluation weakens its conclusions further, and this is our main contribution. Under a joint cluster bootstrap over prompts, only the bottom of the ranking is firm: the two least reproducible models hold rank in 99% and 86% of replicates, the middle four in 27% to 48%, and the top two in 68% each, so the table identifies the worst model reliably but does not reliably identify the best. Two equally defensible rules for merging repeated campaigns change four of eight rows and move the study-wide headline by 7 percentage points. Checking the inferred structure against ground-truth annotations shows reproducibility cannot be read as accuracy. And four of the eight endpoints were withdrawn within ten weeks of measurement, so the study as specified can no longer be run. Small-sample LLM evaluations can therefore look far more definitive than their evidence supports. We recommend reporting rank stability, per-cell provenance, executed sensitivity comparisons, raw per-run outputs, and a measurement date alongside any ranking.

Abstract

Explore similar work

How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility

Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading