DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining
Authors: Yutong Yan, Raphael Tang, Zhenyu Gao, Wenxi Jiang, Yao Lu
Organizations: Department of Finance, CUHK Business School, The Chinese University of Hong Kong · Centre for Artificial Intelligence, University College London
Large language models pretrained on internet-scale data risk lookahead bias in forecasting tasks, as they may have already seen the true outcome during training. To address this, we present DatedGPT, a family of twelve 1.3B-parameter language models trained from scratch on approximately 100 billion tokens each with strict annual data cutoffs spanning 2013 to 2024, together with DatedInstruct, an instruction dataset grounded in each year's documents to prevent leakage during post-training. The models are competitive with open models of similar scale, and perplexity-based probing confirms that each model's knowledge is bounded by its cutoff year. On stock return prediction over 61,000 firm-day news headlines, DatedGPT-instruct achieves an annualised Sharpe ratio of 3.20 under the lookahead-bias-free setup. Lookahead-biased models, whose training data covers the outcome period, add a lookahead premium of 26.4 b.p. per standard deviation, significant at the 1% level. The series thus enables direct analysis of lookahead bias in financial forecasting. We provide an interactive web demo that allows users to query and compare responses from models across different cutoff years, available at www.datedgpt.com.
Figures & tables
Figure 1: Web interface for DatedGPT . When asked “What is Claude Code?” (released February 2025), the DatedGPT-2024 model, trained exclusively on data available before 2024, is unaware of the product and hallucinates an incorrect answer.
Model (0-shot)
ARC-C
ARC-E
HellaSwag
PIQA
IFEval
MMLU
TruthfulQA
Avg.
Baseline Models
SmolLM-1.7B-Instruct
34.9
54.3
56.1
72.4
15.5
27.1
27.7
41.1
GPT2-XL
28.4
51.0
50.8
70.5
15.0
25.3
22.4
37.6
TinyLlama-1.1B
30.5
43.7
55.0
72.5
14.8
24.8
26.0
38.2
OPT-1.3B
27.8
51.3
53.7
70.9
17.6
25.2
23.8
38.6
Pythia-1B
27.1
49.0
47.1
69.3
16.8
23.1
23.6
36.6
Table 1: Zero-shot evaluation results on language understanding benchmarks. IFEval reports prompt-level strict accuracy. The DatedGPT-Instruct series models are identified by their training data cutoff year. We report four representative models here, and the full series performance is detailed in Appendix C . For each time-locked model, we use the latest available checkpoint of its family and report both the original instruction-tuned release and the base version tuned on DatedInstruct .
Opponent
Win
Tie
Loss
ChronoGPT-Instruct
81.1
0.62
18.3
PIT-Instruct
46.6
0.50
52.9
Table 2: Win, tie, and loss rates (%) of DatedGPT-Instruct against each time-locked family’s 2024 instruct version, following AlpacaEval 2.0.
Panel regression
Portfolio
LH-B
LH-B
Model
β^ (pp)
t
β^⋅SD (bp)
> LH-F
Sharpe
μLS
αFF5
t(α)
> LH-F
Panel A. Lookahead-free (vintage y−1 scores year y )
Base models:
PIT-4B
0.331
13.30
29.6
✓
3.37
0.57
0.56
10.16
✓
DatedGPT -base
0.112
2.64
4.9
✓
0.78
0.09
0.09
2.48
✓
Table 3: Panel regression and portfolio results for stock return prediction using LLM-based sentiment signals. See Section 4.3 for variable definitions and setup.
Family
β^2 (pp)
t(β^2)
β^2⋅SD (bp)
Panel A. Base models
PIT-4B
0.465
10.98
40.1
DatedGPT -base
0.138
3.13
7.7
ChronoGPT-base
0.033
0.90
0.0
Panel B. Instruct models
DatedGPT -instruct
0.472
10.65
26.4
Table 4: Lookahead premium test. See Section 4.3 for the full specification.
Figure 2: Average relative perplexity of DatedGPT-base-2020 evaluated on quarterly public company news headlines from 2013 to 2024.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Source-Document Evaluation Prompt
System: You are a data quality evaluator for SFT (Supervised Fine-Tuning) dataset creation. Given a document, determine whether it is suitable to be rewritten as SFT data and output a JSON object with exactly two fields: -- "score": integer 1-10. 1=garbage/SEO/truncated/chaotic, 10=perfect. -- "category": one of ["reasoning / logical-analysis", "knowledge / factual-QA", "writing / creative", "low-quality / unsuitable"]. Consider: information density, clarity, coherence, factual content, educational value, and whether the text could support good training examples. Output ONLY the JSON object. No other text. User: Document: {text} Evaluate. Output JSON only.
Appendix
Table 5: Prompt used for source-document selection. Qwen3-4B rates each candidate document on a 1–10 quality scale and assigns it to one of the three content categories or rejects it as unsuitable.
Hyperparameter
Value
Sequence Length
2048
Number of Layers
24
Embedding Size
2048
FFN Hidden Size
5504
Number of Heads
16
Position Encodings
RoPE
Appendix
Table 6: Model and pretraining hyperparameters.
Figure 3: Training loss curves for 2013 and 2024 DatedGPT-base models. All models exhibit smooth convergence without loss spikes or instability.
Figure 4: Relative perplexity of DatedGPT-base-2017 evaluated on quarterly public company news head-lines from 2013 to 2024.
Model (0-shot)
ARC-C
ARC-E
HellaSwag
PIQA
IFEval
MMLU
TruthfulQA
Baseline Models
SmolLM-1.7B-Instruct
34.9
54.3
56.1
72.4
15.5
27.1
27.7
GPT2-XL
28.4
51.0
50.8
70.5
15.0
25.3
22.4
TinyLlama-1.1B
30.5
43.7
55.0
72.5
14.8
24.8
26.0
OPT-1.3B
27.8
51.3
53.7
70.9
17.6
25.2
23.8
Pythia-1B
27.1
49.0
47.1
69.3
16.8
23.1
23.6
Appendix
Table 7: Zero-shot evaluation results on language understanding benchmarks. IFEval reports prompt-level strict accuracy. The DatedGPT-Instruct series models are identified by their training data cutoff year.
Model (5-shot)
ARC-C
ARC-E
HellaSwag
PIQA
IFEval
MMLU
TruthfulQA
Baseline Models
SmolLM-1.7B-Instruct
43.9
74.0
62.4
74.1
4.6
28.3
25.3
GPT2-XL
29.9
59.4
51.2
70.6
15.0
26.3
22.0
TinyLlama-1.1B
35.5
66.4
60.8
74.8
3.1
25.0
23.6
OPT-1.3B
29.2
59.6
54.1
71.0
17.4
24.8
24.0
Pythia-1B
28.2
56.9
47.5
70.1
17.4
25.8
23.6
Appendix
Table 8: Five-shot evaluation results on language understanding benchmarks. IFEval reports prompt-level strict accuracy. The DatedGPT-Base series models are identified by their training data cutoff year.