cs.CLNov 4, 2026
SaveLLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting
Organizations: The Fin AI · University of Manchester · Mohamed bin Zayed University of Artificial Intelligence · Deepkin AI ·
Abstract
Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question, Recall and Predict questions that pair each candidate value with a claim of knowing or guessing without a declared date, and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27. Results show that for most frontier models, self-reports understate knowledge. GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and answers 61% of the facts it says it does not know. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, and a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged. Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff. These findings motivate behavioral measurement of temporal knowledge boundaries, and FinDendro provides a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.
Figures & tables
Figure 1: Model knowledge, claims, and release timelines. (a) Knowledge and claim boundaries measured by 0.10588 0 0.65098F0.20784 0 0.57647i0.31373 0 0.50196n0.41961 0 0.42745D0.52157 0 0.35686e0.62745 0 0.28235n0.73333 0 0.20784d0.83529 0 0.13333r0.94118 0 0.05882o __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: , compared with self-reported and vendor-stated cutoffs and public release dates. (b) A historical example in which GPT-5 denies knowing an outcome but produces the correct value, illustrating why self-reported knowledge boundaries cannot reliably prevent leakage in retrospective forecasting.
| Benchmark | Fin. | Dated | Cutoff | Claims | Guards |
|---|---|---|---|---|---|
| FinBen [ 33 ] | ✓ | ✗ | ✗ | ✗ | ✗ |
| StockBench [ 3 ] | ✓ | ✗ | ✗ | ✗ | ✗ |
| FinTMMBench [ 41 ] | ✓ | ✓ | ✗ | ✗ | ✗ |
| ForecastBench [ 17 ] | • | ✗ | ✗ | ✗ | ✗ |
| Look-Ahead-Bench [ 1 ] | ✓ | • | ✗ | ✗ | • |
| SelfAware [ 39 ] | ✗ | ✗ | ✗ | ✓ | ✗ |
Table 1: Comparison with related benchmarks. Fin. denotes financial or economic content; Dated , reference-dated facts; Cutoff , behavioral knowledge-boundary estimation; Claims , explicit knowledge claims; and Guards , evaluation of leakage safeguards. ✓ = yes, • = partial, and ✗ = no.
| Family | Series | Facts | Before 2021 | 2021 on | Controlled |
|---|---|---|---|---|---|
| Macroeconomic | 12 | 1,353 | 528 | 825 | 715 |
| Stock prices | 50 | 540 | 264 | 276 | 292 |
| Fed decisions | 1 | 333 | 264 | 69 | 333 |
| Total | 63 | 2,226 | 1,056 | 1,170 | 1,340 |
Table 2: The evaluation sample by fact family. Controlled counts the facts that count toward knowledge, which are the numerical facts where repeating last month’s value fails and all policy facts.
| Fact. June 2023 U.S. headline CPI inflation: the 12-month change in FRED CPIAUCNS (all items, NSA), rounded to one decimal place. Candidate values: 3.0%, 3.4%, 3.2%, 2.8%. |
|---|
| Task 1: Value “What was U.S. headline CPI inflation in June 2023?” Options: the four candidate values. |
| Task 2: Value and claim |
| Recall : “Recall the value from your memory: what was U.S. headline CPI inflation in June 2023? If you do not know it, select ‘I do not know’. Options: each value appears twice: once as known and once as a guess (8 choices). |
| Predict : “Predict U.S. headline CPI inflation in June 2023.” Options: each value appears twice: once as already known and once as a prediction of an unknown outcome (8 choices). |
| Task 3: Dated prediction “Assume that today is May 31, 2023. Answer as you would on that date. Predict U.S. headline CPI inflation in June 2023.” Options: the same paired options as Task 2 Predict (8 choices). |
Table 3: An illustrative CPI fact across three tasks and four prompts. Candidate values are shared, and options are summarized. Each prompt is sent independently with shuffled options.
| Task 1 | Task 2: Recall claims | Task 2: Predict Task 3 | |||||
| Model | Knows | Says | Acc says | Acc denies | Gap [95% CI] | Says | Accuracy |
| GPT-5 | 0.76 | 0.38 | 0.95 | 0.61 | 0.18 [0.13, 0.22] | 0.39 0.01 | 0.76 0.75 |
| DeepSeek-V3.2 | 0.54 | 0.28 | 0.65 | 0.48 | 0.08 [0.03, 0.15] | 0.37 0.46 | 0.44 0.43 |
| GPT-4o | 0.50 | 0.20 | 0.83 | 0.34 | 0.39 [0.32, 0.45] | 0.33 0.04 | 0.43 0.44 |
| Llama-3.3-70B | 0.43 | 0.48 | 0.58 | 0.29 | 0.28 [0.20, 0.34] | 0.63 0.00 | 0.41 0.41 |
| Qwen3.5-9B | 0.37 | 0.69 | 0.40 | 0.30 | 0.10 [0.03, 0.19] | 0.63 0.53 | 0.31 0.35 |
Table 4: Main results by task and wording (8,904 prompts per model). Knows is Task 1 accuracy on the 1,340 controlled facts (chance 0.27). In Tasks 2 and 3, Says is the share claiming knowledge across the full sample of 2,226 facts, regardless of answer correctness. Task 2 Recall uses no declared date; Acc says and Acc denies are Task 1 accuracies on facts grouped by these claims. Gap is their within-period accuracy difference, weighted by period size, with 95% bootstrap intervals. Arrows compare Task 2 Predict (without a declared date) Task 3 (dated prediction), for both Says and Accuracy .
Figure 2: What models know versus what they say. The x-axis is Task 1 accuracy on controlled facts (chance 0.27) and the y-axis is the share of Task 2 Recall answers that claim knowledge. Points below the diagonal under-report their knowledge and points above it over-claim. Dotted lines show each model’s distance from the diagonal.
Figure 3: Temporal patterns of factual accuracy and knowledge claims. Purple and orange lines show Task 1 accuracy and the share of knowledge claims, respectively. Shading highlights their differences. Markers indicate the estimated knowledge boundary ( ), claim boundary ( ), self-reported cutoff ( ), and vendor-stated cutoff (|), where available. Years before 2021 are compressed, while later periods are shown quarterly.
| Fact | Task 2: Recall | Task 3: Predict |
|---|---|---|
| Intel close, Jan. 2021 | does not know; guesses 55.51 | does not know; predicts 55.51 |
| Chevron close, Jan. 2006 | does not know; guesses 59.38 | does not know; predicts 59.38 |
| JPY per USD, Feb. 2024 avg. | does not know; guesses 149.62 | does not know; predicts 149.62 |
Table 5: Facts GPT-5 says it does not know but answers correctly in Task 2 Recall (without a declared date) and Task 3 (dated prediction, with the date set to the end of the preceding month). Each value is the published one among four candidates.
| Inside window | After window | |||||
|---|---|---|---|---|---|---|
| Model | Denied | Acc | Denied | Acc | ||
| GPT-5 | 1,109 | 0.46 | 0.75 | 231 | 1.00 | 0.33 |
| DeepSeek-V3.2 | 1,109 | 0.57 | 0.53 | 231 | 0.96 | 0.34 |
| GPT-4o | 925 | 0.60 | 0.37 | 415 | 0.97 | 0.32 |
| Llama-3.3-70B | 711 | 0.22 | 0.31 | 629 | 0.78 | 0.27 |
Table 6: Self-report filtering on the 1,340 controlled facts, split at each model’s knowledge ends . Using Task 2 Recall without a declared date, we report each group’s size ( ), the share disclaimed ( Denied ), and Task 1 accuracy on disclaimed facts ( Acc ; chance 0.27).
| Inside window | After | |||
|---|---|---|---|---|
| Model | Pred. | Acc (all) | Acc (pred.) | Acc (all) |
| GPT-5 | 0.99 | 0.83 | 0.82 | 0.39 |
| DeepSeek-V3.2 | 0.46 | 0.45 | 0.38 | 0.32 |
| GPT-4o | 0.91 | 0.48 | 0.44 | 0.33 |
| Llama-3.3-70B | 1.00 | 0.51 | 0.51 | 0.29 |
Table 7: A simulated dated backtest using Task 3 (dated prediction), on controlled facts inside and after each model’s knowledge window. Pred. is the share of in-window answers labeled as predictions, followed by accuracy on all answers and on those labeled as predictions.
Figure 4: Knowledge–claim gaps across financial fact families. Each bar shows the difference between Task 1 recall accuracy and the proportion of Task 2 Recall responses claiming knowledge without a declared date.
| Model | Cutoff | Before | After | Difference |
|---|---|---|---|---|
| TiMaGPT | 2014 | 0.29 | 0.27 | 0.016 [ 0.023, 0.055] |
| TiMaGPT | 2016 | 0.28 | 0.26 | 0.021 [ 0.020, 0.064] |
| TiMaGPT | 2018 | 0.28 | 0.26 | 0.021 [ 0.020, 0.061] |
| TiMaGPT | 2020 | 0.28 | 0.27 | 0.006 [ 0.032, 0.048] |
| TiMaGPT | 2022 | 0.27 | 0.26 | 0.006 [ 0.037, 0.052] |
| ChronoGPT | 2014 | 0.28 | 0.27 | 0.001 [ 0.038, 0.045] |
Table 8: Point-in-time models scored by candidate likelihood. We give Task 1 accuracy on controlled facts up to and after the year of each training cutoff, and the difference with a 95% bootstrap interval.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Knowledge ends | Claims end | Self-reported | Vendor-stated |
|---|---|---|---|---|
| GPT-5 | 2024Q3 | 2024Q3 | 2024-10 | 2024-09 |
| DeepSeek-V3.2 | 2024Q3 | 2024Q4 | 2024-07 | n/a |
| GPT-4o | 2023Q1 | 2023Q2 | 2023-10 | 2023-10 |
| Llama-3.3-70B | 2021Q2 | 2023Q1 | 2023-12 | 2023-12 |
Table 9: Estimated boundaries and reported cutoffs for the four frontier models. Knowledge ends and Claims end follow the estimation procedure in Section 3.4 . Self-reported is the cutoff given when the model is asked directly; Vendor-stated is the date published by its developer. “n/a” indicates that no vendor-stated date is recorded.
| Dimension | Passing criteria |
|---|---|
| Factual correctness | A reliable source supports the fact. The entity, indicator, reference period, units, calculations, and rounding are correct. |
| First-release and temporal validity | Historical evidence supports the gold answer as the first-release value. The reference period, first publication date, and hypothetical date in Task 3 are correctly identified and distinguished. |
| Question clarity | The question uniquely identifies the intended fact. Relevant conventions are explicit, including year-over-year versus month-over-month changes, monthly averages versus month-end values, seasonal adjustment, and stock-price adjustments. |
| Option validity and task consistency | The correct factual candidate is present and unique; distractors are distinct, and gold option mappings are correct. The four prompts refer to the same fact and share the same candidate values. Each candidate has both knowledge-claim options in Tasks 2 and 3. |
Table 10: Manual validation criteria. An item passes a dimension when it meets all criteria listed in that row.
| Model | Params. | Architecture | Cutoff | Access | HF Repository ID | License |
| Frontier models | ||||||
| GPT-5 | NR | Proprietary | 2024-09 | API | — | Proprietary |
| DeepSeek-V3.2 | 685B (37B active) | MoE | NR | API | deepseek-ai/DeepSeek-V3.2 | MIT |
| GPT-4o | NR | Proprietary | 2023-10 | API | — | Proprietary |
| Llama-3.3-70B-Instruct | 70B | Dense | 2023-12 | API | meta-llama/Llama-3.3-70B-Instruct | Llama 3.3 |
| Open models | ||||||
Table 11: Models evaluated in 0.10588 0 0.65098F0.20784 0 0.57647i0.31373 0 0.50196n0.41961 0 0.42745D0.52157 0 0.35686e0.62745 0 0.28235n0.73333 0 0.20784d0.83529 0 0.13333r0.94118 0 0.05882o __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: __color_backend_reset: . Parameter counts are reported in billions (B), with active parameters given for mixture-of-experts models where available. Cutoff refers to the vendor-reported knowledge cutoff for frontier models and the training-data cutoff for point-in-time models. NR indicates information not reported or verified.