Asking a model to check retrieved text can also tell it which source to trust. We introduce FutureBench, a renewable framework for evaluating knowledge-context conflict using timestamped questions, controlled passage triplets, and no-context probes. Across 3,629 questions and six models, Verify (a prompt combining checking with passage priority) raises false-answer adoption in all five models tested on pre-cutoff events, mainly by displacing answers that were correct under Baseline, which allows both passage evidence and model knowledge. On post-cutoff events, only o4-mini increases adoption (+13.7 percentage points). A four-prompt comparison on 902 post-cutoff questions finds that passage priority without checking raises adoption, while moving to Verify lowers it in all five models. Adversarial-Aware, which warns of misleading passages and prioritizes model knowledge, restores 82.5-98.9% of displaced pre-cutoff answers with under one point of change in true-passage accuracy. FutureBench supports repeated evaluation on fixed cohorts and extension to new events as models change.
Figures & tables
Figure 1: Same passage, different answers (o4-mini; pre-cutoff).
Figure 2: FutureBench evaluation framework.
Prompt
Source priority
Check
Instruction
Core prompts and diagnostic references
Baseline
Knowledge
No
Use both sources; resolve conflicts with the answer believed correct.
No context
Knowledge
No
Answer without a passage.
Context only
Passage
No
Ignore outside knowledge.
Verify
Passage
Yes
Check passage reliability; answer from it or abstain.
Four-prompt comparison (§ 5.5 )
Table 1: Prompt guide.
False-answer adoption
True-passage accuracy
Model
Baseline
Verify
Change
Baseline
Verify
Pre-cutoff
DeepSeek-V3
79.9
86.8
+6.9∗∗∗
77.4
55.9
Llama-4-M
80.8
86.8
+6.0∗∗∗
82.9
65.0
GPT-4.1-Mini
82.1
91.6
+9.5∗∗∗
84.2
65.8
GPT-5.1
42.7
74.8
+32.1∗∗∗
92.3
62.5
Table 2: Baseline versus Verify by temporal cohort.
Known
Unknown
Model
N
Δ (pp)
N
Δ (pp)
DeepSeek-V3
749
+16.4
1076
+0.3
Llama-4-M
959
+12.3
866
−0.9
GPT-4.1-Mini
1091
+14.9
734
+1.5
GPT-5.1
1422
+37.3
403
+13.9
o4-mini
1139
+60.4
686
+41.1
Table 3: Adoption changes on known and unknown items.
Figure 3: One question, opposite failures (o4-mini; shortened).
Figure 4: Verify adoption changes by temporal cohort (pp).
Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA