FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs
Organizations: Institute of Information Science, Academia Sinica, Taipei, Taiwan
Abstract
Asking a model to check retrieved text can also tell it which source to trust. We introduce FutureBench, a renewable framework for evaluating knowledge-context conflict using timestamped questions, controlled passage triplets, and no-context probes. Across 3,629 questions and six models, Verify (a prompt combining checking with passage priority) raises false-answer adoption in all five models tested on pre-cutoff events, mainly by displacing answers that were correct under Baseline, which allows both passage evidence and model knowledge. On post-cutoff events, only o4-mini increases adoption (+13.7 percentage points). A four-prompt comparison on 902 post-cutoff questions finds that passage priority without checking raises adoption, while moving to Verify lowers it in all five models. Adversarial-Aware, which warns of misleading passages and prioritizes model knowledge, restores 82.5-98.9% of displaced pre-cutoff answers with under one point of change in true-passage accuracy. FutureBench supports repeated evaluation on fixed cohorts and extension to new events as models change.
Figures & tables
| Prompt | Source priority | Check | Instruction |
| Core prompts and diagnostic references | |||
| Baseline | Knowledge | No | Use both sources; resolve conflicts with the answer believed correct. |
| No context | Knowledge | No | Answer without a passage. |
| Context only | Passage | No | Ignore outside knowledge. |
| Verify | Passage | Yes | Check passage reliability; answer from it or abstain. |
| Four-prompt comparison (§ 5.5 ) | |||
| False-answer adoption | True-passage accuracy | ||||
| Model | Baseline | Verify | Change | Baseline | Verify |
| Pre-cutoff | |||||
| DeepSeek-V3 | 79.9 | 86.8 | 77.4 | 55.9 | |
| Llama-4-M | 80.8 | 86.8 | 82.9 | 65.0 | |
| GPT-4.1-Mini | 82.1 | 91.6 | 84.2 | 65.8 | |
| GPT-5.1 | 42.7 | 74.8 | 92.3 | 62.5 | |
| Known | Unknown | |||
|---|---|---|---|---|
| Model | (pp) | (pp) | ||
| DeepSeek-V3 | 749 | 1076 | ||
| Llama-4-M | 959 | 866 | ||
| GPT-4.1-Mini | 1091 | 734 | ||
| GPT-5.1 | 1422 | 403 | ||
| o4-mini | 1139 | 686 | ||
| Model | Baseline | Passage-first | Verify | Soft |
|---|---|---|---|---|
| DeepSeek-V3 | 95.1 | 97.0 | 92.2 | 94.6 |
| Llama-4-M | 90.7 | 96.7 | 79.2 | 92.0 |
| GPT-4.1-Mini | 95.1 | 97.5 | 95.0 | 94.5 |
| GPT-5.1 | 93.6 | 97.0 | 92.8 | 92.7 |
| o4-mini | 82.3 | 97.3 | 95.6 | 69.0 |
| Claude-3.7 | 72.6 | — | 61.1 | — |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Model / role | Provider | Cutoff | API identifier |
|---|---|---|---|
| Generator / primary judge | Google DeepMind | — | google/gemini-3-flash |
| DeepSeek-V3 DeepSeek-AI et al. (2025) | DeepSeek | Jul 2024 | deepseek/deepseek-chat |
| Llama-4-M Meta AI (2025) | Meta | Aug 2024 | meta-llama/llama-4-maverick |
| GPT-4.1-Mini OpenAI (2025b) | OpenAI | Jun 2024 | openai/gpt-4.1-mini |
| GPT-5.1 OpenAI (2025a) | OpenAI | Sep 2024 | openai/gpt-5.1 |
| o4-mini OpenAI (2025c) | OpenAI | Jun 2024 | openai/o4-mini-high |
| Primary 2nd | Stick | Follow | Reject | Wrong |
|---|---|---|---|---|
| Stick | 50 | 0 | 0 | 0 |
| Follow | 4 | 45 | 0 | 1 |
| Reject | 0 | 0 | 49 | 1 |
| Wrong | 7 | 3 | 0 | 40 |
| Primary second | Stick | Follow | Reject | Wrong |
|---|---|---|---|---|
| Stick | 174 | 4 | 0 | 1 |
| Follow | 19 | 5969 | 0 | 9 |
| Reject | 1 | 0 | 128 | 0 |
| Wrong | 1 | 1 | 0 | 7 |
| Model | All outputs | Follow agreement | % |
|---|---|---|---|
| DeepSeek-V3 | 1804 | 1720/1728 | 99.5 |
| Llama-4-M | 1804 | 1698/1702 | 99.8 |
| GPT-4.1-Mini | 1804 | 1719/1731 | 99.3 |
| GPT-5.1 | 902 | 832/836 | 99.5 |
| Model | Change (pp) | Exact | ||
|---|---|---|---|---|
| Passage-first minus Baseline | ||||
| DeepSeek-V3 | 22 | 5 | 0.002 | |
| Llama-4-M | 54 | 0 | ||
| GPT-4.1-Mini | 22 | 1 | ||
| GPT-5.1 | 33 | 2 | ||
| o4-mini | 136 | 0 | ||
| Cohort | Baseline | Verify: no adopt. | Verify: adopt. |
|---|---|---|---|
| Pre-cutoff ( =1,825 † ) | No adoption | 231 | 976 |
| Adoption | 6 | 612 | |
| Post-cutoff ( =1,802) | No adoption | 100 | 251 |
| Adoption | 5 | 1446 |
| Pre-cutoff | Post-cutoff | |||||
|---|---|---|---|---|---|---|
| Model | ratio | ratio | ||||
| DeepSeek-V3 | 188 | 62 | 3.0 | 19 | 98 | 0.2 |
| Llama-4-M | 166 | 56 | 3.0 | 26 | 242 | 0.1 |
| GPT-4.1-Mini | 203 | 29 | 7.0 | 19 | 21 | 0.9 |
| GPT-5.1 | 599 | 13 | 46.1 | 36 | 49 | 0.7 |
| o4-mini | 976 | 6 | 162.7 | 251 | 5 | 50.2 |
| Question | Gold | False (in context) | No context | Baseline | Verify |
|---|---|---|---|---|---|
| How long had the insurgency waged by Boko Haram in Borno State lasted by April 2026? | 17 years | 15 years | 17 years | 17 years | 15 years |
| What political office did Justin Fairfax hold in Virginia between 2018 and 2022? | Lieutenant Governor | Attorney General | Lt. Gov. of Virginia | Lt. Gov. of Virginia | Attorney General |
| On what date did Microsoft end technical support and security updates for Windows 10? | Oct. 14, 2025 | Oct. 10, 2028 | Oct. 14, 2025 | Oct. 14, 2025 | Oct. 10, 2028 |
| Baseline abstention on insufficient passages | Post-cutoff references | |||
| Model | Pre-cutoff | Post-cutoff | Context only false adoption | No-context accuracy |
| DeepSeek-V3 | 31.6 | 62.4 | 92.6 | 1.3 |
| Llama-4-M | 14.7 | 64.4 | 91.0 | 1.8 |
| GPT-4.1-Mini | 6.1 | 27.0 | 95.0 | 4.8 |
| GPT-5.1 | 5.3 | 59.7 | 92.8 | 4.6 |
| o4-mini | 7.8 | 47.0 | 94.8 | 10.0 |
| Baseline adoption (%) | |||
|---|---|---|---|
| Model | Pre-cutoff | Post-cutoff | Change ( ) |
| DeepSeek-V3 | 79.9 | 93.4 | |
| Llama-4-M | 80.8 | 88.8 | |
| GPT-4.1-Mini | 82.1 | 94.1 | |
| GPT-5.1 | 42.7 | 92.0 | |
| o4-mini | 33.9 | 80.5 | |
| Known | Unknown | |||||
|---|---|---|---|---|---|---|
| Model | Baseline | Adv.-Aware | Change (pp) | Baseline | Adv.-Aware | Change (pp) |
| DeepSeek-V3 | 63.2 | 29.8 | 91.6 | 68.6 | ||
| Llama-4-M | 70.0 | 47.8 | 92.8 | 82.1 | ||
| GPT-4.1-Mini | 74.1 | 53.5 | 94.0 | 84.5 | ||
| GPT-5.1 | 33.1 | 12.9 | 76.4 | 54.1 | ||
| o4-mini | 21.9 | 13.8 | 53.8 | 44.0 | ||
Knowledge Conflicts in Language Models
Knowledge conflicts in language models concern how models reconcile stored knowledge with conflicting context, retrieved evidence, tools, or memory.
See all 155 papers