cs.CLNov 17, 2026

FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs

Authors: Yi-Cheng Lai, Hen-Hsen Huang

Organizations: Institute of Information Science, Academia Sinica, Taipei, Taiwan

Abstract

Asking a model to check retrieved text can also tell it which source to trust. We introduce FutureBench, a renewable framework for evaluating knowledge-context conflict using timestamped questions, controlled passage triplets, and no-context probes. Across 3,629 questions and six models, Verify (a prompt combining checking with passage priority) raises false-answer adoption in all five models tested on pre-cutoff events, mainly by displacing answers that were correct under Baseline, which allows both passage evidence and model knowledge. On post-cutoff events, only o4-mini increases adoption (+13.7 percentage points). A four-prompt comparison on 902 post-cutoff questions finds that passage priority without checking raises adoption, while moving to Verify lowers it in all five models. Adversarial-Aware, which warns of misleading passages and prioritizes model knowledge, restores 82.5-98.9% of displaced pre-cutoff answers with under one point of change in true-passage accuracy. FutureBench supports repeated evaluation on fixed cohorts and extension to new events as models change.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

    Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt SensitivityLLM Evaluation

  2. DataDignity: Training Data Attribution for Large Language Models

    May 7, 2026Xiaomin Li, Andrzej Banburski-Fahey, Jaron LanierLLM AuditingInformation Retrieval