Overview

Counterfactual evaluation asks whether a system would change its prediction, explanation, or action appropriately when relevant evidence or conditions change, rather than merely measuring performance on observed cases. Common approaches use paired interventions and benchmarks, simulator or world-model rollouts, controlled replay, and causal or off-policy estimators to compare factual and alternative outcomes. These methods are applied to language and vision models, clinical decisions, recommender systems, and interactive agents, where ordinary accuracy may hide failures in evidence use, safety, or responsiveness. A central challenge is that alternative outcomes are usually unobserved, so conclusions depend on assumptions about causal structure, simulator fidelity, intervention realism, and which metrics reflect useful behavior.

Latest papers

CardsList

103 papers10 papers in the last four weeks, down 50% on the four weeks before. 0.1% of all new papers.

  1. Does Your World Model Know Actions Have Consequences?

    Nov 11, 2026Ayush Gaggar, Jared Berry, Todd D. MurpheyWorld Model EvaluationAction-Conditioned World Models

  2. How Much Can Best-of-N Help a World Action Model? Measuring Selectable Headroom and Realizing It in Closed Loop

    Oct 27, 2026Haojia Gao, Yue Cao, Yumin Liu +6World Model-Based PlanningAction-Conditioned World Models

  3. Counterfactual Model Testing Under Distribution Shift

    Oct 24, 2026Joran Leest, Ilias Gerostathopoulos, Claudia Raibulet +1Distribution Shift RobustnessCounterfactual Evaluation

  4. Measuring Local Acoustic Contribution in Speech LLMs via Word-Span Counterfactual Interventions

    Oct 21, 2026Hardik B. Sailor, Qiongqiong Wang, Dheemant Rastogi +2Audio-Language Model EvaluationSpeech Language Models

  5. Concordance Plausibility Check for Intervention-Conditioned Human Digital Twins

    Oct 17, 2026Saeed Farzi, Christopher Kipps, John McBeth +7Counterfactual EvaluationDigital Twins

  6. CascadeLedger: Counterfactual Auditing of Information and Resource Integrity in LLM Cascades

    Oct 14, 2026Miaobo Hu, Shuhao Hu, Xiaobo Guo +3Language Model CascadingLLM Auditing

  7. RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors

    Oct 9, 2026David Ahmedt-Aristizabal, Abdelwahed Khamis, Mohammad Ali Armin +1Robot Failure DetectionRobot Failure Recovery

  8. CoBe: A Benchmark For Conversational Counterfactual Text Editing

    Oct 9, 2026Arvind Raghavan, Inwoo Hwang, Shreyas Havaldar +6Causal Counterfactual GenerationCounterfactual Reasoning in Language Models

  9. Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation

    Oct 7, 2026Biao Xiang, Ali Eshragh, Yuexing Li +1Off-Policy EvaluationCounterfactual Evaluation

  10. Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists

    Oct 4, 2026Kate Zhang, Yuante LiAI Agent EvaluationScientific Agent Benchmarks

  11. TerrainForge: Physics-Grounded Road Geometry Editing for Counterfactual Autonomous Driving

    Oct 2, 2026Yang Chen, Yicheng Zhu, zhenning Li +2Autonomous Driving Scenario GenerationAutonomous Driving Simulation

  12. Fixing a Model That Learned Worse Cancer Means Lower Risk: Monotonic Constraints in Bladder Cancer Recurrence Prediction

    Oct 1, 2026Saram Abbas, David Thomas, Naeem Soomro +3Counterfactual EvaluationGradient-Boosted Decision Trees

  13. What Happens During Autonomous Deep Research After the User Steps Away?

    Sep 27, 2026Yimin Liu, Yijia Zhang, Yanmin Li +4Counterfactual EvaluationDeep Research Agents

  14. Self-Reports Do Not Identify Self-Models: An Identifiability Test for Counterfactual Reports

    Sep 26, 2026Phongsakon Mark Konrad, Toygar Tanyel, Serkan AyvazLanguage Model Self-AssessmentMetacognition in Language Models

  15. COUNTERMEM: World-Model Verified Counter-Factual Memory for Language Agents

    Sep 25, 2026Hongji Pu, Ruixiang Tang, Yongfeng ZhangMemory-Augmented Language Model AgentsPersistent Memory for Language Models

  16. Robust to Which Model Change? A Unified Evaluation of Robust Counterfactual Explanations

    Sep 25, 2026Marcin Kostrzewa, Maciej ZiębaCounterfactual EvaluationCounterfactual Explanations

  17. Selective Amortization of Full-Budget Counterfactual Reasoning for Visual Token Communication

    Sep 25, 2026Qinglei Qi, Zhihe Liang, Fengzhan Jing +5Generative Image CompressionAdaptive Computation

  18. Memory That Changes Action Is Not Memory That Guides It: Counterfactual Auditing of History-Conditioned Robot Policies

    Sep 23, 2026Jiajie Zhang, Yankai Xiang, Changhao ChenMemory-Augmented AgentsRobotic Policy Evaluation