cs.AINov 18, 2026

Choosing What to Observe: Budget-Paired Counterfactual Audits for Credible Evaluation of Visual Agents

Authors: Ruiqi Liu, Jinming Li, Mo Wang, Yang Lu, Yanlan Kang

Organizations: Fudan University · Institute of Medical Philosophy & Future AI (IMPF-AI) · The University of Warwick · Southern University of Science and Technology

Abstract

Visual agents that read dense documents must ration a finite observation budget across candidate regions, yet answer accuracy conflates where an agent looks, how it adapts to the reads that remain, and how it an?swers once observation stops. We introduce budget-paired counterfactual audits, an evaluation protocol that holds public evidence and region bindings fixed while perturbing only the remaining budget or de?clared task parameters, and credits every acquired history with exact continuation values. The resulting accounting decomposes lost expected accuracy into stepwise acquisition loss and terminal decision loss, separating planning from fixed-role heuristics. Across forty paired routing anchors, five open-weight

Figures & tables

Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets

    Sep 30, 2026Priyanath Maji, Spandan Ghose ChowdhuryThompson SamplingAI Agent Evaluation

  2. One Human, NN Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

    Jul 30, 2026Cesare Zavattari, Alessandro Tommasi, Giuseppe PrencipeScalable OversightAI Agent Reliability

  3. No Judgment Without a Reason: Counterfactual Receipts for Versioned AI Evaluators

    Aug 21, 2026Ye Chen, Weining ZhangLLM EvaluationCounterfactual Evaluation