DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

Jintao Cheng; Weibin Li

cs.CVAug 25, 2026

DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

Authors: Jintao Cheng, Weibin Li

Organizations: The Hong Kong University of Science and Technology · University of Macau

Abstract

Visual Language Models (VLMs) excel at describing visible scene content but struggle to reason about dynamic multi-agent interactions, where action semantics depend on coordinated roles and spatial-temporal dependencies. We formalize this capability as multi-agent tactical reasoning and introduce DoublesEval, a diagnostic evaluation framework that leverages professional doubles badminton as a structurally tractable testbed. DoublesEval employs a key-moment-based protocol that decomposes rallies into tactically salient instants and probes models across four interpretable dimensions: atomic recognition, intra-segment composite understanding, cross-segment causal reasoning, and high-level tactical abstraction. This design isolates where reasoning fails, rather than merely measuring answer correctness. To address observed failure modes, we propose TacticCheck, a lightweight constraint-guided test-time consistency checker that reranks candidate answers using the model's own lower-level tactical predictions, requiring no parameter updates or ground-truth labels at inference time. Evaluating four representative open-source VLMs on 60 curated rallies (yielding $\sim$ 9.6K structured instances) via a zero-shot protocol, we find that models remain weak across all diagnostic levels, with especially clear bottlenecks in spatial state, interaction binding, and terminal evidence. TacticCheck delivers consistent gains across all evaluated models, while still leaving a substantial gap to robust tactical reasoning. These results highlight the need for structured, interaction-aware evaluation paradigms for next-generation VLMs. The source code is available in blue{our GitHub repository}.

Figures & tables

Figure 1: Comparison between existing action-level evaluations and our proposed DoublesEval framework. Left: Current VLM benchmarks primarily focus on perception-level surface descriptions, such as isolated action recognition (e.g., distinguishing a smash from a clear). Middle: The core semantic gap lies in multi-agent tactical coordination, which requires mapping atomic actions to dynamic role attributions, spatial coverage constraints, and causal chains. Right: DoublesEval addresses this by introducing a diagnostic framework tailored for tactical-level multi-agent reasoning, systematically probing models across four structured reasoning dimensions.

Figure 2: Overview of the DoublesEval diagnostic pipeline. By anchoring on the scoring moment, we extract structured tactical evidence across the bilateral causal chain. Layered reasoning probes (L1-L4) are then applied to precisely localize where VLM multi-agent reasoning fails, rather than merely evaluating outcome correctness.

Rally-level basics
Statistic	Mean	Std	Range
Duration (seconds)	$10.2$	$3.1$	$[6.5,18.9]$
Strokes per rally ( $\overline{S}$ )	$12.4$	$3.8$	$[7,23]$
Diagnostic segment density
Segments per rally	$40.1$	$5.3$	$[32,51]$
Avg. segment duration (seconds)	$2.8$	$0.6$	$[1.8,4.2]$

Table 1: Dataset statistics for the 60 curated rallies. Diagnostic segments are tactically decisive intervals (e.g., scoring windows, role transitions) anchored within the full-rally context. Instance counts per dimension reflect balanced coverage across the four diagnostic targets. Annotation reliability metrics are computed on the 30% expert-audited subset.

Figure 3: Two-Stage Evaluation and TacticCheck Pipeline. The framework processes key moments to generate two complementary zero-shot outputs: SCQs (evaluated via exact match) and SAQs (evaluated by a GPT-5 Judge). The diagnosed bottlenecks then inform a test-time checking loop, where a structured tactical checklist and constraint prompts guide the VLM to iteratively revise its initial answers.

Model	Overall	L1	L2	L3	L4
Qwen2.5VL	27.57	33.50	27.50	23.80	25.50
Qwen3VL	29.00	35.00	29.00	25.00	27.00
VideoLLaMA3	32.30	35.65	28.79	29.33	35.43
Molmo2	31.31	37.50	29.35	26.11	26.86

Table 2: Zero-shot accuracy over single-choice and short-answer questions (%). Accuracy remains low across all four diagnostic levels. L1 is easier than the other levels for most models, but even atomic recognition is far from saturated.

Figure 4: Level-wise zero-shot performance over single-choice and short-answer questions. The benchmark does not only test final answer correctness: it separates atomic recognition, composite understanding, causal reasoning, and tactical abstraction.

Figure 5: Capability-level diagnosis. Temporal phase is relatively easier, while spatial state, interaction binding, and score-detail reasoning are consistently weak. This suggests that models often see parts of the scene but fail to bind players, teams, space, and outcome into a coherent tactical state.

Figure 6: L1 field-level diagnosis. Even within atomic recognition, not all fields are equally easy. Phase and scoring side are more reliable, while near/far interaction fields and stroke pattern remain difficult.

Model	Base	TacticCheck	$\Delta$	W $\rightarrow$ R	R $\rightarrow$ W
Qwen2.5VL	27.57	35.20	+7.63	12.41	4.78
Qwen3VL	29.00	37.20	+8.20	13.46	5.26
VideoLLaMA3	32.30	39.43	+7.13	12.84	5.71
Molmo2	31.31	39.14	+7.83	12.97	5.14

Table 3: Test-time improvement with TacticCheck (%). TacticCheck improves all models, but also introduces a smaller number of right-to-wrong flips. This confirms that consistency checking is useful but not error-free.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

AgentVQA: A Unified Benchmark for Agentic Visual Understanding
Oct 22, 2026Advait Gupta, Sachin Konan, Tom Yin +2VLM Evaluation Agent Benchmarks
University of Maryland, College Park · Princeton University
Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models
Apr 20, 2026Haiweng Xu, Sipeng Zheng, Hao Luo +3VLM Evaluation VLM Reasoning
1Peking University · 3BeingBeyond · 2Tsinghua University
RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models
Jun 17, 2026San Kim, Daechul Ahn, Reokyoung Kim +3Multi-Agent Coordination Game-Playing Agents
Seoul National University

Oct 22, 2026J/K move · Enter open · S save

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Advait Gupta, Sachin Konan, Tom Yin, Gabriel Sarch+1

University of Maryland, College Park · Princeton University

VLM Evaluation Agent Benchmarks Visual Grounding

Open paper AskPDF arXiv

Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.

Abstract

VLM Evaluation

Explore similar work

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models

RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models