DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

Jintao Cheng; Weibin Li

cs.CVAug 25, 2026

DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

Authors: Jintao Cheng, Weibin Li

Organizations: The Hong Kong University of Science and Technology · University of Macau

Abstract

Visual Language Models (VLMs) excel at describing visible scene content but struggle to reason about dynamic multi-agent interactions, where action semantics depend on coordinated roles and spatial-temporal dependencies. We formalize this capability as multi-agent tactical reasoning and introduce DoublesEval, a diagnostic evaluation framework that leverages professional doubles badminton as a structurally tractable testbed. DoublesEval employs a key-moment-based protocol that decomposes rallies into tactically salient instants and probes models across four interpretable dimensions: atomic recognition, intra-segment composite understanding, cross-segment causal reasoning, and high-level tactical abstraction. This design isolates where reasoning fails, rather than merely measuring answer correctness. To address observed failure modes, we propose TacticCheck, a lightweight constraint-guided test-time consistency checker that reranks candidate answers using the model's own lower-level tactical predictions, requiring no parameter updates or ground-truth labels at inference time. Evaluating four representative open-source VLMs on 60 curated rallies (yielding $\sim$ 9.6K structured instances) via a zero-shot protocol, we find that models remain weak across all diagnostic levels, with especially clear bottlenecks in spatial state, interaction binding, and terminal evidence. TacticCheck delivers consistent gains across all evaluated models, while still leaving a substantial gap to robust tactical reasoning. These results highlight the need for structured, interaction-aware evaluation paradigms for next-generation VLMs. The source code is available in blue{our GitHub repository}.

Figures & tables

Figure 1: Comparison between existing action-level evaluations and our proposed DoublesEval framework. Left: Current VLM benchmarks primarily focus on perception-level surface descriptions, such as isolated action recognition (e.g., distinguishing a smash from a clear). Middle: The core semantic gap lies in multi-agent tactical coordination, which requires mapping atomic actions to dynamic role attributions, spatial coverage constraints, and causal chains. Right: DoublesEval addresses this by introducing a diagnostic framework tailored for tactical-level multi-agent reasoning, systematically probing models across four structured reasoning dimensions.

Figure 2: Overview of the DoublesEval diagnostic pipeline. By anchoring on the scoring moment, we extract structured tactical evidence across the bilateral causal chain. Layered reasoning probes (L1-L4) are then applied to precisely localize where VLM multi-agent reasoning fails, rather than merely evaluating outcome correctness.

Rally-level basics
Statistic	Mean	Std	Range
Duration (seconds)	$10.2$	$3.1$	$[6.5,18.9]$
Strokes per rally ( $\overline{S}$ )	$12.4$	$3.8$	$[7,23]$
Diagnostic segment density
Segments per rally	$40.1$	$5.3$	$[32,51]$
Avg. segment duration (seconds)	$2.8$	$0.6$	$[1.8,4.2]$

Table 1: Dataset statistics for the 60 curated rallies. Diagnostic segments are tactically decisive intervals (e.g., scoring windows, role transitions) anchored within the full-rally context. Instance counts per dimension reflect balanced coverage across the four diagnostic targets. Annotation reliability metrics are computed on the 30% expert-audited subset.

Figure 3: Two-Stage Evaluation and TacticCheck Pipeline. The framework processes key moments to generate two complementary zero-shot outputs: SCQs (evaluated via exact match) and SAQs (evaluated by a GPT-5 Judge). The diagnosed bottlenecks then inform a test-time checking loop, where a structured tactical checklist and constraint prompts guide the VLM to iteratively revise its initial answers.

Model	Overall	L1	L2	L3	L4
Qwen2.5VL	27.57	33.50	27.50	23.80	25.50
Qwen3VL	29.00	35.00	29.00	25.00	27.00
VideoLLaMA3	32.30	35.65	28.79	29.33	35.43
Molmo2	31.31	37.50	29.35	26.11	26.86

Table 2: Zero-shot accuracy over single-choice and short-answer questions (%). Accuracy remains low across all four diagnostic levels. L1 is easier than the other levels for most models, but even atomic recognition is far from saturated.

Figure 4: Level-wise zero-shot performance over single-choice and short-answer questions. The benchmark does not only test final answer correctness: it separates atomic recognition, composite understanding, causal reasoning, and tactical abstraction.

Figure 5: Capability-level diagnosis. Temporal phase is relatively easier, while spatial state, interaction binding, and score-detail reasoning are consistently weak. This suggests that models often see parts of the scene but fail to bind players, teams, space, and outcome into a coherent tactical state.

Figure 6: L1 field-level diagnosis. Even within atomic recognition, not all fields are equally easy. Phase and scoring side are more reliable, while near/far interaction fields and stroke pattern remain difficult.

Model	Base	TacticCheck	$\Delta$	W $\rightarrow$ R	R $\rightarrow$ W
Qwen2.5VL	27.57	35.20	+7.63	12.41	4.78
Qwen3VL	29.00	37.20	+8.20	13.46	5.26
VideoLLaMA3	32.30	39.43	+7.13	12.84	5.71
Molmo2	31.31	39.14	+7.83	12.97	5.14

Table 3: Test-time improvement with TacticCheck (%). TacticCheck improves all models, but also introduces a smaller number of right-to-wrong flips. This confirms that consistency checking is useful but not error-free.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Oct 22, 2026

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.

Advait Gupta, Sachin Konan, Tom Yin +2

University of Maryland, College Park · Princeton University

VLM Evaluation Agent Benchmarks Visual Grounding

Ask PDF

Apr 20, 2026

Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models

Recent Vision-Language-Action (VLA) models report impressive success rates on standard robotic benchmarks, fueling optimism about general-purpose physical intelligence. However, recent evidence suggests a systematic misalignment between standard benchmark success and true embodied reasoning, raising the question of whether these high scores reflect genuine cognitive capability. To address this gap, we introduce BeTTER, a diagnostic Benchmark for Testing True Embodied Reasoning in robotic policies. BeTTER applies targeted causal interventions (e.g., spatial layout shifts, temporal extrapolation) while enforcing kinematic isolation to explicitly decouple high-level reasoning failures from low-level execution limits. Through systematic evaluation, we reveal that state-of-the-art VLAs catastrophically fail in dynamic scenarios, exhibiting severe lexical-kinematic shortcuts, behavioral inertia, and semantic feature collapse. Crucially, our mechanistic analysis traces these symptoms to fundamental architectural bottlenecks - such as capacity compression and myopic downsampling - which systematically degrade the model's foundational semantic representation. We demonstrate that highly static evaluation protocols effectively mask this degradation by allowing optimization to overfit to sensorimotor priors. Supported by real-world robotic validation, our findings confirm that this representational breakdown is not a simulation artifact, highlighting the critical need for future VLA paradigms to resolve the structural tension between high-frequency control and high-level reasoning.

Haiweng Xu, Sipeng Zheng, Hao Luo +3

1Peking University · 3BeingBeyond · 2Tsinghua University

VLM Evaluation VLM Reasoning Vision-Language-Action Models Robot Manipulation Benchmarks Embodied AI Shortcut Learning+2

Ask PDF

Jun 17, 2026

RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models

Modern Vision-Language Models (VLMs) often struggle with strategic reasoning, i.e., anticipating and influencing other agents' actions, under uncertainty in competitive and cooperative settings. Real-time strategy (RTS) games can be a natural testbed for diagnosing this limitation, as they demand coordination with allies, adaptation to opponents' strategy, and long-horizon planning under partial observability. However, existing RTS benchmarks offer limited evaluation scope, lack systematic competency diagnosis, and remain fixed in the pre-designed scenario coverage. To address these limitations, we present RTSGameBench, which is built on Beyond All Reason, a large-scale RTS game with an expanded battlefield that demands broader strategy diversity than the existing testbeds. The proposed benchmark provides evaluations through diverse gameplay across various matchup structures, diagnostic assessment via mini-games, each targeting an individual strategic competency, and extensible coverage via a self-evolving generation framework that converts free-form queries into new mini-games, improving over successive cycles. Additionally, for VLMs to operate in large-scale RTS games, we provide RTSGameAgent that manages units by an FSM with agentic memory. We empirically validate that multiple state-of-the-art VLMs do not perform well when matchups demand tighter coordination, multiagent coordination and when task scale increases.

San Kim, Daechul Ahn, Reokyoung Kim +3

Seoul National University

Multi-Agent Coordination Game-Playing Agents VLM Reasoning Agent Benchmarks

Ask PDF

Abstract

VLM Evaluation

Explore similar work

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models

RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models