cs.CVMar 12, 2026

MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning

Authors: Haozhan Shen, Shilin Yan, Hongwei Xue, Shuaiqi Lu, Xiaojun Tang, Guannan Zhang, Tiancheng Zhao, Jianwei Yin

Organizations: Accio Team, Alibaba Group · Zhejiang University · ZJU-BJ

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal Large Language Models (MLLMs) are increasingly used to carry out visual workflows such as navigating GUIs, where the next step depends on verified visual compositional conditions (e.g., "if a permission dialog appears and the color of the interface is green, click Allow") and the process may branch or terminate early. Yet this capability remains under-evaluated: existing benchmarks focus on shallow-compositions or independent-constraints rather than deeply chained compositional conditionals. In this paper, we introduce MM-CondChain, a benchmark for visually grounded deep compositional reasoning. Each benchmark instance is organized as a multi-layer reasoning chain, where every layer contains a non-trivial compositional condition grounded in visual evidence and built from multiple objects, attributes, or relations. To answer correctly, an MLLM must perceive the image in detail, reason over multiple visual elements at each step, and follow the resulting execution path to the final outcome. To scalably construct such workflow-style data, we propose an agentic synthesis pipeline: a Planner orchestrates layer-by-layer generation of compositional conditions, while a Verifiable Programmatic Intermediate Representation (VPIR) ensures each layer's condition is mechanically verifiable. A Composer then assembles these verified layers into complete instructions. Using this pipeline, we construct benchmarks across three visual domains: natural images, data charts, and GUI trajectories. Experiments on a range of MLLMs show that even the strongest model attains only 53.33 Path F1, with sharp drops on hard negatives and as depth or predicate complexity grows, confirming that deep compositional reasoning remains a fundamental challenge.

Figures & tables

Figure 1: MM-CondChain targets visually grounded deep conditional reasoning beyond prior benchmarks. Top : existing benchmarks typically evaluate either shallow single-layer visual compositions or independent instruction constraints. Bottom left : MM-CondChain introduces nested, inter-layer conditional chains with rich intra-layer compositional predicates, where a minimally perturbed condition can create a hard negative that changes the execution path and causes early termination. Bottom right : experiments show that even advanced MLLMs achieve limited performance on this benchmark, highlighting visually grounded deep compositional reasoning as a fundamental challenge.
BenchmarkComposeNestedVisualHard Neg.Prog. Verif.Determ.Auto.
Visual Reasoning
SugarCrepe Hsieh et al. (2023)✓✗✓✓✗✓✓
Winoground Thrush et al. (2022)✓✗✓✓✗✓✗
ARO Yuksekgonul et al. (2023)✓✗✓✓✗✓✗
MMComposition Hua et al. (2024)✓✗✓✗✗✓✗
Instruction Following
Table 1: Comparison with existing benchmarks. Compose : intra-layer multi-attribute composition; Nested : inter-layer chained conditions; Visual : conditions grounded in visual input; Hard Neg. : contrastive pairs with minimal perturbation; Prog. Verif. : ground truth verified via code execution; Determ. : deterministic evaluation without LLM-as-judge; Auto. : automated data construction.
Figure 2: Overview of the MM-CondChain agentic synthesis pipeline. Given a multimodal input, the Planner iteratively extends a conditional chain: at each layer, structured facts are extracted, a VPIR predicate pair is generated and verified via code execution, and the logic is rendered into natural language. The Composer then compiles the verified chain into paired True-path and False-path instances for evaluation.
AspectNaturalChartGUI
InputSingle imageImage + metadataImage seq. + annotation
PreprocessingNoneCSV align + LLM repairCompleteness + CoAT parse
Fact FocusVisual attributesNumerical statsTemporal actions
Table 2: Domain-specific adaptations within the unified VPIR framework.
Figure 3: Top attribute frequencies in extracted facts and VPIR variables across domains. (a,c,e) show the top 20 attributes in extracted facts for the Natural, Chart, and GUI domains, respectively; (b,d,f) show the top 20 variables used in VPIR predicates for the corresponding domains.
Figure 4: Logic pattern composition of VPIR expressions. Left: overall distribution of high-level VPIR logic families. Middle: top-20 dominant concrete VPIR templates. Right: an example showing how a VPIR template is instantiated into executable predicates and natural-language conditions.
 NaturalChartGUIAvg
ModelTrueFalseF1TrueFalseF1TrueFalseF1F1
Open-Source MLLMs
Qwen3.5-0.8B33.172.264.2331.503.005.4833.951.863.524.41
GLM-4.6V-Flash83.929.5517.1481.915.5310.3687.530.531.059.52
InternVL3-8B65.338.2914.7247.508.5014.4263.665.319.7912.98
InternVL3.5-8B82.4110.3018.3176.0019.5031.0482.231.332.6117.32
Table 3: Main results on MM-CondChain across domains. All numbers are percentages (%). Path F1 is the harmonic mean of True- and False-path accuracy. Avg(F1) is the mean of the three domain F1 scores. Rows are sorted by Avg(F1) in ascending order within each category.
ModelD=2D=4D=6Δ2→6\Delta_{2\to 6}
Gemini-3-Flash70.6853.8547.19−- 33.2%
Qwen3-VL-Plus61.5152.5643.58−- 29.1%
GPT-4o-112031.3927.6722.23−- 29.2%
Table 4: Effect of chain depth and predicate complexity on Path F1 (%). Left : Performance degrades as chain depth increases, with ∼\sim 30% relative drop from D=2 to D=6. Right : Increasing intra-layer predicate complexity ( Simple vs. Complex ) causes 28–36% degradation at fixed depth.
More in this topic

Visually Grounded Reasoning

Visually grounded reasoning links visual evidence to multi-step conclusions and actions, aiming to make multimodal systems accurate, auditable, and robust.

See all 342 papers

Explore similar work

CardsList
May 25, 2026

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.

Longteng Guo, Yifan Wang, Pengkang Huo +4

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Visually Grounded ReasoningVisual ReasoningVLM ReasoningMultimodal Model EvaluationMultimodal UnderstandingReasoning for VLMs+1
AskPDF
Nov 21, 2025

VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning

Chain-of-Thought (CoT) prompting has proven remarkably effective for eliciting complex reasoning in large language models (LLMs). Yet, its potential in multimodal large language models (MLLMs) remains largely untapped, hindered by the absence of large-scale datasets that capture the rich, spatially grounded reasoning intrinsic to visual understanding. Existing visual-CoT resources are typically small, domain-specific, or lack the human-like stepwise structure necessary for compositional visual reasoning. In this paper, we introduce VisReason, a large-scale dataset designed to advance visual Chain-of-Thought reasoning. VisReason comprises 489K annotated examples spanning four diverse domains, each featuring multi-round, human-like rationales that guide MLLMs through interpretable visual reasoning steps. Building upon this, we curate VisReason-Pro, a 165K subset produced with a stronger expert-level GPT annotator, enriched with detailed reasoning traces and 3D spatial grounding via depth-informed annotations. Fine-tuning the state-of-the-art Qwen2.5-VL model on VisReason and VisReason-Pro yields substantial improvements in step-by-step visual reasoning accuracy, interpretability, and cross-benchmark generalization. These results demonstrate that VisReason equips MLLMs with more systematic and generalizable reasoning capabilities. We envision VisReason as a cornerstone for cultivating human-like visual reasoning, paving the way toward the next generation of multimodal intelligence.

Lingxiao Li, Yifan Wang, Xinyan Gao +3

Boston University, Boston, MA, USA · Stony Brook University, Stony Brook, NY, USA · MMLab, The Chinese University of Hong Kong, Hong Kong SAR, China

Visual Spatial ReasoningVisual ReasoningMultimodal Large Language ModelsMultimodal CoT ReasoningVisually Grounded ReasoningVisual CoT
AskPDF
Jun 5, 2026

VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?

When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence? Correct answers can coexist with flawed reasoning, making accuracy alone an incomplete test of grounding. We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question perturbation pairs across cardinality, attribute, spatial, and logic tasks. Each pair keeps the question fixed but minimally changes the evidence so the gold answer deterministically flips. We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images. Together, these metrics show that paired correctness and evidence dependence are related but distinct: capable models can still fail to update after task-critical visual changes, and collapse becomes more severe for some models when the edited image follows an earlier answer in a sequential setting. Further details are available on our project page: https://didizhu-judy.github.io/VisualFLIP/

Didi Zhu, Changrui Chen, Stefanos Zafeiriou +1

Imperial College London

VLM EvaluationMultimodal GroundingVLM RobustnessCounterfactual Evaluation of VLMsMultimodal ReasoningVisually Grounded Reasoning+3
AskPDF