cs.CVSep 1, 2026

Part Grounding, Not Action Knowledge: Locating the Bottleneck in VLM Affordance Prediction

Authors: Sarthak Sattigeri

Organizations: Manipal University Jaipur

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Benchmarks agree that vision-language models reason poorly about low-level manipulation, but an aggregate accuracy score does not say which step fails. We separate two steps that affordance questions conflate: identifying which part of an object to act on, and knowing what action that part requires. Across 19 articulated objects we asked eight models, spanning three developers, what motion a robot should apply. Under an open prompt, push was produced once in 64 evaluations where it was correct, despite being correct for 8 of 19 objects and appearing in the offered label set every time. Inspecting the outputs showed why: models described a different part than the one being scored, e.g. explaining how to pick up a camera rather than press its button. Naming the target part raises action accuracy by 0.32 to 0.63 for every model, from 0.158-0.474 to 0.684-0.947, and push recall from 0-1/8 to 7-8/8. No model beats a constant answer that ignores the image under the open prompt; once the part is named, all eight do. Asked to describe the same part in free prose with no label set, models produce pressing language for 6 to 8 of 8. These results are hard to reconcile with missing action knowledge, and instead point to part grounding as the dominant bottleneck, a pattern that holds across all three model families and does not diminish with capability. Naming the part supplies the grounding variable, so this bounds what a perfect part detector would offer rather than demonstrating a general model of mechanics. Two supporting results agree: on real photographs only three of eight models localize grasp points better than a constant baseline, and on rendered objects none do. We also document two measurement errors of our own, a threshold that let a constant baseline score 0.929 and a labelling rule wrong on 4 of 19 objects, both caught only by testing our numbers against trivial alternatives.

Figures & tables

  Action accuracypush recall (of 8)
ModelFamilyACΔ\DeltaACB
Qwen3-VL-8BAlibaba0.1580.737+0.58+0.58087
gpt-4o-miniOpenAI0.2630.842+0.58+0.58088
gpt-5-miniOpenAI0.3160.684+0.37+0.37088
gpt-5OpenAI0.4740.947+0.47+0.47188
Claude Haiku 4.5Anthropic0.4740.789+0.32+0.32076
Table 1: Prompt A lets the model choose the part; C names it; B names it and asks for prose. 19 objects per cell, 456 calls total, no failures. Under A no model reaches the constant baseline; under C all eight exceed it.
 Photographs ( n=28n=28 )Renders ( n=19n=19 )
ModelHit rateMean dist.Hit rateMean dist.
Qwen3-VL-8B1.0000.03560.6840.1744
gpt-4o-mini0.7860.13670.7370.2069
gpt-5-mini0.9290.13270.3680.2693
gpt-51.0000.06310.4210.2363
Claude Haiku 4.50.7500.16200.3680.2485
Table 2: Spatial localization, all 47 objects per model. Hit rate uses a 0.15 threshold on the image diagonal; mean distance is normalized Euclidean error, lower is better. The constant baseline ignores the image. Bold marks the three models that clearly beat it on photographs.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ObjectIDPart categoryJointLimitsAxisLabel
Safe101613hinge_doorhinge0, -159.12(0.00, -1.00, 0.00)pull
Camera102505slider_buttonslider0, 0.008(-0.99, 0.13, 0.00)push
Remote101015slider_buttonslider0, 0.01(0.00, 0.00, -1.00)push
Refrigerator11178hinge_doorhinge0, -180(0.00, -1.00, 0.00)pull
Printer104016slider_buttonslider0, 0.004(-0.00, -0.99, -0.15)push
WashingMachine103425hinge_doorhinge0, -97.92(0.00, 1.00, 0.00)pull
Appendix
Table 3: All 19 articulated objects with the mechanics their labels derive from.
More in this topic

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

Explore similar work

CardsList
Sep 10, 2026

GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures

A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language models, with no model outperforming a constant baseline until the part was named. However, naming the part supplies information that a real system must infer, confounding visual grounding, mechanical reasoning, and category-to-action association. We introduce GroundBench, a diagnostic benchmark that separates these explanations through six branch-and-merge conditions, each adding a controlled information bundle, and a counterfactual re-ask targeting a real alternate part visible in the same image. Across three OpenAI models and 1,068 predictions, supplying the target region without its identity leaves action accuracy at or below the 0.53 majority baseline (0.26, 0.26, and 0.53), although the models largely reproduce the supplied region. Supplying identity without location instead yields 0.74, 0.68, and 0.68. Every above-baseline gain in this curated set occurs where the supplied part category itself determines the action. A no-vision control leaves GPT-5's scores unchanged or improved, providing evidence consistent with substantial category-to-action association. GPT-4o mini declines on one condition, so this interpretation is not universal. Adding joint type and motion axis does not improve accuracy across six model-stratum comparisons. On 74 counterfactual pairs from 32 objects, GPT-5 achieves 0.86 pair-weighted compliance with a 0.07 shortcut rate but fails all observed push-to-lift-vertical cases. GroundBench identifies which supplied information changes affordance behavior and tests whether apparently grounded performance can be reproduced through textual shortcuts.

Sarthak Sattigeri

Manipal University Jaipur

Vision-Language ModelsVision-Language GroundingRobot Manipulation BenchmarksCounterfactual Evaluation of VLMs
AskPDF
Oct 7, 2026

Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding

Vision-Language-Action models are designed to generalise across environments and task descriptions, raising the question of whether their action generation actually depends on the language instruction, or whether they largely rely on visual cues and superficial correlations. Robustness to variance in the visual and linguistic observation space is critical for real-world deployment, yet VLAs lack explicit grounding modules and instead rely on the intrinsic language grounding capabilities of their Vision-Language model backbones. For this reason, we conduct a controlled mechanistic interpretability study on the language grounding capabilities of two state-of-the-art Vision-Language-Action models, π0.5π_{0.5} and GR00T N1.7, by applying activation and attribution patching to the residual stream of the action generation modules. We systematically corrupt the task instruction of input samples of the LIBERO benchmark following five strategies: synonym replacement, semantic scaling, directional corruption, random object substitution, and empty string. Our experiments find that both models are comparatively insensitive to abstract rephrasing and to referencing non-existent objects, but react strongly to empty task descriptions and, especially, to directional language. During action generation, this sensitivity is concentrated in different loci for each model: mainly in the early, periodic cross-attention layers for GR00T N1.7, versus distributed across the earliest and selected later layers for π0.5π_{0.5}. For GR00T N1.7, directional perturbations drive some of the largest causal effects while leaving the internal representational geometry comparatively unchanged, a dissociation we do not observe clearly for π0.5π_{0.5}. Finally, the reliability of attribution patching is model-dependent: it closely tracks activation patching for GR00T N1.7 but not for π0.5π_{0.5}.

Theodor Wulff, Angelo Cangelosi

Department of Computer Science The University of Manchester Manchester, United Kingdom

Vision-Language-Action ModelsRobustness of VLA Models
AskPDF
Jun 28, 2026

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

Vision-Language-Action (VLA) systems, built on pretrained vision-language models (VLMs), have shown rapidly improving performance on robot manipulation benchmarks. These gains are commonly interpreted as evidence that semantic representations learned from internet-scale data transfer to physical execution generalization. This position paper argues that the assumption underlying this interpretation -- that semantic generalization is sufficient to support physical action decisions -- has not been independently verified and cannot be tested under current evaluation protocols. We support this claim by decomposing VLA policies into semantic mapping and physical action decision, and showing that task success rate -- the dominant evaluation metric -- cannot distinguish between these two sources of capability. As a result, improvements in benchmark performance are consistent with multiple competing explanations, including semantic matching, distributional overlap, and genuine physical generalization. We further argue that this identifiability gap has been reinforced through narrative drift, whereby successive systems inherit and strengthen prior interpretations of performance gains without isolating the underlying causal mechanism. To address this limitation, we propose a research direction based on evaluation designs that introduce controlled variation to separately measure semantic and physical generalization. Such designs make it possible to causally attribute performance without requiring access to model internals, and to empirically assess the role of VLM backbones as semantic interfaces rather than implicit sources of physical competence. Our goal is not to refute the role of VLMs in robotics, but to clarify the conditions under which claims of physical generalization can be meaningfully evaluated.

Taozhao Chen, Ian Manchester, Huaming Chen

School of Electrical and Computer Engineering The University of Sydney Sydney, NSW, Australia · School of Aerospace, Mechanical and Mechatronic Engineering The University of Sydney Sydney, NSW, Australia

VLM EvaluationPhysical ReasoningGeneralization in Robotic ManipulationRoboticsVision-Language-Action ModelsRobot Manipulation Benchmarks+1
AskPDF