Probing and Bridging Geometry-Interaction Cues for Affordance Reasoning in Vision Foundation Models

Qing Zhang; Xuesong Li; Jing Zhang

cs.CVFeb 24, 2026

Probing and Bridging Geometry-Interaction Cues for Affordance Reasoning in Vision Foundation Models

Authors: Qing Zhang, Xuesong Li, Jing Zhang

Organizations: 1The Australian National University · 2CSIRO

Abstract

What does it mean for a visual system to truly understand affordance? We argue that this understanding hinges on two complementary capacities: geometric perception, which identifies the structural parts of objects that enable interaction, and interaction perception, which models how an agent's actions engage with those parts. To test this hypothesis, we conduct a systematic probing of Visual Foundation Models (VFMs). We find that models like DINO inherently encode part-level geometric structures, while generative models like Flux contain rich, verb-conditioned spatial attention maps that serve as implicit interaction priors. Crucially, we demonstrate that these two dimensions are not merely correlated but are composable elements of affordance. By simply fusing DINO's geometric prototypes with Flux's interaction maps in a training-free and zero-shot manner, we achieve affordance estimation competitive with weakly-supervised methods. This final fusion experiment confirms that geometric and interaction perception are the fundamental building blocks of affordance understanding in VFMs, providing a mechanistic account of how perception grounds action.

Figures & tables

Figure 1: From geometry to interaction: uncovering affordance perception in visual foundation models. Affordance understanding emerges from the fusion of geometric structure and generative interaction priors. (a) Models with stronger geometric awareness yield richer part-level representations. (b) Generative models reveal verb-conditioned attention that naturally localizes interaction regions without supervision.

Model	Architecture	Supervision
DINO [ 3 ]	ViT-B/16	SSL
DINOv2 [ 31 ]	ViT-B/14	SSL
DINOv3 [ 35 ]	ViT-B/16	SSL
CLIP [ 16 ]	ViT-B/16	VLM
SigLIP [ 45 ]	ViT-B/16	VLM
SAM [ 19 ]	ViT-B/16	Segmentation

Table 1: Probed Visual Models. We used supervised classification from Probing3D [ 12 ] to explore and analyze these models.

Figure 2: Learning Affordances: From Geometry to Interaction. Fully-supervised methods learn geometric structures from pixel masks; weakly-supervised ones infer interactions from human-object imagery.

Figure 3: Distinct geometric representations across Visual Foundation Models. We project complex scenes into the PCA subspace of a reference object (a mug) to visualize and compare geometric representations. DINOv3 produces part-level structures that generalize across scenes and materials, as confirmed by cosine similarity responses from a mug-handle patch, while CLIP, SAM, and Stable Diffusion emphasize semantics, edges, or smooth surface continuity instead.

Figure 4: Geometric awareness predicts affordance perception ability. We evaluate visual models under different supervision paradigms using linear probing on the UMD dataset [ 28 , 10 ] . Models with stronger geometric awareness [ 12 ] —achieve higher mIoU in affordance segmentation, while others improve notably when augmented with depth or normal cues.

Figure 5: Stable part-level structure in DINO representations. PCA of DINOv3 embeddings reveals consistent part-wise activations across the first three components, indicating interpretable geometric decomposition aligned with object functionality.

Figure 6: Semantic assimilation of geometry in DINO. A mug-handle patch(Fig. 3 left top) activates all ring-like shapes regardless of material or color (left), confirming that DINO’s local correspondence arises from genuine geometric similarity rather than shared semantics. When the full scene is shown (right), these responses vanish, indicating that global object semantics suppress purely geometric alignment.

Figure 7: Verb-conditioned attention in generative models. Cross-attention maps show that verbs such as cut, drink, and support consistently localize to contact regions, revealing implicit interaction priors in Flux.

Figure 8: Qualitative visualization of verb-conditioned interaction maps extracted by Flux Kontext.

Figure 9: Verb-conditioned localization across models. The first and third lines are used to query verbs, and the second and fourth lines are object nouns corresponding to verbs.

Figure 10: Verb attention persists despite generation failure.

Figure 11: Fusion pipeline of geometry and interaction. DINOv3 captures geometric parts via PCA, Flux Kontext encodes verb-conditioned attention, and Normalized Scanpath Saliency links them to align structure with interaction semantics.

Method	Supervision	KLD $\downarrow$	SIM $\uparrow$	NSS $\uparrow$
AffordanceLLM	Fully-supervised	$1.463$	$0.377$	$1.070$
Cross-View-AG	Weakly-supervised	$1.787$	$0.285$	$0.829$
LOCATE	Weakly-supervised	$1.405$	$0.372$	$1.157$
Affordance-R1	Post-Training	$9.730$	$0.360$	$0.980$
Ours(Only Interaction)	Training-Free	$1.825$	$0.271$	$1.050$
Ours(Interaction $\times$ Geometry)	Training-Free	$1.493$	$0.326$	$1.090$

Table 2: Quantitative comparison on AGD20K [ 27 ] . We evaluate our zero-shot, training-free approach against both fully- and weakly-supervised baselines. Using only verb attention from Flux Kontext achieves competitive scores.

Figure 12: Affordance segmentation from geometry–interaction fusion. The fused model localizes plausible action regions (e.g., handles, blades, rims) on the UMD dataset—without affordance-specific training.

Vision Foundation Models

Vision foundation models are large pretrained visual systems adapted to recognition, segmentation, geometry, and video tasks.

See all 139 papers

SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance
Aug 1, 2026Yufei Zhang, Chenlu Zhan, Donghui Sun +2Vision-Language Grounding VLMs for Robotics
Zhejiang University · Vivo Mobile Communication Co., Ltd
AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignment
May 17, 2026Weijie Kong, Zhian Su, Wei Yu +1Language-Conditioned Robot Manipulation Vision-Language-Action Models
Grasp Lab, School of Mechanical Engineering of Zhejiang University, Hangzhou 310027, China · Torch Kernel Co., Ltd., Hangzhou, 310000, P. R. China
Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances
May 12, 2026Qirui Wang, Jingyi He, Yining Pan +23D Affordance Grounding Visually Grounded Reasoning
TUM · A*STAR

Aug 1, 2026J/K move · Enter open · S save

SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance

Yufei Zhang, Chenlu Zhan, Donghui Sun, Xiaoxin Chen+1

Zhejiang University · Vivo Mobile Communication Co., Ltd

Vision-Language Grounding VLMs for Robotics

Open paper AskPDF arXiv

Affordance grounding aims to localize the functional region for interaction, such as the handle to grasp or the button to press, rather than the whole object. This makes it more challenging than generic visual grounding because the target region is smaller, more ambiguous, and more dependent on task context, especially for compact vision-language models (VLMs) used in embodied settings. Recent sequence-level supervision and reinforcement learning improve coordinate prediction quality, yet compact autoregressive VLMs still lack reliable affordance-aware visual focus before coordinate generation: the model can produce better coordinate tokens while its cross-modal attention remains diffuse and weakly anchored to the true affordance evidence. To address it, we propose SpatialAfford, a two-stage framework that first aligns attention to the ground-truth affordance region through Spatial Attention Alignment (SAA), then refines coordinate prediction with Spatial-Aware GRPO. By explicitly teaching the model where to look before optimizing where to ground, SpatialAfford turns affordance grounding from a purely output-constrained objective into attention-grounded spatial reasoning. Across ShareRobot-Bench, ReasonAff, and PartAfford, SpatialAfford consistently improves affordance grounding, with a compact 4B model outperforming stronger 7B+ baselines.

Abstract

Vision Foundation Models

Explore similar work

SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance

AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignment

Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances