What does it mean for a visual system to truly understand affordance? We argue that this understanding hinges on two complementary capacities: geometric perception, which identifies the structural parts of objects that enable interaction, and interaction perception, which models how an agent's actions engage with those parts. To test this hypothesis, we conduct a systematic probing of Visual Foundation Models (VFMs). We find that models like DINO inherently encode part-level geometric structures, while generative models like Flux contain rich, verb-conditioned spatial attention maps that serve as implicit interaction priors. Crucially, we demonstrate that these two dimensions are not merely correlated but are composable elements of affordance. By simply fusing DINO's geometric prototypes with Flux's interaction maps in a training-free and zero-shot manner, we achieve affordance estimation competitive with weakly-supervised methods. This final fusion experiment confirms that geometric and interaction perception are the fundamental building blocks of affordance understanding in VFMs, providing a mechanistic account of how perception grounds action.
Figures & tables
Figure 1: From geometry to interaction: uncovering affordance perception in visual foundation models. Affordance understanding emerges from the fusion of geometric structure and generative interaction priors. (a) Models with stronger geometric awareness yield richer part-level representations. (b) Generative models reveal verb-conditioned attention that naturally localizes interaction regions without supervision.
Model
Architecture
Supervision
DINO [ 3 ]
ViT-B/16
SSL
DINOv2 [ 31 ]
ViT-B/14
SSL
DINOv3 [ 35 ]
ViT-B/16
SSL
CLIP [ 16 ]
ViT-B/16
VLM
SigLIP [ 45 ]
ViT-B/16
VLM
SAM [ 19 ]
ViT-B/16
Segmentation
Table 1: Probed Visual Models. We used supervised classification from Probing3D [ 12 ] to explore and analyze these models.
Figure 2: Learning Affordances: From Geometry to Interaction. Fully-supervised methods learn geometric structures from pixel masks; weakly-supervised ones infer interactions from human-object imagery.
Figure 3: Distinct geometric representations across Visual Foundation Models. We project complex scenes into the PCA subspace of a reference object (a mug) to visualize and compare geometric representations. DINOv3 produces part-level structures that generalize across scenes and materials, as confirmed by cosine similarity responses from a mug-handle patch, while CLIP, SAM, and Stable Diffusion emphasize semantics, edges, or smooth surface continuity instead.
Figure 4: Geometric awareness predicts affordance perception ability. We evaluate visual models under different supervision paradigms using linear probing on the UMD dataset [ 28 , 10 ] . Models with stronger geometric awareness [ 12 ] —achieve higher mIoU in affordance segmentation, while others improve notably when augmented with depth or normal cues.
Figure 5: Stable part-level structure in DINO representations. PCA of DINOv3 embeddings reveals consistent part-wise activations across the first three components, indicating interpretable geometric decomposition aligned with object functionality.
Figure 6: Semantic assimilation of geometry in DINO. A mug-handle patch(Fig. 3 left top) activates all ring-like shapes regardless of material or color (left), confirming that DINO’s local correspondence arises from genuine geometric similarity rather than shared semantics. When the full scene is shown (right), these responses vanish, indicating that global object semantics suppress purely geometric alignment.
Figure 7: Verb-conditioned attention in generative models. Cross-attention maps show that verbs such as cut, drink, and support consistently localize to contact regions, revealing implicit interaction priors in Flux.
Figure 8: Qualitative visualization of verb-conditioned interaction maps extracted by Flux Kontext.
Figure 9: Verb-conditioned localization across models. The first and third lines are used to query verbs, and the second and fourth lines are object nouns corresponding to verbs.
Figure 11: Fusion pipeline of geometry and interaction. DINOv3 captures geometric parts via PCA, Flux Kontext encodes verb-conditioned attention, and Normalized Scanpath Saliency links them to align structure with interaction semantics.
Method
Supervision
KLD ↓
SIM ↑
NSS ↑
AffordanceLLM
Fully-supervised
1.463
0.377
1.070
Cross-View-AG
Weakly-supervised
1.787
0.285
0.829
LOCATE
Weakly-supervised
1.405
0.372
1.157
Affordance-R1
Post-Training
9.730
0.360
0.980
Ours(Only Interaction)
Training-Free
1.825
0.271
1.050
Ours(Interaction × Geometry)
Training-Free
1.493
0.326
1.090
Table 2: Quantitative comparison on AGD20K [ 27 ] . We evaluate our zero-shot, training-free approach against both fully- and weakly-supervised baselines. Using only verb attention from Flux Kontext achieves competitive scores.
Figure 12: Affordance segmentation from geometry–interaction fusion. The fused model localizes plausible action regions (e.g., handles, blades, rims) on the UMD dataset—without affordance-specific training.
Affordance grounding aims to localize the functional region for interaction, such as the handle to grasp or the button to press, rather than the whole object. This makes it more challenging than generic visual grounding because the target region is smaller, more ambiguous, and more dependent on task context, especially for compact vision-language models (VLMs) used in embodied settings. Recent sequence-level supervision and reinforcement learning improve coordinate prediction quality, yet compact autoregressive VLMs still lack reliable affordance-aware visual focus before coordinate generation: the model can produce better coordinate tokens while its cross-modal attention remains diffuse and weakly anchored to the true affordance evidence. To address it, we propose SpatialAfford, a two-stage framework that first aligns attention to the ground-truth affordance region through Spatial Attention Alignment (SAA), then refines coordinate prediction with Spatial-Aware GRPO. By explicitly teaching the model where to look before optimizing where to ground, SpatialAfford turns affordance grounding from a purely output-constrained objective into attention-grounded spatial reasoning. Across ShareRobot-Bench, ReasonAff, and PartAfford, SpatialAfford consistently improves affordance grounding, with a compact 4B model outperforming stronger 7B+ baselines.
Yufei Zhang, Chenlu Zhan, Donghui Sun +2
Zhejiang University · Vivo Mobile Communication Co., Ltd
Recent advances in Vision-Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation. However, the visual representations of most VLA models are often dominated by global object appearance and struggle to focus on task-relevant functional interaction regions, which limits their robustness in unstructured environments. Existing affordance-based methods typically rely on explicit mask injection or external perception modules, requiring additional annotations while introducing cascading perception errors and inference overhead. To address these limitations, we propose AffordVLA, an affordance-enhanced VLA framework that internalizes manipulation-centric affordance perception into VLA visual representations through implicit representation alignment. Specifically, we construct a zero-shot affordance teacher to extract task-conditioned affordance visual representations from RGB observations and language instructions. AffordVLA aligns the intermediate visual representations of the VLA with the affordance visual representations extracted by the teacher, thereby implicitly injecting manipulation-centric affordance perception into VLA visual representations and improving action accuracy. Extensive simulation and real-world experiments demonstrate that AffordVLA and its affordance teacher achieve state-of-the-art performance and outperform strong baselines. Ablation analyses show that AffordVLA effectively reshapes VLA visual representations while preserving inference efficiency, leading to improved manipulation success rates and training efficiency.
Weijie Kong, Zhian Su, Wei Yu +1
Grasp Lab, School of Mechanical Engineering of Zhejiang University, Hangzhou 310027, China · Torch Kernel Co., Ltd., Hangzhou, 310000, P. R. China
Functional affordance grounding requires more than recognizing an object: an agent must localize the specific region that supports an interaction, such as the handle to pull or the button to press. This is difficult for training-free vision-language pipelines because actionable regions are often small, visually ambiguous, and repeated across multiple same-category instances in a scene. We propose AFFORDMEM, a framework that grounds 3D functional affordances by remembering geometry at two levels. The first is cross-scene affordance memory: the agent maintains a category-level memory bank of RGB images with affordance regions rendered as overlays, and recalls the most informative examples at query time to guide a frozen VLM toward small operable subregions that text-only prompting consistently misses. The second is in-scene spatial memory: as the agent processes the scene, it organizes candidate instances and their 3D spatial relations into a structured scene graph, enabling the language model to resolve references over distant or currently unobserved candidates such as "the second handle from the top." AFFORDMEM requires no model fine-tuning and no target-scene annotation, using a reusable memory bank built from source scenes. On SceneFun3D, our method improves AP50 over the prior training-free state of the art by 3.23 on Split 0 and 3.7 on Split 1. Ablation studies support complementary benefits: cross-scene affordance memory improves fine-grained localization, while in-scene spatial memory provides the larger gain on spatially qualified queries. The project homepage is available at the project page.