AffordJEV: Affordance Grounding by Confirming Candidate Regions

Jincheng Li; Yutian Zhang; Jianyu Zhang; Xinshun Wang; Mengyuan Liu

cs.RONov 5, 2026

AffordJEV: Affordance Grounding by Confirming Candidate Regions

Authors: Jincheng Li, Yutian Zhang, Jianyu Zhang, Xinshun Wang, Mengyuan Liu

Organizations: Peking University

Abstract

Affordance grounding asks which visible part of an object performs a requested operation, a basic capability for embodied manipulation and human--robot interaction. Most existing methods follow a System Two paradigm: a multimodal language model first writes a chain of thought and then turns its conclusion into a box or points for a segmenter. This incurs considerable inference cost, gives the semantic decision little direct control over the extent of the final region, and makes mistakes hard to correct. Yet locating an affordance is essentially a perceptual judgment that need not be argued out in words. We therefore propose, in which frozen segmenters propose candidate regions and a System One model decides directly which of them perform the operation, generating no text and reading each decision from a single forward pass. Our contributions are twofold. (1) An inference framework that confirms candidate regions level by level, from object to part to sub-part, with evidence passed between the levels in both directions so that the levels correct one another. (2) A System One model trained for this framework: existing System One models are built for general-purpose decisions, judge one candidate per request and are ill-suited to the framework's region questions, so ours learns directly from the requests the framework itself produces and judges all candidates of a level at once. Trained only on InstructPart and evaluated with definition-aligned operation words, outperforms chain-of-thought methods in domain and in zero-shot transfer to UMD and IIT-AFF, is better overall than existing System One models run in the same framework, and needs fewer judge requests. To our knowledge, is the first approach to bring System One models to affordance grounding.

Figures & tables

Method	gIoU $\uparrow$	cIoU $\uparrow$	P ${}_{50\text{--}95}{\uparrow}$	P ${}_{50}{\uparrow}$
Affordance-R1 [ 42 ]	59.49	60.93	46.51	64.52
VisionReasoner-7B [ 26 ]	47.26	42.01	31.94	48.92
Valen-4B, released [ 40 ]	51.54	51.35	36.67	54.30
H2O-Lightning-4B, released [ 12 ]	46.96	48.10	30.91	49.46
JevAny-4B-LoRA, released [ 37 ]	35.10	33.91	17.74	32.80
Valen, retrained	65.93	70.61	53.66	72.04

Table 1: InstructPart (%), in domain. The alignment of Sec. 4.1 touches no InstructPart query, so these are both protocols at once. Released are the published judges in our pipeline; retrained are their native heads trained on our supervision.

Method	gIoU $\uparrow$	cIoU $\uparrow$	P ${}_{50\text{--}95}{\uparrow}$	P ${}_{50}{\uparrow}$
Affordance-R1 [ 42 ]	63.89	62.43	43.00	64.29
VisionReasoner-7B [ 26 ]	54.79	55.19	22.14	48.57
Valen-4B, released [ 40 ]	74.97	75.49	58.71	90.00
H2O-Lightning-4B, released [ 12 ]	68.36	71.87	49.14	84.29
JevAny-4B-LoRA, released [ 37 ]	59.09	46.03	35.86	62.86
Valen, retrained	71.46	72.88	52.86	81.43

Table 2: UMD novel/test, zero-shot (%), under the aligned operation words of Sec. 4.1 . The alignment is applied identically to every method.

Method	gIoU $\uparrow$	cIoU $\uparrow$	P ${}_{50\text{--}95}{\uparrow}$	P ${}_{50}{\uparrow}$
Affordance-R1 [ 42 ]	60.94	68.55	42.78	63.33
VisionReasoner-7B [ 26 ]	53.27	65.81	32.78	50.00
Valen-4B, released [ 40 ]	57.16	66.78	38.00	57.78
H2O-Lightning-4B, released [ 12 ]	54.02	60.82	33.22	57.78
JevAny-4B-LoRA, released [ 37 ]	55.98	63.38	34.22	62.22
Valen, retrained	66.43	73.05	49.22	75.56

Table 3: IIT-AFF test, zero-shot (%), aligned operation words. cIoU pools pixels over queries and is the one column we do not win; Sec. 4.5 identifies the queries responsible.

Panel of Figure 3 — Figure 3: Qualitative comparison against the two retrained baselines, all under the aligned protocol. Prediction purple, ground-truth contour green; numbers are IoU. The six left columns are the largest margins in our favour on each set, one query per object and operation. The last column is the largest margin against us: a second suitcase whose telescopic handle our judge rejects entirely while both baselines accept it.

Framework
Variant	gIoU $\uparrow$	cIoU $\uparrow$	P ${}_{50\text{--}95}{\uparrow}$	P ${}_{50}{\uparrow}$
AffordJEV	70.44	72.84	59.52	78.49
A1 Flat judgement	66.76	70.32	55.05	72.04
A2 w/o evidence passing	65.69	68.23	54.30	70.97
Judge architecture
B1 Drawn region markers	65.57	73.03	52.85	70.97

Table 4: Ablations on InstructPart (%), grouped by framework, judge architecture and training. All rows use

T=1

and one training seed. A1–A2 change inference only; B1–C3(b) are retrained. Bold marks the best value in each column.

Word	Test sets mark	InstructPart marks	Aligned to
grasp	tool handle	vessel body	hold
wrap grasp	vessel body	—	grasp
contain	hollow interior or opening	body, interior or opening	contain in the hollow interior or opening
support	UMD: shovel blade	furniture seat or surface	dig
support	IIT-AFF: spatula blade	furniture seat or surface	support food on the flat working blade

Table 5: Definition alignment on UMD and IIT-AFF.

Robotic Affordance Inference

See all 29 papers

Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation
Jun 2, 2026Litao Liu, Yifan Han, Pengfei Yi +9Language-Conditioned Robot Manipulation Robotic Manipulation
Department of Computer Science, Rutgers University-New Brunswick · The Hong Kong University of Science and Technology (GZ) · Shanghai AI Laboratory
Zero-shot 2D Grounding with Novel Affordance Types
Aug 9, 2026Haomeng Zhang, Raymond A. YehZero-Shot Learning Visual Grounding
Department of Computer Science, Purdue University
Part Grounding, Not Action Knowledge: Locating the Bottleneck in VLM Affordance Prediction
Sep 1, 2026Sarthak SattigeriVision-Language Grounding VLM Evaluation
Manipal University Jaipur

Jun 2, 2026J/K move · Enter open · S save

Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation

Litao Liu, Yifan Han, Pengfei Yi, Wenbo Yu+8

Department of Computer Science, Rutgers University-New Brunswick · The Hong Kong University of Science and Technology (GZ) · Shanghai AI Laboratory

Language-Conditioned Robot Manipulation Robotic Manipulation Robot Manipulation Benchmarks Affordance-Based Robotic Manipulation

Open paper AskPDF arXiv

Task-conditioned manipulation requires grounding instructions to task-relevant functional parts rather than object categories. This setting is scene-dependent and often one-to-many in cluttered scenes: the same object may afford different interactions across tasks, while a single task may correspond to either one functional region or multiple valid functional regions, depending on the scene layout. Existing affordance datasets and benchmarks remain misaligned with this setting, as they typically focus on grasping or object-level affordances, rely on synthetic scenes, or assume a single instruction-region correspondence. We present Affordance2Action (A2A), a benchmark-centered learning framework for scene-level, task-conditioned part affordance grounding. At its core is A2A-Bench, a manipulation-oriented benchmark that covers both single-region and multi-region instruction correspondences in everyday scenes, with the latter highlighting the ambiguity and diversity of affordance grounding in realistic multi-object environments. To construct it at scale, we build A2A-AffordGen, an agent-assisted annotation pipeline that combines language-model filtering, interactive part segmentation, instance-level mask-out refinement, task-reasoning instruction generation, and human verification. A2A-Bench's supervision further supports diverse downstream applications, with real-time affordance grounding and affordance-conditioned manipulation policies as two representative examples. Experiments show that A2A exposes substantial gaps in generic segmentation, VLM-based grounding, and affordance distillation baselines, while improving task-level localization and providing useful spatial priors for downstream manipulation.

Abstract

Robotic Affordance Inference

Explore similar work

Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation

Zero-shot 2D Grounding with Novel Affordance Types

Part Grounding, Not Action Knowledge: Locating the Bottleneck in VLM Affordance Prediction