CRAFT-E: An Interpretable and Editable Neuro-Symbolic Framework for Affordance-Grounded Active Object Retrieval

Zhou Chen; Joe Lin; Carson Bulgin; Sathyanarayanan N. Aakur

cs.RODec 3, 2025

CRAFT-E: An Interpretable and Editable Neuro-Symbolic Framework for Affordance-Grounded Active Object Retrieval

Authors: Zhou Chen, Joe Lin, Carson Bulgin, Sathyanarayanan N. Aakur

Organizations: CSSE Department, Auburn University, Auburn, 36849, AL, USA.

Abstract

Assistive robots operating in unstructured environments must reason not only about what objects are, but how they can be used, including non-canonical and improvised uses. This requires grounding language-based action queries to objects that afford the requested function, are visually present, and can be physically retrieved. Existing approaches often rely on black-box models, fixed affordance labels, or direct object predictions, limiting transparency and controllability. We introduce CRAFT-E, a modular neuro-symbolic framework that composes an LLM-derived verb-property-object knowledge graph with CLIP-based visual alignment and energy-based grasp reasoning. The system exposes interpretable grounding paths, incorporates grasp feasibility into affordance inference, and supports targeted graph edits without retraining. We also construct a benchmark dataset with unified annotations for verb-object compatibility, segmentation, and grasp candidates, and deploy the full pipeline on a physical robot. Across static scenes, real-world trials involving 20 verbs and 39 objects, ImageNet-based functional retrieval, and COCO-Tasks transfer, CRAFT-E provides a competitive training-free prior for affordance-grounded retrieval while exposing component-level diagnostics and graph-level interventions. Under staged and full-pipeline perception, CRAFT-E matches strong flat LLM priors in several settings while adding failure attribution and editable reasoning paths. By coupling symbolic reasoning with embodied perception, CRAFT-E offers an inspectable and customizable framework for affordance-grounded object retrieval.

Figures & tables

Figure 1: Overview. Given a scene and verb query, CRAFT-E predicts graspable candidate regions, reasons over an affordance-centric verb subgraph, and uses frozen vision-language alignment to select a functionally appropriate and retrievable object.

Figure 2: LLM-derived affordance knowledge (left) versus ConceptNet subgraphs (right). CRAFT-E links verbs to objects through interpretable functional properties, while ConceptNet relies on general-purpose relations such as RelatedTo , which can yield noisy paths with limited explanatory structure.

Grasp Model	Affordance Prior	3D (%)	Detect (%)	Grasp Rate (%)
GraspNet	Gemini	83.89	56.67	44.44 [32.2, 57.2]
	GPT-4o	82.22	52.22	46.11 [33.3, 58.9]
	CRAFT	86.11	37.78	33.33 [25.0, 43.3]
	CRAFT-E	85.00	52.22	46.67 [33.9, 60.0]
KpNet	Gemini	72.78	45.00	23.89 [17.2, 30.6]
	GPT-4o	78.89	45.00	33.33 [27.8, 38.9]

Table 1: Real-world evaluation of 3D grasp feasibility, detection accuracy, and grasp execution success. Only

E_{\mathrm{aff}}

is varied; the retrieval scaffold is fixed. Grasp Rate includes 95% paired-bootstrap confidence intervals over 20 verb-object pairs.

Setting	Grasp	ROI	Affordance Prior	Accuracy (%)
Perfect Grasp & Detect	GT	GT	Gemini	59.79
			GPT-4o	62.89
			CRAFT	58.76
			CRAFT-E	64.95
			CRAFT-E+GPT-4o	52.58
Perfect Grasp	GT	SAMv2	Gemini	56.70

Table 2: Static evaluation accuracy under perfect grounding, perfect grasping, and full pipeline settings. Only

E_{\mathrm{aff}}

is varied; the retrieval scaffold is fixed. CRAFT-E+GPT-4o uses GPT-4o to derive the LLM-KG.

Approach	Accuracy (%)	MRR (%)	nDCG (%)
Object-Aware Oracle	100.00	69.90	76.80
Affordance-Aware Oracle	73.80	68.70	82.70
CLIP+verb prompting	38.00	54.80	56.10
ResNet-RNN [ 27 , 28 ]	60.20	65.70	76.50
ALGO [ 21 ]	42.52	52.38	53.33
Gemini-2.0-Flash [ 2 ]	42.80	57.50	59.70

Table 3: Functional grounding evaluation on ImageNet. Top-1 accuracy and MRR, and nDCG are reported for the single-affordance and multi-affordance settings.

Method	Train-free	Frozen VLM	Prior	AP@50
Pick best class [ 36 ]	✗	–	freq.	14.1
Ranker [ 36 ]	✗	–	learned	9.1
Task detector [ 36 ]	✗	–	learned	20.6
Classification [ 36 ]	✗	–	learned	28.8
GGNN + fusion [ 36 ]	✗	–	learned	32.6
TOIST [ 23 ]	✗	✗	task text	41.3

Table 4: COCO-Tasks AP 50 comparison. Train-free methods use no COCO-Tasks training labels or parameter updates. Our variants use frozen Faster R-CNN proposals and frozen CLIP features, varying only the task-to-object prior. The edited-KG result is excluded and studied separately as a knowledge intervention.

Grasp Model	Affordance Prior	MRR (%)	Grasp Success (%)
GraspNet	Gemini	78.33	90
	GPT-4o	75.83	70
	CRAFT	75.00	70
	CRAFT-E	76.67	90
KpNet	Gemini	80.00	60
	GPT-4o	85.00	70

Table 5: Real-world multi-affordance evaluation. MRR measures ranking quality; Grasp Success denotes successful physical execution. Only

E_{\mathrm{aff}}

is varied; the retrieval scaffold is fixed.

Panel of Figure 3 — Figure 3: Qualitative results for three queries: write , pick up , and construct . Columns show the input scene, grasp predictions, region proposals (green), and final grounding (blue). Top: successful retrieval. Middle: grasp failure excludes the correct object. Bottom: grounding failure selects a graspable but functionally incorrect object.

Grasp Model	Model	3D	Detect	Grasp Rate	Case 1	Case 2	Case 3	Case 4
GraspNet	Gemini	83.89	56.67	44.44	44.44	12.22	27.22	16.11
	GPT-4o	82.22	52.22	46.11	46.11	6.11	30.00	17.78
	CRAFT	86.11	37.78	33.33	33.33	4.44	48.33	13.89
	CRAFT-E	85.00	52.22	46.67	46.67	5.56	32.78	15.00
KpNet	Gemini	72.78	45.00	23.89	23.89	21.11	27.78	27.22
	GPT-4o	78.89	45.00	33.33	33.33	11.67	33.89	21.11

Table 6: Real-world error analysis across grasp backends and affordance priors (%). Cases: (1) full success, (2) correct grounding/grasp but execution failure, (3) correct grounding but grasp failure, and (4) incorrect grounding.

Robotic Affordance Inference

See all 24 papers

May 1, 2026

Affordance Agent Harness: Verification-Gated Skill Orchestration

Affordance grounding requires identifying where and how an agent should interact in open-world scenes, where actionable regions are often small, occluded, reflective, and visually ambiguous. Recent systems therefore combine multiple skills (e.g., detection, segmentation, interaction-imagination), yet most orchestrate them with fixed pipelines that are poorly matched to per-instance difficulty, offer limited targeted recovery from intermediate errors, and fail to reuse experience from recurring objects. These failures expose a systems problem: test-time grounding must acquire the right evidence, decide whether that evidence is reliable enough to commit, and do so under bounded inference cost without access to labels. We propose Affordance Agent Harness, a closed-loop runtime that unifies heterogeneous skills with an evidence store and cost control, retrieves episodic memories to provide priors for recurring categories, and employs a Router to adaptively select and parameterize skills. An affordance-specific Verifier then gates commitments using self-consistency, cross-scale stability, and evidence sufficiency, triggering targeted retries before a final judge fuses accumulated evidence and trajectories into the prediction. Experiments on multiple affordance benchmarks and difficulty-controlled subsets show a stronger accuracy-cost Pareto frontier than fixed-pipeline baselines, improving grounding quality while reducing average skill calls and latency. Project page: https://tenplusgood.github.io/a-harness-page/.

Haojian Huang, Jiahao Shi, Yinchuan Li +1

HKUST(GZ) · Knowin AI · Harbin Engineering University +1

Multimodal Grounding LLM Agent Orchestration Robotic Affordance Inference

Ask PDF

Jun 29, 2026

GROW $^2$ : Grounding Which and Where for Robot Tool Use

Can the robot use a plate to cut a cake if no knife is available? Tool use greatly expands robot capabilities, but to use tools creatively beyond their intended functions, the robot faces the challenge of

\textit{open-world affordance grounding}

: select an open-category object to act as a tool and localize its specific region of action. To this end, we introduce GROW

^2

(GROunding Which and Where), which leverages object parts as a natural abstraction to split the grounding process hierarchically into semantic and geometric levels, thus bypassing the need for data-heavy, end-to-end training. Semantically, GROW

^2

harnesses the commonsense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction, select a suitable object as the tool, and identify task-relevant parts on the tool and the target object. Geometrically, vision foundation models then ground the selected parts into precise 3D regions from a single RGB-D image. Experiments on established benchmarks show that GROW

^2

outperforms state-of-the-art baselines on affordance prediction benchmarks. Further, it achieves zero-shot generalization over open-category objects and outperforms baselines in both simulated and real-world robot tool use experiments.

Yuhong Deng, Yuyao Liu, David Hsu

National University of Singapore · Massachusetts Institute of Technology

3D Visual Grounding Robotic Tool Use Visually Grounded Reasoning Visual Grounding for Robotics 3D Affordance Grounding Robotic Affordance Inference+1

Ask PDF

Sep 16, 2026

Finder: Agentic Closed-Loop Object Finding for Embodied Grounding

Finding the object referred to by language in a partially observed 3D scene is a core capability for embodied agents. Existing approaches either couple object search with online exploration, which can be costly when relevant observations have already been captured, or query pre-built open-vocabulary maps and scene graphs in a static, one-shot fashion. We present Finder, an agentic closed-loop object-finding primitive for embodied grounding. Instead of treating grounding as passive retrieval from a fixed scene representation, Finder maintains a typed loop state that links query-conditioned planning, scoped evidence gathering, candidate verification, and accept/continue/abort control. When evidence is incomplete or ambiguous, the loop can redirect subsequent perception and comparison rather than simply returning the top retrieved object. On open-vocabulary embodied Object Retrieval in Habitat/HM3D and real-world RGB-D scenes, Finder improves the averaged 1m success rate by 15.75 points over strong baselines. The same primitive also transfers to sequential object grounding and embodied object-centric question answering, improving spatial and temporal localization without changing the inner grounding protocol. Project page: https://finder-vln.github.io.

Shixiong Xu, Zhiyuan Chen, Song Ding +4

Xiaomi Robotics

Robotic Object Search Embodied AI 3D Visual Grounding Visually Grounded Reasoning

Ask PDF

Abstract

Robotic Affordance Inference

Explore similar work

Affordance Agent Harness: Verification-Gated Skill Orchestration

GROW2^22: Grounding Which and Where for Robot Tool Use

Finder: Agentic Closed-Loop Object Finding for Embodied Grounding

GROW $^2$ : Grounding Which and Where for Robot Tool Use