CRAFT-E: An Interpretable and Editable Neuro-Symbolic Framework for Affordance-Grounded Active Object Retrieval

Zhou Chen; Joe Lin; Carson Bulgin; Sathyanarayanan N. Aakur

cs.RODec 3, 2025

CRAFT-E: An Interpretable and Editable Neuro-Symbolic Framework for Affordance-Grounded Active Object Retrieval

Authors: Zhou Chen, Joe Lin, Carson Bulgin, Sathyanarayanan N. Aakur

Organizations: CSSE Department, Auburn University, Auburn, 36849, AL, USA.

Abstract

Assistive robots operating in unstructured environments must reason not only about what objects are, but how they can be used, including non-canonical and improvised uses. This requires grounding language-based action queries to objects that afford the requested function, are visually present, and can be physically retrieved. Existing approaches often rely on black-box models, fixed affordance labels, or direct object predictions, limiting transparency and controllability. We introduce CRAFT-E, a modular neuro-symbolic framework that composes an LLM-derived verb-property-object knowledge graph with CLIP-based visual alignment and energy-based grasp reasoning. The system exposes interpretable grounding paths, incorporates grasp feasibility into affordance inference, and supports targeted graph edits without retraining. We also construct a benchmark dataset with unified annotations for verb-object compatibility, segmentation, and grasp candidates, and deploy the full pipeline on a physical robot. Across static scenes, real-world trials involving 20 verbs and 39 objects, ImageNet-based functional retrieval, and COCO-Tasks transfer, CRAFT-E provides a competitive training-free prior for affordance-grounded retrieval while exposing component-level diagnostics and graph-level interventions. Under staged and full-pipeline perception, CRAFT-E matches strong flat LLM priors in several settings while adding failure attribution and editable reasoning paths. By coupling symbolic reasoning with embodied perception, CRAFT-E offers an inspectable and customizable framework for affordance-grounded object retrieval.

Figures & tables

Figure 1: Overview. Given a scene and verb query, CRAFT-E predicts graspable candidate regions, reasons over an affordance-centric verb subgraph, and uses frozen vision-language alignment to select a functionally appropriate and retrievable object.

Figure 2: LLM-derived affordance knowledge (left) versus ConceptNet subgraphs (right). CRAFT-E links verbs to objects through interpretable functional properties, while ConceptNet relies on general-purpose relations such as RelatedTo , which can yield noisy paths with limited explanatory structure.

Grasp Model	Affordance Prior	3D (%)	Detect (%)	Grasp Rate (%)
GraspNet	Gemini	83.89	56.67	44.44 [32.2, 57.2]
	GPT-4o	82.22	52.22	46.11 [33.3, 58.9]
	CRAFT	86.11	37.78	33.33 [25.0, 43.3]
	CRAFT-E	85.00	52.22	46.67 [33.9, 60.0]
KpNet	Gemini	72.78	45.00	23.89 [17.2, 30.6]
	GPT-4o	78.89	45.00	33.33 [27.8, 38.9]

Table 1: Real-world evaluation of 3D grasp feasibility, detection accuracy, and grasp execution success. Only

E_{\mathrm{aff}}

is varied; the retrieval scaffold is fixed. Grasp Rate includes 95% paired-bootstrap confidence intervals over 20 verb-object pairs.

Setting	Grasp	ROI	Affordance Prior	Accuracy (%)
Perfect Grasp & Detect	GT	GT	Gemini	59.79
			GPT-4o	62.89
			CRAFT	58.76
			CRAFT-E	64.95
			CRAFT-E+GPT-4o	52.58
Perfect Grasp	GT	SAMv2	Gemini	56.70

Table 2: Static evaluation accuracy under perfect grounding, perfect grasping, and full pipeline settings. Only

E_{\mathrm{aff}}

is varied; the retrieval scaffold is fixed. CRAFT-E+GPT-4o uses GPT-4o to derive the LLM-KG.

Approach	Accuracy (%)	MRR (%)	nDCG (%)
Object-Aware Oracle	100.00	69.90	76.80
Affordance-Aware Oracle	73.80	68.70	82.70
CLIP+verb prompting	38.00	54.80	56.10
ResNet-RNN [ 27 , 28 ]	60.20	65.70	76.50
ALGO [ 21 ]	42.52	52.38	53.33
Gemini-2.0-Flash [ 2 ]	42.80	57.50	59.70

Table 3: Functional grounding evaluation on ImageNet. Top-1 accuracy and MRR, and nDCG are reported for the single-affordance and multi-affordance settings.

Method	Train-free	Frozen VLM	Prior	AP@50
Pick best class [ 36 ]	✗	–	freq.	14.1
Ranker [ 36 ]	✗	–	learned	9.1
Task detector [ 36 ]	✗	–	learned	20.6
Classification [ 36 ]	✗	–	learned	28.8
GGNN + fusion [ 36 ]	✗	–	learned	32.6
TOIST [ 23 ]	✗	✗	task text	41.3

Table 4: COCO-Tasks AP 50 comparison. Train-free methods use no COCO-Tasks training labels or parameter updates. Our variants use frozen Faster R-CNN proposals and frozen CLIP features, varying only the task-to-object prior. The edited-KG result is excluded and studied separately as a knowledge intervention.

Grasp Model	Affordance Prior	MRR (%)	Grasp Success (%)
GraspNet	Gemini	78.33	90
	GPT-4o	75.83	70
	CRAFT	75.00	70
	CRAFT-E	76.67	90
KpNet	Gemini	80.00	60
	GPT-4o	85.00	70

Table 5: Real-world multi-affordance evaluation. MRR measures ranking quality; Grasp Success denotes successful physical execution. Only

E_{\mathrm{aff}}

is varied; the retrieval scaffold is fixed.

Panel of Figure 3 — Figure 3: Qualitative results for three queries: write , pick up , and construct . Columns show the input scene, grasp predictions, region proposals (green), and final grounding (blue). Top: successful retrieval. Middle: grasp failure excludes the correct object. Bottom: grounding failure selects a graspable but functionally incorrect object.

Grasp Model	Model	3D	Detect	Grasp Rate	Case 1	Case 2	Case 3	Case 4
GraspNet	Gemini	83.89	56.67	44.44	44.44	12.22	27.22	16.11
	GPT-4o	82.22	52.22	46.11	46.11	6.11	30.00	17.78
	CRAFT	86.11	37.78	33.33	33.33	4.44	48.33	13.89
	CRAFT-E	85.00	52.22	46.67	46.67	5.56	32.78	15.00
KpNet	Gemini	72.78	45.00	23.89	23.89	21.11	27.78	27.22
	GPT-4o	78.89	45.00	33.33	33.33	11.67	33.89	21.11

Table 6: Real-world error analysis across grasp backends and affordance priors (%). Cases: (1) full success, (2) correct grounding/grasp but execution failure, (3) correct grounding but grasp failure, and (4) incorrect grounding.

Robotic Affordance Inference

See all 24 papers

Affordance Agent Harness: Verification-Gated Skill Orchestration
May 1, 2026Haojian Huang, Jiahao Shi, Yinchuan Li +1Multimodal Grounding LLM Agent Orchestration
HKUST(GZ) · Knowin AI · Harbin Engineering University +1
GROW $^2$ : Grounding Which and Where for Robot Tool Use
Jun 29, 2026Yuhong Deng, Yuyao Liu, David Hsu3D Visual Grounding Robotic Tool Use
National University of Singapore · Massachusetts Institute of Technology
Finder: Agentic Closed-Loop Object Finding for Embodied Grounding
Sep 16, 2026Shixiong Xu, Zhiyuan Chen, Song Ding +4Robotic Object Search Embodied AI
Xiaomi Robotics

May 1, 2026J/K move · Enter open · S save

Affordance Agent Harness: Verification-Gated Skill Orchestration

Haojian Huang, Jiahao Shi, Yinchuan Li, Yingcong Chen

HKUST(GZ) · Knowin AI · Harbin Engineering University+1

Multimodal Grounding LLM Agent Orchestration Robotic Affordance Inference

Open paper AskPDF arXiv

Affordance grounding requires identifying where and how an agent should interact in open-world scenes, where actionable regions are often small, occluded, reflective, and visually ambiguous. Recent systems therefore combine multiple skills (e.g., detection, segmentation, interaction-imagination), yet most orchestrate them with fixed pipelines that are poorly matched to per-instance difficulty, offer limited targeted recovery from intermediate errors, and fail to reuse experience from recurring objects. These failures expose a systems problem: test-time grounding must acquire the right evidence, decide whether that evidence is reliable enough to commit, and do so under bounded inference cost without access to labels. We propose Affordance Agent Harness, a closed-loop runtime that unifies heterogeneous skills with an evidence store and cost control, retrieves episodic memories to provide priors for recurring categories, and employs a Router to adaptively select and parameterize skills. An affordance-specific Verifier then gates commitments using self-consistency, cross-scale stability, and evidence sufficiency, triggering targeted retries before a final judge fuses accumulated evidence and trajectories into the prediction. Experiments on multiple affordance benchmarks and difficulty-controlled subsets show a stronger accuracy-cost Pareto frontier than fixed-pipeline baselines, improving grounding quality while reducing average skill calls and latency. Project page: https://tenplusgood.github.io/a-harness-page/.

Abstract

Robotic Affordance Inference

Explore similar work

Affordance Agent Harness: Verification-Gated Skill Orchestration

GROW2^22: Grounding Which and Where for Robot Tool Use

Finder: Agentic Closed-Loop Object Finding for Embodied Grounding

GROW $^2$ : Grounding Which and Where for Robot Tool Use