Embodied RAG++: Retrieval-Augmented Generation From What Exists to What Works in Embodied Scenes

Fan Yang; Tatsuya Kikuzuki; Jun Liang; Jun Liu; Wenkai Li; Ryosuke Kawamura; Ananya Hazarika; Prajit KrisshnaKumar

cs.RONov 5, 2026

Embodied RAG++: Retrieval-Augmented Generation From What Exists to What Works in Embodied Scenes

Authors: Fan Yang, Tatsuya Kikuzuki, Jun Liang, Jun Liu, Wenkai Li, Ryosuke Kawamura, Ananya Hazarika, Prajit KrisshnaKumar,

Organizations: Fujitsu · Carnegie Mellon University

Abstract

To serve a request in a human environment, a robot system must find an object that satisfies a functional requirement and a robot that can feasibly reach it. Scene graphs record which objects exist and where, but they often omit capabilities documented only in product manuals. Retrieval-augmented generation methods can read such manuals, yet they rarely reason jointly over object capabilities, physical instances, robot states, and dynamic connectivity. We introduce E2W-Bench, the Exists-to-Works Benchmark, which links dynamic hybrid scene graphs with real product manuals to evaluate capability-grounded robot--object assignment under availability, mobility, vertical-reach, and connector constraints. We also present Embodied RAG++, a spatially grounded retrieval framework that scopes retrieval to the manuals linked to candidate objects, caches extracted capabilities per product model, and resolves robot--object feasibility through deterministic spatial reasoning. On E2W-Bench, Embodied RAG++ reaches a pair accuracy of 0.90 with Qwen2.5-32B and 1.00 with Llama3.3-70B, compared with at most 0.30 and 0.55 for the baselines, while requiring the shortest preprocessing time among the retrieval-based methods and tying for the lowest total token count.

Figures & tables

Figure 1: Conventional scene graph versus Embodied RAG++. Left: a scene graph, represented by 3DSceneGraph [ 1 ] , records which appliances exist and their models, but cannot tell whether a microwave can heat a 27 L container to 240°C, because that fact is stated only in its product manual. Right: Embodied RAG++ retrieves the capability from the linked manual, checks the current connector states and robot mobility, and selects the object–robot pair that satisfies every constraint.

Figure 2: Overview of one validated E2W-Bench instance and the Embodied RAG++ pipeline, including the dynamic hybrid scene graph, manual-linked objects, Capability Cards, scoped retrieval, and spatial reasoning.

Panel of Figure 3 — Figure 3: E2W-Bench dataset composition, computed directly from the 20 dynamic scenes, the appliance instance library, and the manual corpus. (a) Manual-attached electronics, robot, and other-object (distractor) instance counts per scene. (b) The underlying instance category. (c) Number of pages in the manual corpus PDF files, totaling 770 pages.

Benchmark	# Scenes	Scene State	Robots per Scene	Scene Tokens
3DSceneGraph [ 1 ]	572	Static	1	$<10$ K
ScanQA [ 14 ]	800	Static	1	$<10$ K
SQA3D [ 6 ]	650	Static	1	$<10$ K
E2W-Bench (ours)	20	Dynamic	2–7	$\approx\textbf{500}$ K

Table I: Comparison with representative scene-based benchmarks.

System	Qwen2.5-32B			Llama3.3-70B			Gemini-3.5-Flash
System	Robot	Object	Pair	Robot	Object	Pair	Robot	Object	Pair
Naive LLM	0.20	0.45	0.10	0.45	0.65	0.30	0.50	0.95	0.50
Naive LLM + SR	0.40	0.45	0.20	0.70	0.65	0.35	1.00	0.95	0.95
GraphRAG + SR	0.55	0.45	0.30	0.75	0.70	0.55	1.00	0.95	0.95
LightRAG + SR	0.55	0.45	0.25	0.35	0.45	0.35	1.00	0.85	0.85
EmbodiedRAG + SR	0.60	0.45	0.30	0.60	0.75	0.50	1.00	0.95	0.95

Table II: Exact-match accuracy of robot selection (Robot), object selection (Object), and the joint robot–object pair (Pair) on the 20 E2W-Bench queries, one per scene, across systems and LLM backbones. “+ SR” denotes use of the shared deterministic Spatial Reasoning module (§ IV-E ). Naive LLM is reported both without and with this module to isolate its contribution; all other baselines use the same module as Embodied RAG++.

Panel of Figure 4 — Figure 4: Efficiency comparison across systems on the same 20 E2W-Bench scenes as Table II . (a) Average one-time preprocessing time, including graph/index construction where applicable, shown on a log scale. (b) Average per-query retrieval-and-generation time. (c) Total tokens consumed across the full evaluation sweep. Bars mark the midpoint of each system’s reported range; error bars span the full reported minimum–maximum.

Panel of Figure 5 — Figure 5: Capability Card cache analysis: (a) For each query, the number of judged candidate models whose Capability Card was already covered (hit, green) versus needed fresh extraction or refinement from the manual (miss, red). (b) Cumulative hit rate over the run (solid) against the steady-state hit rate measured over a full deployment history (dashed).

Configuration	Qwen2.5-32B			Llama3.3-70B
Configuration	Robot	Object	Pair	Robot	Object	Pair
Embodied RAG++ (full)	0.90	0.90	0.90	1.00	1.00	1.00
w/o Capability Cards	0.90	0.90	0.90	1.00	1.00	1.00
w/o Model-level Sharing	0.85	0.85	0.85	0.75	0.75	0.75
w/o Aspect Splitting	0.80	0.75	0.75	0.65	0.65	0.65
w/o Spatial Reasoning	0.65	0.85	0.65	0.45	0.75	0.45

Table III: Ablation study of Embodied RAG++ on the 20 E2W-Bench queries. Each row removes exactly one component; we report exact-match accuracy of robot selection (Robot), object selection (Object), and the joint pair (Pair).

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 817 papers

Retrieval-Augmented Robots via Retrieve-Reason-Act
Mar 3, 2026Izat Temiraliev, Diji Yang, Yi ZhangRobotics Robot Task Planning
University of California, Santa Cruz Santa Cruz, California, USA
INHerit-SG: Incremental Hierarchical Semantic Scene Graphs for Complex Embodied Retrieval
Feb 13, 2026YukTungSamuel Fang, Zhikang Shi, Jiabin Qiu +5Graph-Based Retrieval 3D SGG
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Beihang University, Beijing, China
CRAFT-E: An Interpretable and Editable Neuro-Symbolic Framework for Affordance-Grounded Active Object Retrieval
Dec 3, 2025Zhou Chen, Joe Lin, Carson Bulgin +1Robotic Affordance Inference Robotic Object Search
CSSE Department, Auburn University, Auburn, 36849, AL, USA.

Mar 3, 2026J/K move · Enter open · S save

Retrieval-Augmented Robots via Retrieve-Reason-Act

Izat Temiraliev, Diji Yang, Yi Zhang

University of California, Santa Cruz Santa Cruz, California, USA

Robotics Robot Task Planning Long-Horizon Robotic Manipulation

Open paper AskPDF arXiv

To achieve general-purpose utility, we argue that robots must evolve from passive executors into active Information Retrieval users. In strictly zero-shot settings where no prior demonstrations exist, robots face a critical information gap, such as the exact sequence required to assemble a complex furniture kit, that cannot be satisfied by internal parametric knowledge (common sense) or past internal memory. While recent robotic works attempt to use search before action, they primarily focus on retrieving past kinematic trajectories (analogous to searching internal memory) or text-based safety rules (searching for constraints). These approaches fail to address the core information need of active task construction: acquiring unseen procedural knowledge from external, unstructured documentation. In this paper, we define the paradigm as Retrieval-Augmented Robotics (RAR), empowering the robot with the information-seeking capability that bridges the gap between visual documentation and physical actuation. We formulate the task execution as an iterative Retrieve-Reason-Act loop: the robot or embodied agent actively retrieves relevant visual procedural manuals from an unstructured corpus, grounds the abstract 2D diagrams to 3D physical parts via cross-modal alignment, and synthesizes executable plans. We validate this paradigm on a challenging long-horizon assembly benchmark. Our experiments demonstrate that grounding robotic planning in retrieved visual documents significantly outperforms baselines relying on zero-shot reasoning or few-shot example retrieval. This work establishes the basis of RAR, extending the scope of Information Retrieval from answering user queries to driving embodied physical actions.

Abstract

Retrieval-Augmented Generation

Explore similar work

Retrieval-Augmented Robots via Retrieve-Reason-Act

INHerit-SG: Incremental Hierarchical Semantic Scene Graphs for Complex Embodied Retrieval

CRAFT-E: An Interpretable and Editable Neuro-Symbolic Framework for Affordance-Grounded Active Object Retrieval