Embodied RAG++: Retrieval-Augmented Generation From What Exists to What Works in Embodied Scenes

To achieve general-purpose utility, we argue that robots must evolve from passive executors into active Information Retrieval users. In strictly zero-shot settings where no prior demonstrations exist, robots face a critical information gap, such as the exact sequence required to assemble a complex furniture kit, that cannot be satisfied by internal parametric knowledge (common sense) or past internal memory. While recent robotic works attempt to use search before action, they primarily focus on retrieving past kinematic trajectories (analogous to searching internal memory) or text-based safety rules (searching for constraints). These approaches fail to address the core information need of active task construction: acquiring unseen procedural knowledge from external, unstructured documentation. In this paper, we define the paradigm as Retrieval-Augmented Robotics (RAR), empowering the robot with the information-seeking capability that bridges the gap between visual documentation and physical actuation. We formulate the task execution as an iterative Retrieve-Reason-Act loop: the robot or embodied agent actively retrieves relevant visual procedural manuals from an unstructured corpus, grounds the abstract 2D diagrams to 3D physical parts via cross-modal alignment, and synthesizes executable plans. We validate this paradigm on a challenging long-horizon assembly benchmark. Our experiments demonstrate that grounding robotic planning in retrieved visual documents significantly outperforms baselines relying on zero-shot reasoning or few-shot example retrieval. This work establishes the basis of RAR, extending the scope of Information Retrieval from answering user queries to driving embodied physical actions.

Object retrieval in 3D scenes remains difficult when queries combine hierarchical context, attributes, negation, and chained spatial relations. Existing semantic maps, scene graphs, and agentic memory support open-vocabulary retrieval, but jointly resolving these constraints remains challenging. In this work, we propose INHerit-SG, an incremental hierarchical semantic scene graph with an accompanying retrieval method for complex embodied retrieval. Asynchronous geometric and semantic processing and structural event-driven maintenance organize observations into a hierarchy whose node semantics and spatial relations are recorded as natural language text. The same memory supports direct reading and reasoning by more capable language models or query parsing followed by structured retrieval. Controlled comparisons show that direct reading is more accurate with powerful hosted models, while structured retrieval improves the accuracy of smaller local models and lowers per-query API cost for hosted models. We also construct HM3DSem-SQR, containing 6,084 queries from 36 HM3D-Sem scenes, and evaluate on Main-864, a class-balanced subset spanning all 36 scenes, and in three real-world environments. Experiments show that INHerit-SG outperforms the strongest baselines in both simulator and real-world evaluations. The largest gains come on attribute and compositional queries (7.6 and 6.3 strict Top-1 points on Main-864), with a 9.8-point overall success margin on real robots. Project Page: https://fangyuktung.github.io/INHeritSG.github.io/

Assistive robots operating in unstructured environments must reason not only about what objects are, but how they can be used, including non-canonical and improvised uses. This requires grounding language-based action queries to objects that afford the requested function, are visually present, and can be physically retrieved. Existing approaches often rely on black-box models, fixed affordance labels, or direct object predictions, limiting transparency and controllability. We introduce CRAFT-E, a modular neuro-symbolic framework that composes an LLM-derived verb-property-object knowledge graph with CLIP-based visual alignment and energy-based grasp reasoning. The system exposes interpretable grounding paths, incorporates grasp feasibility into affordance inference, and supports targeted graph edits without retraining. We also construct a benchmark dataset with unified annotations for verb-object compatibility, segmentation, and grasp candidates, and deploy the full pipeline on a physical robot. Across static scenes, real-world trials involving 20 verbs and 39 objects, ImageNet-based functional retrieval, and COCO-Tasks transfer, CRAFT-E provides a competitive training-free prior for affordance-grounded retrieval while exposing component-level diagnostics and graph-level interventions. Under staged and full-pipeline perception, CRAFT-E matches strong flat LLM priors in several settings while adding failure attribution and editable reasoning paths. By coupling symbolic reasoning with embodied perception, CRAFT-E offers an inspectable and customizable framework for affordance-grounded object retrieval.