cs.ROSep 25, 2026

Representation-Guided Generation and Integration of Executable Programs for Robot Manipulation

Authors: Ruixiao Yang, Mingxin Yu, Chuchu Fan

Organizations: Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Building a robotic manipulation system requires connecting perception, planning, and control through carefully designed representations and interfaces. VLM code generation offers a way to automate this construction, but independently generated components may operate on incompatible geometric and task-level information. We present Representation-guided Integration of VLM-generated Executable Task programs (RIVET), a framework for generating complete manipulation systems around a shared object-centric representation. The representation combines per-object 6D poses, which preserve the metric information required for action grounding, with a relation graph that exposes the task-level structure required for planning. Guided by this representation, a VLM generates cooperating perception, rendering, relation-inference, and planning programs, each combining task-specific computation with available packages where useful. The resulting programs are authored once for a manipulation domain and reused on unseen start and goal configurations without code regeneration. We evaluate RIVET on cube stacking, tangram rearrangement, and three-dimensional assembly in simulation and on a physical robot, where we achieve 83% overall success rate in the real world by reusing offline-generated systems. Our results demonstrate that representation-guided program generation can adapt a common manipulation framework to tasks with different geometric, relational, and sequential requirements.

Figures & tables

Fig. 1 : Pipeline of RIVET. The system works under the guidance of an object-centric representation combining metric information from per-object 6D poses and structural information from a relation graph, which specifies the dependency of object arrangement. At Offline stage (A), a VLM uses task examples, object meshes, and available tools to write reusable functions: perceive and relate transform image observations into scene representations, plan generates position-grounded plans to accomplish the task, and render maps the representation back to an image to facilitate the generation of the other functions. At Online stage (B), the offline-generated perceive , relate , and plan functions process a new start-goal instance to generate a grounded, executable plan. The authored functions are reused across instances without instance-specific rewriting.
Fig. 2 : Experimental task configurations used in simulation and the real world. Rows show cube stacking, tangram arrangement, and 3D block assembly. Columns show four target configurations evaluated for each task.
 CubesTangramAssembly
MethodState ACPlan SRFinal SRState ACPlan SRFinal SRState ACPlan SRFinal SR
ViLaN/A43.3%N/AN/A30.0%N/AN/A8.3%N/A
VeriGraph100%86.7%63.3%80.0%33.3%0.0%10.0%10.0%0.0%
CaPN/A86.7%N/AN/A60.0%N/AN/A83.3%N/A
Instruct2ActN/A60.0%46.7%N/A96.7%0.0%N/A0.0%0.0%
RIVET(Ours)100%100%100%90.0%90.0%90.0%93.3%86.7%86.7%
TABLE I : Planning result in simulation. Each column is tested with 60 trials. We report the success rate based on three crucial stages. State AC is the ratio of accurate acquisition of the state information required by the downstream planner (VeriGraph evaluates the relation graph only, while RIVET uses both pose and relation). Plan SR is the success rate of generating a correct plan symbolically. Final SR is the success rate of controlling the robot to manipulate from the start scenario to the goal scenario in the MuJoCo simulator. “N/A” denotes that the metric is not applicable to the evaluated method, where ViLa, CaP, Instruct2Act omit the state and generate the plan directly.
TaskGoal 1Goal 2Goal 3Goal 4Total
Cube5/54/54/55/518/20
Tangram5/55/55/53/518/20
Assembly3/54/54/53/514/20
TABLE II: Real-world execution results. Each entry reports the number of success out of five attempts. For each task domain, the VLM-authored programs are generated using Goal 1, then frozen and reused across all four goals.
TasksOriginalViewpoint 1Viewpoint 2
Cube Stacking100%93.3%100%
Tangram90%100%86.7%
3D block assembly86.7%80%76.7%
TABLE III : Robustness under camera pose change.
More in this topic

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,615 papers

Explore similar work

CardsList

  1. Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation

    Oct 8, 2026Lonnie Tankersley, Noorbakhsh Amiri Golilarz, Soroush KorivandRobotic ManipulationRobot Task Planning

    dept. of Aerospace Engineering, Mississippi State University, Starkville, MS · dept. of Computer Science, The University of Alabama, Tuscaloosa, AL · dept. of Mechanical Engineering, Mississippi State University, Starkville, MS

  2. V2-STRep: VLM-Grounded Structured Task Representations for Reusable Robot Skills Acquired from Generated Videos

    Sep 17, 2026Yexin Hu, Dongheui LeeZero-Shot Robotic ManipulationGeneralization in Robotic Manipulation

    Autonomous Systems Lab, Technische Universität Wien (TU Wien), Austria · Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Germany

  3. RAPID: Robot Agentic Programming from Demonstrations

    Sep 24, 2026Yuyao Liu, Jiayuan Mao, David Hsu +2Program SynthesisContact-Rich Robotic Manipulation

    Massachusetts Institute of Technology · National University of Singapore · University of Pennsylvania +1