cs.CVOct 6, 2026
SaveEmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Xiaomi Robotics · Southeastern University · Tsinghua University · Symbiosis Robotics · Peking University · Zhejiang University · Westlake University · Xi’an Jiaotong University · Beijing Academy of Artificial Intelligence
Abstract
Scaling robotic foundation models requires diverse training data and reliable evaluation environments. Simulation offers a scalable solution, yet existing generation pipelines remain constrained by predefined assets and skills, a disconnect between scene generation and task generation, and limited support for complex embodiments and physics. We introduce EmbodiedSmith, a framework for scalable embodied data generation through recursive self-improvement (RSI). EmbodiedSmith unifies asset, scene, and task generation in a pipeline that supports autonomous creation and language-driven customization. Its core is an agentic refinement loop: scene generation anticipates downstream task requirements, while task generation guides targeted scene edits, allowing scenes and tasks to iteratively improve one another. This joint refinement improves task generation success, including for long-horizon tasks. The framework further supports mobile manipulators, humanoids, and dexterous hands, as well as interactions involving deformable objects and fluids, broadening the range of behaviors and physical phenomena represented in generated data. Together, these capabilities provide a flexible simulation engine for both robot pretraining and evaluation. Extensive experiments validate the quality, diversity, and generation efficiency of the resulting data, while downstream policy experiments demonstrate that increased data diversity improves generalization.
Figures & tables
Figure 1 : Overview of our EmbodiedSmith. Conditioned on a task and robot embodiment, EmbodiedSmith couples scene generation with task generation and execution, using feedback to iteratively refine scenes and replan.
Figure 2 : Scene overview. Our system transforms scene specifications, task instructions, and robot configurations into simulation-ready 3D scenes through functional zone planning, visual grounding, asset retrieval and generation, and physics-validated scene and tabletop layout optimization.
Figure 3 : Closed-loop robot task generation and validation. (a) Instructions are grounded in the scene, robot configuration, and available skills. (b) The Agent proposes a skill sequence and success conditions, which are used to build and compile a candidate task configuration. (c) The candidate is executed in simulation and evaluated through programmatic acceptance checks. (d) Independent VLM review complements these checks, while failures trigger diagnosis, revision, and revalidation. (e) The executable configuration is delivered upon final acceptance.
| Method | CF | SL | PP | TF | VQ | COL | NAV | OOB | OPN |
| Holodeck | 5.56 1.72 | 5.47 1.50 | 6.02 1.09 | 7.32 3.15 | 5.64 0.98 | 0.62 6.24 | 0.9967 0.0073 | 0.37 0.84 | 0.5997 0.2951 |
| LayoutVLM | 5.27 1.84 | 5.32 1.78 | 6.00 1.19 | 7.05 3.66 | 5.50 0.79 | 0.43 0.89 | 0.9954 0.0135 | 0.52 0.70 | 0.5417 0.4147 |
| SAGE | 5.20 1.98 | 5.13 1.60 | 5.92 1.08 | 7.07 3.31 | 5.15 0.78 | 0.35 1.09 | 0.9976 0.0081 | 0.03 0.18 | 0.8500 0.2311 |
| EmbodiedGen V2 | 5.57 1.68 | 5.87 1.65 | 5.43 1.39 | 6.75 3.80 | 5.95 0.98 | 2.37 1.45 | 0.9988 0.0052 | 0.19 0.48 | 0.9145 0.1363 |
| SceneSmith | 5.34 1.44 | 6.18 1.38 | 6.59 1.01 | 8.92 2.33 | 5.52 0.96 | 2.05 1.87 | 0.9916 0.0217 | 0.00 0.00 | 0.9621 0.0742 |
| Ours | 7.13 1.48 | 7.27 1.41 | 7.68 0.83 | 8.33 2.88 | 7.32 0.79 | 0.29 0.35 | 0.9984 0.0130 | 0.00 0.00 | 0.9875 0.0481 |
Table 1 : Quantitative comparison with prior scene-generation methods. Entries report mean sample standard deviation; / denote higher/lower is better, bold denotes the best result.
Figure 5 : Qualitative comparison with indoor scene generation baselines; our method yields more realistic layouts and more complete content.
Figure 6 : Qualitative comparison. Blue regions indicate reachable workspaces. Red regions indicate execution difficulties or failures. Green check marks indicate successful interactions; red crosses indicate failed interactions.
| Category | Task | SceneSmith + Planner | EmbodiedGen V2 + Planner | EmbodiedSmith | ||||||
| ASR | Top-5 | Token | ASR | Top-5 | Token | ASR | Top-5 | Token | ||
| Fixed-base manipulation | Open Drawer | ✗ | ✗ | ✗ | ||||||
| Cup to Tray | ||||||||||
| Click Button | ||||||||||
| Basket to Tray | ||||||||||
| Open Cabinet | ✗ | ✗ | ✗ | |||||||
Table 2 : Per-task demonstration generation performance on fixed-base and mobile manipulation. Short- and long-horizon tasks are distinguished by background color, with long-horizon tasks shaded in orange. Darker cells indicate the best result for each metric. ✗ indicates that EmbodiedGen V2 does not support generating the articulated assets required by the task. m. denotes memory tasks.
| Fixed-Short | Fixed-Long | Mobile-Short | Mobile-Long | |||||||||
| Method | ASR | Top-5 | Token | ASR | Top-5 | Token | ASR | Top-5 | Token | ASR | Top-5 | Token |
| w/o Feedback | 3721 | 6529 | 2979 | 5448 | ||||||||
| w/o Placement Validation | ||||||||||||
| Ours | 69.4 % | 75.2 % | 71.6 % | 79.2 % | 70.0 % | 86.0 % | 54.0 % | 65.3 % | ||||
Table 3 : Agent Workflow Ablation. We evaluate the effects of execution feedback and robot placement validation across fixed-base and mobile manipulation tasks. Token denotes the mean cumulative VLM output tokens per session.
Figure 7 : Visualization of generated robot demonstrations of diverse embodiments: mobile manipulator with a single arm, mobile manipulator with dexterous hands, and humanoid. Specifically, the humanoid robot first moves to the table and then manipulates the target.
| Name | Description |
| navigate | Move the mobile base to a working location before local manipulation. |
| pick | Execute pre-grasp, approach, gripper closure, and lift for a target object. |
| manualpick | Execute a grasp with explicitly adjusted grasp pose parameters. |
| dexpick | Execute a grasp using a selected predefined grasp pose. |
| dynamicpick | Plan and execute a grasp for a moving target. |
| place | Transport a held object to a target, descend, release it, and retreat. |
Table S1 : Robot Skills. We summarize the parameterized atomic skills supported by the simulation execution layer.
| Scene | Prompt |
|---|---|
| Kitchen with Cups | A modern minimalist rectangular kitchen is arranged in a top-down layout. The floor is covered with large light grayish-white square tiles, and the center of the room remains largely open to provide a wide, unobstructed area for movement and operation. Along the left wall stands a large silver-gray double-door refrigerator, while a smaller dark gray storage cabinet or mini fridge is placed in the lower-left corner. On the upper-left wall, there are white wall cabinets as well as a set of light wood-paneled cabinets. Along the upper wall, a continuous light gray kitchen countertop is installed, with a black glass induction cooktop embedded slightly to the right of center. The cooktop features multiple circular heating zones. On the left side of the countertop, a small number of utensils and a blue compartment organizer are placed. On the upper-right wall, there is a gray open wall shelf divided into multiple storage compartments. In the lower-right area, a white freestanding kitchen appliance or storage cabinet is placed, with a front panel featuring a round knob, several buttons, and a black control module. At the bottom center of the room is a white sink cabinet with a rectangular sink embedded in the countertop, and cabinet doors and drawers below. All large furniture and appliances are arranged along the edges of the room so as not to occupy the center, creating a spacious pathway suitable for robot navigation and manipulation. The overall color palette consists of white, light gray, dark gray, and light wood tones, with soft and even indoor lighting. In the middle of the room, a tabletop workspace contains three cups, three cubes, and one tray. The cups are colored red, blue, and green, while the cubes are colored yellow, white, and black. Each cup is positioned directly in front of its assigned cube, forming the initial cup-to-cube spatial relations: the red cup is directly in front of the yellow cube, the blue cup is directly in front of the white cube, and the green cup is directly in front of the black cube. The tray is placed in front of the cups. The drawers are positioned near the edges of the workspace and are spatially separated from the cup-cube pairs. |
| Study with Drawer | Create a modern minimalist rectangular study room or workspace in a top-down layout. The floor is covered with light natural wood flooring, with the wood grain running continuously along the long side of the room. Along the top wall, place a large white work desk with a wide tabletop and slightly rounded edges. On the desk, arrange a small number of office supplies and electronic devices in a neat but slightly random manner, including a dark gray rectangular device, a small white device, a slender red stationery item, a round cup or mug, a gray-black upright organizer, and a brownish-red cup, with appropriate spacing between all items.Along the left wall, place a light beige open bookshelf or storage cabinet with horizontal shelves. In the center-left area of the room, place a low dark reddish-brown rectangular storage box or cover-like object with rounded corners, a top surface featuring dark texture and visible scratches, and a light gold or light wood decorative band running across the middle. In the center-right area of the room, place a large light gray circular rug with a darker gray border and a slightly mottled texture.Maintain sufficient spacing between the work desk, bookshelf, rectangular storage box, and circular rug, so that the center of the room forms a continuous, unobstructed open area suitable for robot navigation and manipulation. Use soft, even indoor lighting, and adopt a color palette dominated by light wood, white, beige, gray, and dark brown tones.A tabletop workspace contains 3 cups, 3 cubes, 1 tray. The cups are red, blue, and green, and the cubes are yellow, white, and black. Each cup is positioned directly in front of its assigned cube, forming the initial cup-to-cube spatial relations: the red cup is directly in front of the yellow cube, the blue cup is directly in front of the white cube, and the green cup is directly in front of the black cube. The tray is positioned in front of the cups. The drawers are positioned near the edges of the workspace and are spatially separated from the cup-cube pairs. |
| Living Room with Basket | Create a modern minimalist rectangular living room in a top-down layout. Use light natural wood flooring with a consistent wood-grain direction. Arrange most large furniture along the edges of the room and preserve a large, continuous, unobstructed open area in the center for robot navigation and manipulation.Place a light gray three-seat sofa along the bottom side of the room. The sofa should have three clearly separated seat cushions and a simple modern design.Place a rectangular light-wood table slightly left of the center of the room. Use the table as the main tabletop manipulation workspace. Scatter several small everyday objects across the tabletop with clear spacing between them, including a blue rectangular package or book, several small circular objects, a yellow handle-shaped object, a small frame-like object, and a few small red, white, and dark-colored items. Keep the tabletop objects sparse and clearly separated so that each object is individually accessible for robotic manipulation.Along the left wall, place a long and narrow light-wood storage cabinet or sideboard. Place a few small dark decorative objects or containers on top of it. In the lower-left area, place a low dark-brown rectangular storage box or small piece of furniture with a lighter vertical decorative strip.Add additional dark or brown wooden furniture along the upper and right edges of the room, while keeping these objects away from the central navigation area.Use a warm modern color palette dominated by light wood, white, light gray, dark brown, and small black accents. Use soft and even indoor lighting. Keep the overall scene clean, spacious, and suitable for robotics simulation, tabletop manipulation, object rearrangement, and indoor navigation tasks.In the middle of the room, there is a tabletop workspace containing a wooden basket with a handle and a shallow tray. |
| Bedroom with Button | Create a modern minimalist rectangular bedroom in a top-down layout. Cover the floor with light natural wood flooring, with the wood grain running continuously along the long side of the room. Keep the central area of the room largely open and unobstructed for robot navigation and manipulation. Place a large dark gray double bed in the upper-left area of the room. The bed should have a rectangular shape with slightly rounded edges and a dark gray to charcoal-colored mattress and frame. Along the right wall, place a large brown wooden three-door wardrobe. Each wardrobe door should have recessed rectangular panel details and a long curved white or silver handle. Use a light gray frame around the wardrobe. At the bottom center of the room, place a low light-wood bench, storage cabinet, or bed-end furniture piece. Place a dark gray or black low cabinet in the lower-left area, and a dark brown wooden storage cabinet or bedside cabinet in the lower-right area. In the upper-right area, place a small orange-brown square ottoman or padded stool. Add a small dark wooden cabinet or furniture piece nearby along the upper wall. Arrange all large furniture close to the room boundaries and preserve a wide, continuous open area in the center. Use a warm modern color palette dominated by light wood, dark gray, brown, beige, and small orange-brown accents. Use soft and even indoor lighting. Ensure the scene is clean, spacious, and suitable for robotics simulation, indoor navigation, object retrieval, and wardrobe interaction tasks.In the middle of the room, there is a table with a blue circular button placed on top of it. |
| Kitchen Counter with Cabinet | The scene contains a long kitchen-style counter measuring approximately 2.0 m in length and 0.65 m in depth. A microwave is placed on one end of the counter at a height reachable by the robot. The microwave has a single graspable hinged door that can be opened and closed. A food container is placed on the countertop near the microwave. At the opposite end of the counter, place a small storage cabinet with a single graspable hinged door. Two graspable storage boxes are placed on the countertop near the cabinet. A storage basket is positioned on the counter a short distance away from the cabinet. The microwave area and cabinet area should be sufficiently separated so that the robot must reposition its mobile base when moving between them. A continuous unobstructed navigation corridor must remain along the entire front side of the counter. |
| Bedroom with Pose References | Create a modern minimalist rectangular bedroom in a top-down layout. Cover the floor with light natural wood flooring, with the wood grain running continuously along the long side of the room. Keep the central area of the room largely open and unobstructed for robot navigation and manipulation. Place a large dark gray double bed in the upper-left area of the room. The bed should have a rectangular shape with slightly rounded edges and a dark gray to charcoal-colored mattress and frame. Along the right wall, place a large brown wooden three-door wardrobe. Each wardrobe door should have recessed rectangular panel details and a long curved white or silver handle. Use a light gray frame around the wardrobe. At the bottom center of the room, place a low light-wood bench, storage cabinet, or bed-end furniture piece. Place a dark gray or black low cabinet in the lower-left area, and a dark brown wooden storage cabinet or bedside cabinet in the lower-right area. In the upper-right area, place a small orange-brown square ottoman or padded stool. Add a small dark wooden cabinet or furniture piece nearby along the upper wall. Arrange all large furniture close to the room boundaries and preserve a wide, continuous open area in the center. Use a warm modern color palette dominated by light wood, dark gray, brown, beige, and small orange-brown accents. Use soft and even indoor lighting. Ensure the scene is clean, spacious, and suitable for robotics simulation, indoor navigation, object retrieval, and wardrobe interaction tasks. In the middle of the room, there is a table. On the tabletop, three reference objects — a red cube, a yellow cube, and a blue cylinder — are arranged in a row as the reference region, each at a distinct position and orientation. An opaque rectangular memory cover with a graspable handle on its top surface is placed beside the reference row; it is large enough to be moved over the reference region and lifted vertically to reveal the objects underneath. On the other side of the table, an empty tray serves as the work region, and the three matching work objects — identical copies of the red cube, the yellow cube, and the blue cylinder — are placed next to the tray. |
Table S2 : Scene Prompts. We list the full scene generation prompts used in the demonstration-generation evaluation. Prompts shared by multiple tasks appear once.
| Task | Scene | Task prompt | Completion condition |
| Fixed-base manipulation / Easy | |||
| Open Drawer | Study with Drawer | Pull the drawer fully open. | The drawer reaches its full-opening threshold. |
| Cup to Tray | Kitchen with Cups | Pick the green cup and place it on the tray. | The correct cup is released in the valid tray region and supported. |
| Click Button | Bedroom with Button | Press the blue circular button. | Robot interaction activates the correct button. |
| Basket to Tray | Living Room with Basket | Pick up the basket and place it on the tray. | The basket is released in the valid tray region and supported. |
| Open Cabinet | Kitchen Counter with Cabinet | Open the cabinet door. | The cabinet door reaches its hinge-opening threshold. |
Table S3 : Task Prompts and Scene Assignments. We list the task instruction, generated scene, and success condition for ten fixed-base and six mobile manipulation tasks, grouped into easy and long-horizon tasks.
Figure S1 : Data of Block stacking with randomization
| Task | Task-generation prompt |
|---|---|
| A1: Spatially Guided Placement | Initial state. Use the existing study-room scene. Reset three differently colored small blocks to separate tabletop positions with clear left, middle, and right relations in the main camera view. The target tray is on the right side of the image, and the position instruction identifies the block to manipulate. Actions. Pick the designated block, approach from a suitable direction, close the gripper securely, and lift high enough to clear nearby objects. Place it inside the target tray, release, retreat, and return the arm to its initial pose. Do not displace either of the other blocks. Success. The block at the instructed image-relative position is fully inside the tray and stable after release, while the other blocks remain in place. Generate left, middle, and right target instructions and align each with its trajectory; color must not be the sole selection cue. Policy instruction. “Put the leftmost block in the image into the tray on the right, keeping the other two blocks in place.” Generate the corresponding instructions for the middle and rightmost blocks. |
| A2: Cuboid Placement | Initial state. Use the existing study-room scene with the robot mounting position unchanged. A long box lies flat on the table, and an empty tray is within reach nearby. Their long axes initially form a clear angle. Actions. Pick the box, adjust its horizontal orientation, and place it at the center of the tray. Choose a grasp compatible with the gripper opening. After lifting the box, align it while keeping it flat, and preserve the target orientation through placement. Leave clearance for the gripper and end effector during approach and release to avoid the table and tray. Success. The box is fully inside the tray, with its long axis parallel to the tray’s long axis and orientation error no greater than . It remains stable after release and retreat, and the arm returns to its initial pose. Do not add slot insertion, flipping, or precision assembly. Policy instruction. “Place the long box at the center of the tray with its long side parallel to the tray’s long side.” |
| A3: Block Stacking | Initial state. Use the existing study-room scene. A wide base block and a small block lie separately on the table. The base block’s top surface is clearly larger than the small block, with sufficient grasping and placement clearance. Actions. Pick only the small block and place it at the center of the wide base block. Keep the base block in place. Lift high enough to clear the base block, approach its support surface from above, and release without striking the base block or pushing the upper block out of alignment. Build only a two-block stack. Success. The small block is physically supported by the base block, does not overhang its edges, and remains stable for at least 1 second after the gripper retreats. The arm returns to its initial pose. Height alone is insufficient, and the task must not pass while the gripper still supports the block. Policy instruction. “Stack the small block at the center of the wide base block.” |
| A4: Unstack and Place | Initial state. Use the existing study-room scene. Reset two blocks into a stack so that the upper block obstructs the main grasping space of the lower block. Place an empty tray nearby and reserve a temporary placement area beside the stack. Actions. Pick the upper block, place it in the temporary area, pick the lower block, and place it in the tray. Confirm that the upper block has been released and settled before grasping the lower block. Do not topple the stack or grasp both blocks together. The temporary placement must not obstruct the approach to the lower block. Release, retreat, and return to the initial arm pose. Success. The upper block remains in the temporary area, the lower block is fully inside the tray, and both are stable. Preserve evidence that the upper block was removed before the lower block was grasped; final positions alone do not verify the required order. Policy instruction. “Move the upper block aside first, then put the lower block into the tray.” |
| A5: Drawer Opening | Initial state. Use the existing study-room scene. Fix an empty tabletop drawer module within reach, with its handle facing the robot. Reset the drawer to a closed or nearly closed state. Check its actual sliding direction, joint zero, limits, and handle keypoints, and leave clearance for opening and retreat. Actions. Use an open skill appropriate for a prismatic drawer. Approach and grasp the handle securely, pull the drawer approximately 7 cm along its actual sliding direction, release, retreat safely, and return to the initial arm pose. Keep the target opening within the actual joint limits. Do not lift the entire module with an ordinary pick or add manipulation inside the drawer. Success. The joint reaches the target opening and remains open after release and retreat. The cabinet remains in place, without obvious interpenetration or sustained oscillation. Opening alone is required; subsequent closing is not part of the task. Policy instruction. “Pull open the tabletop drawer.” |
| B1: Sequential Object Placement | Initial state. Use the existing kitchen scene. A small cup and a small block begin outside the tray. Reserve two non-overlapping placement regions inside the tray, with left and right defined relative to the robot’s initial heading. Actions. Pick the block and place it on the left, then pick the cup and place it on the right. Use stable grasps and avoid tilted insertion that may cause slipping. Keep the cup upright and avoid disturbing the placed block during the second transfer. Keep the tray in place, then release, retreat, and return to the initial arm pose. Success. Both objects are physically supported by the tray, with their horizontal projections inside its valid interior, and satisfy the specified left–right relation without crowding each other. The cup remains upright and the gripper has released. The cup may extend above the shallow tray rim. For a reversed left–right variant, update the instruction, target regions, and action configuration together. Policy instruction. “Put the block on the left side of the tray and the small cup on the right.” |
Table S4 : Task-generation Prompts for Policy Training and Evaluation. We present the revised task-generation prompts for the five study-room tasks (A1–A5) and five kitchen tasks (B1–B5). Each entry specifies the initial state, action sequence, success conditions, and policy instruction.
| Task | Atomic Actions | Avg. Duration (s) | Success Rate (%) |
| Spatially Guided Placement | 3 | 13.40 | 88.55 |
| Cuboid Placement | 3 | 14.80 | 97.32 |
| Block Stacking | 4 | 13.37 | 97.09 |
| Unstack and Place | 6 | 25.99 | 85.25 |
| Drawer Opening | 3 | 19.60 | 79.38 |
| Sequential Object Placement | 6 | 31.98 | 80.32 |
Table S5 : Task statistics for trajectory collection.
| Hyperparameters | FastWAM | |
| Batch Size (Total) | 256 | 64 |
| Learning Rate | ||
| Learning Rate Schedule | Constant | Cosine Decay |
| Training Steps (Max.) | 20k | 70k |
Table S6 : Training hyperparameters. Learning-rate schedules use 10k warmup steps for and 5k for FastWAM. FastWAM retains a 100k-step cosine schedule.
Figure S2 : Representative rollouts of the ten evaluation tasks. Each row corresponds to one task, while the columns show representative execution stages in temporal order from left to right. For clearer visualization of task execution, tabletop distractors are disabled in these illustrative rollouts.