cs.ROOct 24, 2026
SaveMOTIO: Agentic Skill Engineering for Reality-Grounded Humanoid Loco-Manipulation
Abstract
We present MOTIO, an agentic skill-engineering system that turns ordinary human-action videos, reality-grounded 3D assets, motion references, and reusable robot skills into executable humanoid loco-manipulation programs. Through in-context task and contact reasoning, an agent constructs staged robot-native interactions and improves them using physics execution, multi-frame feedback, and human validation---without task-specific teleoperation or robot demonstration collection. Agentic 3D supplies metric product and environment assets, while existing whole-body, inverse-kinematics, and dexterous-hand interfaces realize the programs. Across 33 timestamped first approvals, the median recorded agent-turn time is 97.0 minutes and request-to-approval duration is 2.80 hours. Simulation-side layout and object variation prepare a long-horizon bottle-disposal skill for a continuous 101.75-second physical G1 execution in a scanned office; physical refinement reduces screw-tightening thread residual by 76.0%. Agent-generated skills additionally supervise BC/DAgger distillation into one shared reference policy, completing 18 simulated tasks under the same checkpoint without inference-time expert intervention. Project page: https://iiiiqiiii.github.io/motio/
Figures & tables
Figure 2: MOTIO workflow illustrated with G1 sitting on a black office chair. A reviewed asset enters video- and skill-conditioned program synthesis, followed by MuJoCo execution, multi-frame feedback, and human validation with physical G1 execution. Skill-library executions support downstream shared-policy learning.
Figure 3: Representative Agentic 3D assets from everyday tools, engineered interfaces and office objects. Metric geometry and component structure support task-specific physical interaction models (Supplementary Section S1).
Figure 4: Video-conditioned task engineering for a HEUER vise. Top: an illustrative third-person vise operation; middle: asset and robot-skill context; bottom: G1 MuJoCo grasp, turn, regrasp and clamp at 12, 24 and 43 s, followed by handle release and workpiece retention.
Figure 5: Reality-grounded office reconstruction and task preparation. (a) Room 2 appearance rendered from the scanned 3D Gaussian model. (b) Metric scan geometry across three connected rooms; Room 2 spans 29.3 m 2 . (c) The interactive MuJoCo bottle-to-bin scene. (d) Table, bottle and bin locations across 31 executed layout configurations.
Figure 6: Physical G1 deployment. Top: selected moments from the continuous 101.75-second bottle grasp, carry and disposal episode. Bottom: door opening followed by cart transport in a 98.41-second episode. Insets magnify contact and task details from the corresponding frames.
| Study | Variation or sequence | Observed result |
| Simulation preparation | ||
| Full-task layouts | 31 configurations | 31/31 complete sequences |
| Bottle properties | 44 unique settings | 44/44 full-window grasps |
| Paired grasp refinement | Same 12 settings | 11/12 12/12 full-window retention |
| Perception-conditioned execution | 3 whole-body variants | 3 complete eight-stage sequences |
| Physical G1 execution | ||
Table 1: Simulation preparation and physical G1 execution.
Figure 7: Diverse humanoid skills spanning long-horizon mobility, environment interaction, balance, multi-robot cooperation, dynamic object handling, articulated tools and physical–digital interaction. Close-ups highlight task-critical hand–object interactions.
Figure 8: Development budget across 33 skills with matched first human approval: empirical cumulative distributions for agent-turn time, retained dynamic trajectories and visual inspection calls. Dashed vertical lines mark medians.
| Measure | Median | IQR |
|---|---|---|
| Agent-turn (min) | 97.0 | 42.5–170.3 |
| Request-to-approval (h) | 2.80 | 1.33–9.80 |
| Agent turns | 2 | 1–2 |
| Retained dynamic versions | 18 | 7–31 |
| Visual inspections | 10 | 7–18 |
| Human revision prompts | 1 | 0–1 |
Table 2: Development budget to first human approval across 33 skills (median and interquartile range).
Figure 9: Physical refinement of screw tightening from first to human-approved final candidate: maximum thread residual 269.44 to 64.58 m (76.0% lower) and final advancement 0.7697 to 0.8019 mm against 0.8 mm target.
Figure 10: Agent-engineered simulated skill executions provide BC initialization and DAgger expert references for a shared reference generator, executed through the frozen ScaleBFM whole-body controller and retained task-feedback interfaces.
Figure 11: Shared-policy development across ten added tasks (BC 2/10 to D7 10/10) and final one-checkpoint eighteen-task performance (18/18 development and 18/18 same-condition repeat; zero inference-time expert queries/takeovers). The stage trace includes program, representation and execution-interface refinements.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Agentic 3D families across tools, engineered interfaces, product mechanisms and office furnishings.
| Product asset | Physical model | Skill interface / event |
|---|---|---|
| Rapesco 159 41 53 mm | Base, lever, magazine, press cap, driver, anvil and contact proxies; articulated lever and magazine. | Right-palm press-cap target, left-hand paper stabilization, contact-gated paper connection and withdrawal. |
| HEUER vise | Fixed seat, translating jaw, screw, moving handle, supported workpiece; screw/jaw coupled at model lead 4 mm/turn. | Hand arc referenced to screw angle; opposing clamp, jaw clearance and retained workpiece measured at task completion. |
Table S1: Task-facing asset to physics interface examples.
| Window | Program phase | Observation |
|---|---|---|
| 0–8 s | Reach and grasp handle bar; workpiece initially supported. | Initial reach/contact. |
| 8–36 s | Four 7-s half-turn cycles with unflex, retreat, reposition, regrasp. | Rotation, contact and jaw travel. |
| 36–41 s | Final tighten, release hand and withdraw support. | Clamp loading and release. |
| 41–48 s | Hold test after both hand and support withdraw. | Workpiece retention and gap (38 to 29.987 mm). |
Table S2: HEUER vise right-hand stages and physical checks.
| Review input | Program revision | Physical result |
| Palm-driven press and natural hand withdrawal. | 13–16 s palm press, 16–17 hold, 17–19 unload, 19–21 paper release, 21–24 elbow retract, 24–29 lower arms. | Staple connection event 14.8125 s; paper joined, hands withdrawn by 31 s; human approval. |
Table S3: Rapesco feedback-driven program change.
Figure S2: Detailed simulation-side variation. Left: positions of table, bottle and bin for 31 full-task layouts. Right: 44 unique simulated bottle-parameter settings across diameter, height and mass; each connected line represents one tested setting.
| Study | Variation and criterion | Observed result |
|---|---|---|
| Room layout | 11 named, 12 development-random, 8 post-freeze; full grasp, carry, bin release, return. | 31/31 full-task completions; 61.2–138.6 s. |
| Bottle properties | 58–68 mm diameter, 220–235 mm height, 20–600 g; hand fixture at 8–10 s. | 44/44 opposing-grasp windows. |
| Paired refinement | The same 12 LHS settings, earlier versus refined contact window. | 11/12 to 12/12 full-window retention. |
| Visual whole-body | Measured, narrow-heavy and wide-light parameter conditions. | 3/3 eight-stage completions. |
Table S4: Bottle-to-bin simulation variation units and completion checks.
Figure S3: Time to first human approval for 33 matched skills. Left: recorded agent-turn minutes; right: request-to-approval calendar hours on separate logarithmic axes.
| ID | Skill | Agent | Calendar | Versions | Visual |
|---|---|---|---|---|---|
| 40 | Mango peeling | 80.6 | 1.67 | 9 | 17 |
| 41 | Caliper measurement | 204.2 | 18.52 | 7 | 18 |
| 43 | Vise clamping | 93.8 | 19.51 | 5 | 6 |
| 44 | Drawer retrieval | 170.3 | 19.53 | 8 | 13 |
| 45 | Dowel insertion | 79.3 | 19.48 | 3 | 9 |
| 46 | Gluing paper | 123.2 | 2.42 | 31 | 20 |
Table S5: First human-approval development records for 33 numbered skills. Agent time in minutes, calendar time in hours.
Figure S4: Human-guided refinement and approval milestones for Cases 66, 71 and 72: two-humanoid sofa transport, electric skateboard and flexible-bag manipulation.
Figure S5: Complete measured screw-tightening candidate sequence full01–full04, including the early-ended full03 trace.
| Component | Shared policy configuration |
|---|---|
| Observation | 176 task/robot state slots, 32 mechanism values with 32 validity-mask bits, skill and phase identifiers, public clock and explicit geometry memory. |
| History | reference_history=False ; current public phase and geometry memory condition reference prediction. |
| Output | Six references s, each 59 channels: 29 body, 12 hand, 6 palm positions and 12 palm 6D rotations. |
| Executor | 50 Hz reference decoder, frozen ScaleBFM tracker, bounded arm IK and task-event/contact feedback. |
| Training | BC initialization, student-state DAgger expert labeling and event-gated recovery corrections. |
Table S6: Final BC/DAgger student observation and reference interface.
| Task | Time (s) | Dev. | Repeat |
|---|---|---|---|
| Bottle pickup | 60 | yes | yes |
| Light switch | 48 | yes | yes |
| Drawer | 66 | yes | yes |
| Stapler | 31 | yes | yes |
| Standing fan | 50 | yes | yes |
| Faucet | 28 | yes | yes |
Table S7: Eighteen simulated tasks under the same shared reference checkpoint.
Figure S6: Eighteen simulated tasks executed under one shared reference-policy checkpoint.
Figure S7: Six time-ordered task sequences under the same shared reference checkpoint.
| Task | Duration | Physical sequence |
|---|---|---|
| Bottle disposal | 101.7543 s | Grasp, turn, carry, bin release, retreat. |
| Door and cart transport | 98.41 s | Door opening and pushing a cart through doorway. |
| Chair sit-to-stand (unsupported) | 24.74 s | Sit, hold and return to stand. |
| Chair sit-to-stand (assisted) | 24.99 s | Sit, hold and return to stand. |
| Light switch | 20.48 s | Wall-switch actuation. |
| Standing mouse drawing | 28.26 s | Desktop drawing with physical mouse. |
Table S8: Six G1 demonstrations spanning five physical task categories.