cs.ROOct 14, 2026
SaveHuman Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand
Organizations: Dxtr AI Co.
Abstract
Human demonstrations are today used largely for pretraining vision-language-action models, with little ability to use the same data for target-embodiment or task-oriented training, which scalable deployment of robots in industrial and general environments depends on. A human demonstration recorded on a simple and scalable device such as a mobile phone can serve as post-training data for a robot whose body differs in structure and capability from the human who recorded it. This is made possible not by retargeting the hand's motion but the object's. A capture yields a body-free record: the object's pose over time, its contact intervals, and a reconstructed 3D object with provenance. Any robot, given as a URDF and a short descriptor, is solved against that record and scored in simulation against the demonstrated object motion as ground truth: the human-to-simulation leg of human-to-simulation-to-robot transfer, measured per robot. Each (robot, episode) pair gets one outcome: pass, failure with a named mode, or refusal, with its reason, before any motion. The method runs as a working system over a growing collection; the set analysed here is a sample of it: 42 captures solved on 8 robots, 157 pairs, of which 31.2% pass, 52.2% fail, 14.0% are refused and 2.5% are unauditable (kinematic feasibility, +/-50 mm / +/-15 deg, in simulation). Refusals held up: every pair the checks refuse on stored evidence yet ran anyway failed (13 of 13; 5 on a body whose public model welds its hand). A description file could not be shown to screen a robot: its outcomes were barely more predictable than guessing. No policy has yet been trained on the record. The record decides what a robot may post-train on; whether the policy then works on the real robot is the simulation-to-robot question, answered downstream.
Figures & tables
Figure 1: One phone capture, multiple robots, multiple passes. The demonstrator’s own frame as the grip closes in a pouring episode, and the same moment replayed in simulation by two fixed arms and a mobile manipulator, each with its own grasp; orange marks where the hand held the object. Nothing is retouched. [kinematic feasibility 50 mm / 15 ∘ in simulation]
Figure 2: One demonstration, one body-free record, an answer for every robot. A demonstration is recorded on a phone worn on the body, here the chest. Only what the object did is kept, its path, its contact intervals and its reconstructed 3D object, and that is the ground truth. Any robot given as a URDF and a short descriptor is solved against it, and its replay is scored against the demonstrated motion by object-path error inside the tube. Only a pass enters that robot’s post-training set, and a refusal is decided before any motion. The bodies and the outcomes shown are examples: each outcome is one that a body of that class received in § 6 , and a robot the method has never seen is admitted by filling in the descriptor. [kinematic feasibility 50 mm / 15 ∘ in simulation]
Figure 3: Where a demonstration enters. Human video reaches robot learning mostly at pretraining. The record lets one demonstration enter at post-training, for one robot and one task, which is where the body decides what can be used.
| post-training set | a target-embodiment training set: tens to a few hundred episodes of one robot on one task |
| record | the body-free part of a demonstration: the object’s pose over time, its contact intervals, and its reconstructed 3D object |
| reference motion | the demonstrated object motion, the ground truth every robot is scored against |
| pair | one robot’s answer on one solve run; a capture solved under more than one retargeting mode contributes one pair per solve run |
| outcome | a pair’s answer at the stated tier: pass, failure with a named mode, refusal with a reason, or unauditable |
| tube | the declared tolerance around the reference motion, 50 mm and 15 ∘ |
| tube coverage | the fraction of scored frames inside the tube; a pass needs 85% |
Table 1: Terms used throughout.
Figure 4: The hand as target, the object as target. Copying the hand gives a body of a different build nothing to be measured against. Retargeting the object gives every body the same target, scored the same way.
| Method | Input | Per-robot data | Bodies | Verification | Failures / refusals published |
|---|---|---|---|---|---|
| OKAMI [ 21 ] | 1 RGB-D human video | robot rollouts for the policy | 1 humanoid | hardware | aggregate failure modes |
| RSRD [ 19 ] | 1 video + object scan | none | 1 bimanual | hardware, 90 trials | per-phase and end-to-end rates |
| PhysV2A [ 15 ] | video 6-DoF object motion + RGB-D | kinematic model | 1 arm | hardware, 80 trials, 88.75% | gates grasp–path pairs, then ranks the survivors continuously; decision not published |
| PSI [ 54 ] | RGB-D human videos per task | none (sim as a filter) | 1 arm | hardware, 20 episodes per task | keeps a per-pair feasibility label and trains a scorer on it; not published as a record |
| Human2Sim2Robot [ 24 ] | 1 RGB-D human video | none; RL in simulation per task | 1 arm + dex hand | hardware | per-task success rates only |
| Video2Sim2Real [ 11 ] | 1 video, fixed RGB-D | none | 1 arm + dex hand | hardware, 10 trials per task | per-task misses |
Table 2: Where the record differs. The closest systems and this one, as their sources state them. “Internal” means the system decides and does not publish the decision.
Figure 5: What an episode carries, and what a body adds to it. The record (left, cream) is body-free. It holds the object’s pose over time, the contact intervals with the witness that certified each frame, and one mesh that keeps the observed surface apart from the completion. The executability block (bottom) is an annotation with one row per body, and it never edits the record. An axis nobody measured is hatched and is not read as a pass. Only a pass enters a body’s post-training set. A named failure or a refusal is kept with its certificate and stays out. [kinematic feasibility 50 mm / 15 ∘ in simulation]
Figure 6: Refusal before motion. The pre-flight checks read the description file and the record. The first check that fails ends the pair in a refusal with its certificate, and the robot does not move. A pair that clears every check is solved and replayed.
| reason | quantity measured | what it missed |
|---|---|---|
| object not rigid | held frames that fit the object’s shape at rest | too few, on an object declared deformable |
| no closing joint | actuated joints at the end effector, from the descriptor | none declared |
| no candidate grasps | antipodal pairs in both friction cones, with a clear approach | none survive sampling; the test that emptied the set is named |
| stroke against object width | usable stroke against the narrowest candidate | every candidate is wider |
| object against arm clearance | object depth inside a non-jaw link | over 5 mm for every candidate |
| no feasible stance | a wheeled base’s clearance from the scene | inside it at every stance |
Table 3: The refusal vocabulary. Each reason, the quantity its certificate carries and the limit that quantity missed, in the order of Fig. 6 . No check reads a replay.
| property | read by | decides |
|---|---|---|
| closing joints | candidate synthesis | a refusal when none is declared; a closure is never faked |
| usable stroke | width filter | a stroke-against-width refusal when no candidate fits |
| reach radius | coverage and placement | a sweep-against-reach refusal when no stance covers the interval |
| wrist actuation | target construction | orientation commanded, or position tracked and orientation reported |
| base type | placement search | whether the stance may also move vertically, or follow a base path |
| joint count | the solver’s null space | how much freedom is left once the pose is fixed |
Table 4: The body descriptor. What the method reads out of a body description beyond the model’s own joint list, joint limits and rest pose, what reads it, and what it can decide. A property the description omits still produces an answer.
Figure 7: Capture to policy, and the two exits. Only a passing trajectory enters the body’s post-training set. A named failure and a refusal are kept with their certificates and go no further. The refusal is decided from solve-time artefacts, before any motion is generated. The export and post-training stages are drawn to show where the record sits in the chain. This paper’s measurements stop at the gate.
Figure 8: One path, two bodies, two grasps. The demonstrated object path is the same for both arms and is never rescaled. Each body picks the grasp that keeps the most of that path solvable under strict inverse kinematics, so sweep coverage, not geometric quality, ranks the grasps. Conditioning a grasp on the motion that follows is prior work [ 9 , 1 , 15 , 54 ] . The sweep-coverage statistic and the outcome kept per robot are ours.
Figure 9: Touch opens a contact, and co-motion may only extend it. The object-side witness certifies frames that a fingertip distance cannot, as in a raised carry. It is clamped to the touch-only onset, so the first frame of an interval is exactly what it was before the witness existed. Over the 3 517 scored contact frames of the 79 pairs that carry the witness split, 6.7% are held by co-motion alone.
Figure 10: The human-to-sim gap, measured. The demonstrated object path is the ground truth and the tube is the declared tolerance. Coverage is the fraction of scored frames inside the tube, and a pass needs at least 85%. The inner dashed band is the strict tube, a tighter reference tolerance recorded beside every outcome as strict coverage; the outcome itself is decided on the declared tube.
| body | class | DoF | end effector | stroke (mm) | reach (m) | base | pairs | passed | refused |
|---|---|---|---|---|---|---|---|---|---|
| Franka Panda | fixed arm | 7 | parallel jaw | 80 | 0.85 | fixed | 45 | 18 | 5 |
| PR2 | mobile manipulator | 7 | parallel jaw | 80 | 0.82 | mobile | 20 | 8 | 1 |
| myCobot 280 | fixed arm | 6 | parallel jaw | 45 | 0.30 | fixed | 19 | 2 | 3 |
| Unitree G1 | humanoid | 7 | multi-finger hand | 90 | 0.41 | mobile | 19 | 3 | 0 |
| UR5e | fixed arm | 6 | parallel jaw | 85 | 0.85 | fixed | 19 | 7 | 1 |
| Apptronik Apollo | humanoid | 7 | welded hand | – | 0.53 | mobile | 17 | 0 | 11 |
Table 5: The robots studied. One row per body, with the properties the method reads out of its description and the outcomes it was given. The method never reads the first column. One humanoid has a three-fingered hand read as a closure with its effective stroke; the other wears a five-fingered hand that its published model welds, with no finger joint, so it declares no closing joint. [kinematic feasibility 50 mm / 15 ∘ in simulation]
Figure 11: Outcomes by task family and body class. Every pair that was offered has an answer here. A body here is a robot on a given base, so an arm solved standing on the wheeled cart counts as a mobile manipulator. The corpus fills 40.5% of the capture-by-body matrix; a class and family that received no pair is labelled as such rather than drawn as a zero. The refusal band shows the pairs that other pipelines discard silently. [kinematic feasibility 50 mm / 15 ∘ in simulation]
Figure 12: Robot-minutes per capture-minute. Of 157 answered pairs, 131 are attempted and 49 pass. The 19.2 minutes of recorded takes credit 6.1 robot-minutes at the declared tube. This is the cost side of the gate, in the one unit used throughout. [kinematic feasibility 50 mm / 15 ∘ in simulation]
| ablation | arm | pairs | pass | median coverage | median path ratio |
|---|---|---|---|---|---|
| target: object path vs hand copy | object path | 84 | 28 / 84 (33%) | 100.0% | 1.00 |
| hand copy (wrist path), scored on the same object tube | 84 | no outcome | 18.7% | 1.56 | |
| grasp: coverage-first vs best quality | coverage-first (shipped) | 84 | 28 / 84 (33%) | 100.0% | 1.00 |
| grasp ranked by geometric quality only (sweep coverage ignored) | 84 | 11 / 84 (13%) | 85.9% | 1.13 | |
| contact: full witness vote vs touch only | full vote | 55 | 25 / 55 (45%) | 100.0% | – |
| touch only | 55 | 25 / 55 (45%) | 100.0% | – |
Table 6: Ablations. Each construction is compared with its alternative, re-solved on the same pairs and scored the same way (the contact vote is re-scored, not re-solved), on 2 fixed arms over 42 episodes. These runs are kept apart from the main ledger. A hand copy earns no outcome in this scoring because it does not target the reference motion. Its coverage is measured on the same tube. Coverage is the median tube coverage; path ratio is the median length of the replayed object path relative to the demonstrated one. [kinematic feasibility 50 mm / 15 ∘ in simulation]
Figure 13: Refusals, by reason. The refusal vocabulary, counted. Failure modes among attempts are in the supplement. [kinematic feasibility 50 mm / 15 ∘ in simulation]
Figure 14: Actions from the record, observations from the robot. A passing, gated trajectory supplies the joint targets and the gripper command. Replaying it on the robot with its own cameras supplies the camera frames and joint readings. With the task described in words, these make one post-training episode.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| cause class | superseded pairs |
|---|---|
| a pre-flight check now refuses the pair | 10 |
| bundle produced again and re-scored | 25 |
| same artefact, stricter check | 0 |
Table 7: Superseded passes by cause class.
Figure 15: One episode, several bodies, each with its own answer. One pouring episode, shown mid-transport, with the demonstrated object path in teal; the leftmost tile is the phone’s own frame at that moment. Each body chose its own grasp and stance against the same path, which is never rescaled. Scale bars differ because each tile is fitted to its body. Each tile’s outcome is read from the ledger at build time: a pass, or an attempt that failed with its named mode. The thin ring marks the height of the work surface. [kinematic feasibility 50 mm / 15 ∘ in simulation]
Figure 16: From the phone to the replay. One pouring episode at three moments: the demonstrator’s grip closing on the object, the carry and tilt over the plant, and the release. Above, the phone’s own frame. Below, the same moment in the simulated replay of a fixed arm against the record: the capture mesh, the reconstructed object on its demonstrated path, and the demonstrator’s hand as contact marks where it held the object. The hand is a witness to contact and is never copied; the arm’s jaw closes elsewhere on the object, where the path stays reachable for that body. This body passed the episode. [kinematic feasibility 50 mm / 15 ∘ in simulation]
Figure 17: The grasp is the body’s own. The same pouring episode at the moment the jaw closes, on four bodies that all passed it; the demonstrator’s hand at that moment is drawn where it held the object. Each body closes its jaw where its own kinematics keep the demonstrated path reachable, ranked by sweep coverage (§ 5.2 ): near the top, low on the body or across the middle. None copies the hand. Each tile is fitted to its body, so read the scale bars. [kinematic feasibility 50 mm / 15 ∘ in simulation]