cs.RONov 3, 2026
SavePyRUA-DataFarm: LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data
Organizations: Peking University · NVIDIA
Abstract
Teleoperation remains the default way to collect robot demonstrations, but every episode costs robot and operator time, and every new task starts from scratch. We ask whether a coding agent can take over the teleoperator's job. In our DataFarm, the agent receives a data request and one calibration episode of the robot in the target scene and rebuilds the task in simulation: it writes the scene, an expert controller, and a success judge. The simulator supplies exact actions, and a video model conditioned on real images renders each rollout as photographic video, generating all camera views together consistently. The output is real-ready data: synchronized camera views labelled with the robot's actions, on which a policy can be trained directly and deployed on the real robot. Independent judges certify the expert and every rendered episode before admission. On five real-robot tasks, policies trained only on farm data match policies trained on as many teleoperated episodes (93 vs. 84 successes in 100 trials), and under an unseen tablecloth and coloured light they keep 87 successes where the teleoperation-trained policies fall to 46. Adapting a task is fully automatic and costs $21 of API usage on average, paid once per task, plus $5 for a render contract; every further episode costs $0.80 on average for simulation, rendering, and review, with no robot or operator. One hour of three-view data for a task costs $107 on average, these costs included, slightly below a quote of $117 for teleoperation in China and below $290 in labour alone in North America.
Figures & tables
Figure 1: What the farm delivers. (a) The same frame of a simulator rollout and of our H3 rendering for each of the five tasks; the rendering is photographic and keeps the simulated arm pose, objects, and contacts (insets enlarge the dashed regions). (b) From a request and one calibration episode, the agent builds the task in simulation, and the video model renders each rollout in the original scene or in a new scene described by text. (c) Real-robot successes of policies trained only on farm data or on the same number of teleoperated episodes, in the original scene and under unseen appearance. (d) Mean costs over the five tasks (Section 6.2 ).
| Method | Input from the target task | Simulation task built by | Action labels | Observations | Real robot |
| MimicGen [ 34 ] | a few demos in simulation | people | simulator | simulator rendering | ✓ |
| DemoGen [ 49 ] | one real demo | — (no simulator) | edited demo | point clouds | ✓ |
| RoboTwin 2.0 [ 6 ] | task description, assets | language model | simulator | rendering + DR | ✓ |
| Re 3 Sim [ 16 ] | scene reconstruction | people | simulator | Gaussian splatting | ✓ |
| SplatSim [ 37 ] | object and scene scans | people | simulator | Gaussian splatting | ✓ |
| Real2Render2Real [ 52 ] | object scans, human video | — (no dynamics) | retargeted | rendered scans | ✓ |
Table 1: Comparison with related data-generation pipelines. Among them, only in DataFarm does an agent build the simulation task from a single calibration episode of the scene, reusing none of its actions, with observations rendered from that episode’s own images as appearance reference.
Figure 2: Overview of DataFarm . (1) A request and one calibration episode of the scene are the only inputs. (2) A coding agent rebuilds the task in simulation and writes an expert against a judge it cannot see; the simulator then runs the expert from many initial states and fixes the actions, object states, and contacts. (3) A video model conditioned on real frames renders every rollout in the scene’s appearance, and a new appearance is a text change. (4) An independent reviewer compares each render with its simulated state, and only admitted episodes enter the dataset. Bottom: the farm pays once per task rather than for every episode, and its data matches teleoperated data on the real robot. Shown on the pull-drawer task; the cost plot is schematic.
Figure 3: The DataFarm pipeline, followed on one drawer episode. The agent builds five intermediate representations: the grounding , whose quantities are each labelled by how they are known; the task contract and the expert , written against a hidden judge ; the simulated episodes ; the rendered observations ; and the admission record . Each representation is checked by a component other than its producer (seals), and a rejection returns to the stage that caused it (dashed), as for the render here that shows an arm absent from . The capability layer is shared by all tasks; everything above it is written for this task.
List of algorithms 1 DataFarm for one task.
| Task | What makes it hard | Simulator | Success condition | Real-robot eval. |
| Stack cups | nesting thin-walled rigid cups | SAPIEN | all three cups nested in one stack | ID, OOD |
| Weigh apple | placement on a scale with a live display | SAPIEN | apple at rest on the scale pan | ID, OOD |
| Beat drum | rhythmic contact timing | SAPIEN | drum struck at least twice | ID, OOD |
| Pull drawer | articulated object; pick, place, and close | SAPIEN | block inside the drawer, drawer closed | ID, OOD |
| Fold towel | deformable cloth | Isaac Sim | towel folded twice | ID, OOD |
Table 2: Tasks. Each starts from one calibration episode with a head and two wrist cameras and a request for 100 episodes; the agent generates the scene, camera fit, expert, judge, and render contract. ID: original scene; OOD: unseen tablecloth and coloured light.
Figure 4: Real-robot results. (a) Successes of policies trained on 100 episodes from each source, in 20 trials per task and condition (Total: out of 100), in the original scene (left) and under an unseen tablecloth and coloured light (right). (b) Unseen-scene trials in which the policy trained on teleoperated data fails (top) and the policy trained only on farm data succeeds (bottom).
| Stage | Stack cups | Weigh apple | Beat drum | Pull drawer | Fold towel |
| One-time: grounding to admission (§ 4.2 – 4.4 ) | 18.22 | 10.33 | 19.25 | 19.40 | 37.38 |
| 100 rollouts and audit | 7.43 | 8.34 | 7.66 | 9.31 | 13.41 |
| Render contract (§ 4.5 ) § | 6.11 | 4.72 | 4.11 | 4.49 | 6.23 |
| Total | 31.76 | 23.39 | 31.02 | 33.20 | 57.02 |
Table 3: Agent cost of each task, in USD at standard API pricing, by stage of Section 4 : from the calibration episode to 100 simulated episodes and a render contract. One-time: grounding, simulator qualification, task contract, expert, and admission of five simulated episodes, the scope of Figure 5 (b). Rollouts and audit are charged per episode, and the render contract separately (Table 4 ). Task contracts and experts are written by Claude Opus 5.5, in SoL-Pi [ 30 ] . § Excluding canary renders.
Figure 6: Do renders keep the simulated motion? Optical-flow error and share of moving pixels whose flow agrees, between each render and its untextured simulator video (head view). Sim + DR has exactly the simulated motion; the shifted H3 render is paired with the simulator video 0.5 s later. † H3 renders another towel episode, compared with its own simulator video.
Figure 7: Real, simulated, and farm episodes at the same task phase (head view). Sim + DR adds domain-randomized textures and lighting to the simulator frame, and ours renders the simulator frame (renderer in the corner). Real and simulated frames come from different episodes, so object placements differ.
Figure 8: Following the simulation in every view. The same simulated frame rendered by calling the video model directly, with all cameras on one canvas or with each camera separately, and by the farm’s renderer. Each cell shows the head view (top) and the left and right wrist views (bottom). Red boxes mark the views that contradict the simulator or the other views of the same frame; the notes describe the contradiction.
| Renderer | Motion | Structure | Lag | Jumps |
| Direct call, one canvas | 0.07 | 0.16 | 12.7 | 11.2 |
| Direct call, each camera | 0.04 | 0.10 | 14.7 | 8.8 |
| Edge control only | 0.50 | 0.54 | 0 | 30.2 |
| Ours, each camera † | 0.55 | 0.64 | 0 | 40.6 |
| Ours | 0.74 | 0.85 | 0 | 2.9 |
| Seedance (hosted) | 0.73 | 0.72 | 0.3 | 2.3 |
Table 5: Renderers on the same simulated episode of stack cups, weigh apple, and pull drawer. Motion and structure: correlation with the simulator video of frame-difference energy and of image gradients; lag in frames; jumps per 1,000 frames. Means over the three episodes. † Stack cups only; ours on that episode: 0.75, 0.87, 0, 4.2.
Figure 9: Appearance that the simulator does not model. Crops of a simulator frame (top) and of the farm episode rendered from that frame (middle), with the same object in a real recording at the same task phase (bottom; not the same frame). The simulator draws the scale’s display as a flat texture, the drum as a red cylinder, and the towel as smooth particle cloth; the farm episode shows a lit display with the simulated reading, a lacquered drum with brass tacks, and the weave of the real towel.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Role | Produces | Cannot |
| Orchestrator | grounding, task artifacts, render contract, batches, delivery | sign its own reviews or delivery |
| Task designer | task contract, stages, randomization axes | author the expert |
| Task verifier | frozen judge spec and negative controls | be edited by the expert author |
| Expert author | expert controller inside the harness | read the judge internals or holdout seeds, launch rollouts directly |
| Render reviewer / verifier auditor | decoded-media review of motion, topology, timing, and appearance | reuse producer summaries as evidence |
| Pipeline improver | candidate changes to the capability layer | evaluate or activate its own changes |
Table 6: The seven role instructions (render reviewer and verifier auditor share a row) are loaded explicitly from the capability layer; reviews requested by the agent runtime use the task-verifier instruction. For tasks run by the agent runtime, independence is enforced by separate executions and content-bound review records, not by different names in a JSON field. Reviewers run with the author’s file permissions, so an edit to a reviewed artifact is detected rather than prevented.
| Cameras | Appearance | Object | Actions | States | Admission | |
| Coding agent | ❍ | ❍ | ❍ | ❍ | ❍ | ✗ |
| Calibration episode | ⚫ | ⚫ | ❍ | — | — | — |
| Qualified simulator | — | ✗ | ⚫ | ⚫ | ⚫ | — |
| Renderer | — | ❍ | ✗ | ✗ | ✗ | — |
| Real reference image | — | ⚫ | ✗ | ✗ | ✗ | — |
| Verifiers, reviewers | — | — | — | — | — | ⚫ |
Table 7: Who may certify what: ⚫ certifies, ❍ may propose only, ✗ may not change or certify, — not involved. No component certifies what it produces, and reviewers may admit or reject an episode but cannot relax the task predicates.
| Agent and harness | Price | Stack cups | Weigh apple | Beat drum | Mean | Attempts |
| GPT-6.1 Sol + SoL-Pi | 2/10 | 15.46 | 8.00 | 11.54 | 11.67 | 5, 5, 18 |
| Claude Opus 5.5 + SoL-Pi | 4/20 | 18.22 | 10.33 | 19.25 | 15.93 | 5, 6, 10 |
| GPT-6 Astra + SoL-Pi | 10/50 | 60.97 | 43.19 | 63.44 | 55.87 | 17, 6, 12 |
| Claude Fable 5.1 + Claude Code | 10/50 | 70.60 | 72.46 | 54.95 | 66.01 | 8, 5, 5 |
| Claude Opus 5.5 + Claude Code | – | 30.72 | 28.23 | 26.66 | 28.54 | – |
| GPT-6 Astra + Codex | 10/50 | 151.42 | 112.16 | 107.72 | 123.77 | – |
Table 8: One-time agent cost by agent and harness, in USD at standard API pricing, to five independently admitted simulated episodes. Price: input/output per million tokens. Attempts: capture attempts for the five admitted episodes, in column order, every attempt counted.
| Source | Stack cups | Weigh apple | Beat drum | Pull drawer | Fold towel |
| Real (split half) | -0.02 | -0.16 | -0.01 | -0.00 | -0.03 |
| Simulator | 2.19 | 1.95 | 3.74 | 1.63 | 1.41 |
| Sim + DR | 2.73 | 2.69 | 4.27 | 2.07 | 1.58 |
| DataFarm (Seedance) | 0.69 | 1.21 | 0.38 | 0.73 | 0.66 |
| DataFarm (H3) | 0.93 | 1.03 | 0.27 | 0.53 | 0.76 |
Table 9: KID ( , lower is closer to real) of head-view frames against real frames, per task. Real frames per task: stack cups 225, weigh apple 32, beat drum 114, pull drawer 1277, fold towel 201. The first row splits the real frames into two halves.
More in this topic
Robotic Data Collection
Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.
See all 186 papers