Teleoperation remains the default way to collect robot demonstrations, but every episode costs robot and operator time, and every new task starts from scratch. We ask whether a coding agent can take over the teleoperator's job. In our DataFarm, the agent receives a data request and one calibration episode of the robot in the target scene and rebuilds the task in simulation: it writes the scene, an expert controller, and a success judge. The simulator supplies exact actions, and a video model conditioned on real images renders each rollout as photographic video, generating all camera views together consistently. The output is real-ready data: synchronized camera views labelled with the robot's actions, on which a policy can be trained directly and deployed on the real robot. Independent judges certify the expert and every rendered episode before admission. On five real-robot tasks, policies trained only on farm data match policies trained on as many teleoperated episodes (93 vs. 84 successes in 100 trials), and under an unseen tablecloth and coloured light they keep 87 successes where the teleoperation-trained policies fall to 46. Adapting a task is fully automatic and costs $21 of API usage on average, paid once per task, plus $5 for a render contract; every further episode costs $0.80 on average for simulation, rendering, and review, with no robot or operator. One hour of three-view data for a task costs $107 on average, these costs included, slightly below a quote of $117 for teleoperation in China and below $290 in labour alone in North America.
Figures & tables
Figure 1: What the farm delivers. (a) The same frame of a simulator rollout and of our H3 rendering for each of the five tasks; the rendering is photographic and keeps the simulated arm pose, objects, and contacts (insets enlarge the dashed regions). (b) From a request and one calibration episode, the agent builds the task in simulation, and the video model renders each rollout in the original scene or in a new scene described by text. (c) Real-robot successes of policies trained only on farm data or on the same number of teleoperated episodes, in the original scene and under unseen appearance. (d) Mean costs over the five tasks (Section 6.2 ).
Method
Input from the target task
Simulation task built by
Action labels
Observations
Real robot
MimicGen [ 34 ]
a few demos in simulation
people
simulator
simulator rendering
✓
DemoGen [ 49 ]
one real demo
— (no simulator)
edited demo
point clouds
✓
RoboTwin 2.0 [ 6 ]
task description, assets
language model
simulator
rendering + DR
✓
Re 3 Sim [ 16 ]
scene reconstruction
people
simulator
Gaussian splatting
✓
SplatSim [ 37 ]
object and scene scans
people
simulator
Gaussian splatting
✓
Real2Render2Real [ 52 ]
object scans, human video
— (no dynamics)
retargeted
rendered scans
✓
Table 1: Comparison with related data-generation pipelines. Among them, only in DataFarm does an agent build the simulation task from a single calibration episode of the scene, reusing none of its actions, with observations rendered from that episode’s own images as appearance reference.
Figure 2: Overview of DataFarm . (1) A request and one calibration episode of the scene are the only inputs. (2) A coding agent rebuilds the task in simulation and writes an expert against a judge it cannot see; the simulator then runs the expert from many initial states and fixes the actions, object states, and contacts. (3) A video model conditioned on real frames renders every rollout in the scene’s appearance, and a new appearance is a text change. (4) An independent reviewer compares each render with its simulated state, and only admitted episodes enter the dataset. Bottom: the farm pays once per task rather than for every episode, and its data matches teleoperated data on the real robot. Shown on the pull-drawer task; the cost plot is schematic.
Figure 3: The DataFarm pipeline, followed on one drawer episode. The agent builds five intermediate representations: the grounding G , whose quantities are each labelled by how they are known; the task contract T and the expert C , written against a hidden judge J ; the simulated episodes E ; the rendered observations O ; and the admission record q . Each representation is checked by a component other than its producer (seals), and a rejection returns to the stage that caused it (dashed), as for the render here that shows an arm absent from E . The capability layer K is shared by all tasks; everything above it is written for this task.
List of algorithms 1 DataFarm for one task.
Task
What makes it hard
Simulator
Success condition
Real-robot eval.
Stack cups
nesting thin-walled rigid cups
SAPIEN
all three cups nested in one stack
ID, OOD
Weigh apple
placement on a scale with a live display
SAPIEN
apple at rest on the scale pan
ID, OOD
Beat drum
rhythmic contact timing
SAPIEN
drum struck at least twice
ID, OOD
Pull drawer
articulated object; pick, place, and close
SAPIEN
block inside the drawer, drawer closed
ID, OOD
Fold towel
deformable cloth
Isaac Sim
towel folded twice
ID, OOD
Table 2: Tasks. Each starts from one calibration episode with a head and two wrist cameras and a request for 100 episodes; the agent generates the scene, camera fit, expert, judge, and render contract. ID: original scene; OOD: unseen tablecloth and coloured light.
Figure 4: Real-robot results. (a) Successes of π0.5 policies trained on 100 episodes from each source, in 20 trials per task and condition (Total: out of 100), in the original scene (left) and under an unseen tablecloth and coloured light (right). (b) Unseen-scene trials in which the policy trained on teleoperated data fails (top) and the policy trained only on farm data succeeds (bottom).
Stage
Stack cups
Weigh apple
Beat drum
Pull drawer
Fold towel
One-time: grounding to admission (§ 4.2 – 4.4 )
18.22
10.33
19.25
19.40
37.38
100 rollouts and audit
7.43
8.34
7.66
9.31
13.41
Render contract (§ 4.5 ) §
6.11
4.72
4.11
4.49
6.23
Total
31.76
23.39
31.02
33.20
57.02
Table 3: Agent cost of each task, in USD at standard API pricing, by stage of Section 4 : from the calibration episode to 100 simulated episodes and a render contract. One-time: grounding, simulator qualification, task contract, expert, and admission of five simulated episodes, the scope of Figure 5 (b). Rollouts and audit are charged per episode, and the render contract separately (Table 4 ). Task contracts and experts are written by Claude Opus 5.5, in SoL-Pi [ 30 ] . § Excluding canary renders.
Figure 9
Figure 6: Do renders keep the simulated motion? Optical-flow error and share of moving pixels whose flow agrees, between each render and its untextured simulator video (head view). Sim + DR has exactly the simulated motion; the shifted H3 render is paired with the simulator video 0.5 s later. † H3 renders another towel episode, compared with its own simulator video.
Figure 7: Real, simulated, and farm episodes at the same task phase (head view). Sim + DR adds domain-randomized textures and lighting to the simulator frame, and ours renders the simulator frame (renderer in the corner). Real and simulated frames come from different episodes, so object placements differ.
Figure 8: Following the simulation in every view. The same simulated frame rendered by calling the video model directly, with all cameras on one canvas or with each camera separately, and by the farm’s renderer. Each cell shows the head view (top) and the left and right wrist views (bottom). Red boxes mark the views that contradict the simulator or the other views of the same frame; the notes describe the contradiction.
Renderer
Motion ↑
Structure ↑
∣ Lag ∣
Jumps
Direct call, one canvas
0.07
0.16
12.7
11.2
Direct call, each camera
0.04
0.10
14.7
8.8
Edge control only
0.50
0.54
0
30.2
Ours, each camera †
0.55
0.64
0
40.6
Ours
0.74
0.85
0
2.9
Seedance (hosted)
0.73
0.72
0.3
2.3
Table 5: Renderers on the same simulated episode of stack cups, weigh apple, and pull drawer. Motion and structure: correlation with the simulator video of frame-difference energy and of image gradients; lag in frames; jumps per 1,000 frames. Means over the three episodes. † Stack cups only; ours on that episode: 0.75, 0.87, 0, 4.2.
Figure 9: Appearance that the simulator does not model. Crops of a simulator frame (top) and of the farm episode rendered from that frame (middle), with the same object in a real recording at the same task phase (bottom; not the same frame). The simulator draws the scale’s display as a flat texture, the drum as a red cylinder, and the towel as smooth particle cloth; the farm episode shows a lit display with the simulated reading, a lacquered drum with brass tacks, and the weave of the real towel.
read the judge internals or holdout seeds, launch rollouts directly
Render reviewer / verifier auditor
decoded-media review of motion, topology, timing, and appearance
reuse producer summaries as evidence
Pipeline improver
candidate changes to the capability layer
evaluate or activate its own changes
Appendix
Table 6: The seven role instructions (render reviewer and verifier auditor share a row) are loaded explicitly from the capability layer; reviews requested by the agent runtime use the task-verifier instruction. For tasks run by the agent runtime, independence is enforced by separate executions and content-bound review records, not by different names in a JSON field. Reviewers run with the author’s file permissions, so an edit to a reviewed artifact is detected rather than prevented.
Cameras
Appearance
Object
Actions
States
Admission
Coding agent
❍
❍
❍
❍
❍
✗
Calibration episode
⚫
⚫
❍
—
—
—
Qualified simulator
—
✗
⚫
⚫
⚫
—
Renderer
—
❍
✗
✗
✗
—
Real reference image
—
⚫
✗
✗
✗
—
Verifiers, reviewers
—
—
—
—
—
⚫
Appendix
Table 7: Who may certify what: ⚫ certifies, ❍ may propose only, ✗ may not change or certify, — not involved. No component certifies what it produces, and reviewers may admit or reject an episode but cannot relax the task predicates.
Agent and harness
Price
Stack cups
Weigh apple
Beat drum
Mean
Attempts
GPT-6.1 Sol + SoL-Pi
2/10
15.46
8.00
11.54
11.67
5, 5, 18
Claude Opus 5.5 + SoL-Pi
4/20
18.22
10.33
19.25
15.93
5, 6, 10
GPT-6 Astra + SoL-Pi
10/50
60.97
43.19
63.44
55.87
17, 6, 12
Claude Fable 5.1 + Claude Code
10/50
70.60
72.46
54.95
66.01
8, 5, 5
Claude Opus 5.5 + Claude Code
–
30.72
28.23
26.66
28.54
–
GPT-6 Astra + Codex
10/50
151.42
112.16
107.72
123.77
–
Appendix
Table 8: One-time agent cost by agent and harness, in USD at standard API pricing, to five independently admitted simulated episodes. Price: input/output per million tokens. Attempts: capture attempts for the five admitted episodes, in column order, every attempt counted.
Source
Stack cups
Weigh apple
Beat drum
Pull drawer
Fold towel
Real (split half)
-0.02
-0.16
-0.01
-0.00
-0.03
Simulator
2.19
1.95
3.74
1.63
1.41
Sim + DR
2.73
2.69
4.27
2.07
1.58
DataFarm (Seedance)
0.69
1.21
0.38
0.73
0.66
DataFarm (H3)
0.93
1.03
0.27
0.53
0.76
Appendix
Table 9: KID ( ×1000 , lower is closer to real) of head-view frames against real frames, per task. Real frames per task: stack cups 225, weigh apple 32, beat drum 114, pull drawer 1277, fold towel 201. The first row splits the real frames into two halves.
Adapting visuomotor policies to new manipulation tasks often requires substantial manual engineering or teleoperated data collection. Simulation can provide task-specific data at scale, but constructing the scene, designing expert behavior, and configuring data generation still require significant per-task effort. We present ARSTAG, an agentic Real2Sim2Real system that turns a single RGB image and a natural-language instruction directly into robot policy-learning data. A hierarchy of language agents constructs a task-scoped simulation scene, generates robot-feasible demonstrations, and expands the training distribution through task-consistent randomization, while a coordinator agent manages cross-stage feedback and recovery. Across seven manipulation tasks spanning grasping, placement, and stacking, the ARSTAG-generated demonstrations enable sim-to-real transfer of three visuomotor policy architectures to a dual-arm robot, with pi0.5 achieving an average real-world success rate of 74.6%. Ablations show that task-consistent randomization substantially improves robustness, and policy performance increases with generated dataset size. Project webpage: https://boweili666.github.io/ARSTAG/.
Bowei Li, Yuner Zhang, Changliu Liu
Carnegie Mellon University, Pittsburgh, PA 15213, USA
Scaling robot learning requires massive, diverse trajectory data, yet collection is currently bottlenecked by physical teleoperation, where every demonstration binds operator time to specific hardware and workspaces. We introduce digital teleoperation, a paradigm that decouples data collection from physical constraints by replacing the real robot with a generative world model. In this framework, an operator's hand-pose stream drives a robot-centric generative world model to synthesize high-fidelity egocentric videos from a single reference image. The recorded pose stream serves as an embodiment-agnostic action label transferable to any target robot via standard retargeting, yielding complete state-action trajectories for imitation learning independent of physical hardware. We instantiate this paradigm in RynnWorld-Teleop, a system that integrates depth-aware skeletal conditioning, progressive human-to-robot training on a video Diffusion Transformer (DiT), and streaming autoregressive distillation. This pipeline compresses the generative process into a single-pass inference, enabling 40+ FPS, real-time interactive generation on a single H100 GPU. Policies trained exclusively on RynnWorld-Teleop-generated data achieve effective zero-shot Sim2Real transfer across dexterous and diverse bimanual tasks. Moreover, augmenting real-world datasets with our digitally teleoperated data consistently improves success rates, demonstrating that RynnWorld-Teleop serves as a high-fidelity, scalable data engine for the next generation of robotic agents.
Haoyu Zhao, Xingyue Zhao, Hangyu Li +6
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +3
Human demonstrations are today used largely for pretraining vision-language-action models, with little ability to use the same data for target-embodiment or task-oriented training, which scalable deployment of robots in industrial and general environments depends on. A human demonstration recorded on a simple and scalable device such as a mobile phone can serve as post-training data for a robot whose body differs in structure and capability from the human who recorded it. This is made possible not by retargeting the hand's motion but the object's. A capture yields a body-free record: the object's pose over time, its contact intervals, and a reconstructed 3D object with provenance. Any robot, given as a URDF and a short descriptor, is solved against that record and scored in simulation against the demonstrated object motion as ground truth: the human-to-simulation leg of human-to-simulation-to-robot transfer, measured per robot. Each (robot, episode) pair gets one outcome: pass, failure with a named mode, or refusal, with its reason, before any motion. The method runs as a working system over a growing collection; the set analysed here is a sample of it: 42 captures solved on 8 robots, 157 pairs, of which 31.2% pass, 52.2% fail, 14.0% are refused and 2.5% are unauditable (kinematic feasibility, +/-50 mm / +/-15 deg, in simulation). Refusals held up: every pair the checks refuse on stored evidence yet ran anyway failed (13 of 13; 5 on a body whose public model welds its hand). A description file could not be shown to screen a robot: its outcomes were barely more predictable than guessing. No policy has yet been trained on the record. The record decides what a robot may post-train on; whether the policy then works on the real robot is the simulation-to-robot question, answered downstream.