Tether: Autonomous Functional Play with Correspondence-Driven Trajectory Warping

William Liang; Sam Wang; Hung-Ju Wang; Osbert Bastani; Yecheng Jason Ma; Dinesh Jayaraman

cs.ROMar 3, 2026

Tether: Autonomous Functional Play with Correspondence-Driven Trajectory Warping

Authors: William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Yecheng Jason Ma, Dinesh Jayaraman

Organizations: University of Pennsylvania · University of California, Berkeley · Dyna Robotics

Abstract

The ability to conduct and learn from interaction and experience is a central challenge in robotics, offering a scalable alternative to labor-intensive human demonstrations. However, realizing such "play" requires (1) a policy robust to diverse, potentially out-of-distribution environment states, and (2) a procedure that continuously produces useful robot experience. To address these challenges, we introduce Tether, a method for autonomous functional play involving structured, task-directed interactions. First, we design a novel open-loop policy that warps actions from a small set of source demonstrations (<=10) by anchoring them to semantic keypoint correspondences in the target scene. We show that this design is extremely data-efficient and robust even under significant spatial and semantic variations. Second, we deploy this policy for autonomous functional play in the real world via a continuous cycle of task selection, execution, evaluation, and improvement, guided by the visual understanding capabilities of vision-language models. This procedure generates diverse, high-quality datasets with minimal human intervention. In a household-like multi-object setup, our method is the first to perform many hours of autonomous multi-task play in the real world starting from only a handful of demonstrations. This produces a stream of data that consistently improves the performance of closed-loop imitation policies over time, ultimately yielding over 1000 expert-level trajectories and training policies competitive with those learned from human-collected demonstrations.

Figures & tables

Figure 1: Tether performs autonomous functional play in the real-world for over 24 hours, streaming over 1000 successful trajectories for downstream policy learning.

Figure 2: Demonstration Summaries. Tether summarizes demonstrations into the initial frame, action sequence (red), waypoints (blue), and keypoints.

Figure 3: Policy Inference . During inference, Tether (left) computes correspondences (middle) and produces a warped trajectory action plan (right).

Figure 4: Autonomous Functional Play. Our iterative procedure runs Tether for multiple tasks and uses VLMs for plan generation and success detection.

Figure 5: Evaluation Tasks. Our tasks involving moving fruits and containers with in-distribution (orange) and out-of-distribution (green) objects, as well as challenging manipulation skills (purple).

Figure 6: Main Policy Comparison. We compare the Tether policy with baselines across 12 tasks.

Figure 7: Autonomous Play Statistics. In around 26 hours of play, our method produces over 1000 cumulative trajectories across 6 tasks (left) and significantly expands data diversity, as shown by the heatmaps of object poses at the beginning and end of play (right).

Figure 8: Downstream Policy Learning Results. The stream of data generated by autonomous play consistently improves policy performance over time. Policies trained on human-collected demos, equal in number to the final set of successful Tether-generated trajectories, are shown in black.

Figure 9: Comparison on Play Distribution. We compare Diffusion Policies trained on human-collected demos with Tether policy on environment states encountered during autonomous play.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Pineapple from Shelf	Pineapple to Bowl	Bowl to Shelf	Bowl from Shelf
Tether (10 Demos)	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	0.9 $\pm$ 0.09	1.0 $\pm$ 0.0
Tether (5 Demos)	0.8 $\pm$ 0.13	0.8 $\pm$ 0.13	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
Tether (1 Demo)	0.8 $\pm$ 0.13	0.9 $\pm$ 0.09	0.7 $\pm$ 0.14	1.0 $\pm$ 0.0
$\pi_{0}$ ( Black et al., 2024 )	0.5 $\pm$ 0.16	0.7 $\pm$ 0.14	0.0 $\pm$ 0.0	0.4 $\pm$ 0.15
DP ( Chi et al., 2023 )	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.1 $\pm$ 0.09
Method	Apple to Bowl	Strawberry to Bowl	Pineapple to Basket	Pineapple to Cup

Appendix

Table 1: Success rates and standard error measured over 10 trials.

Figure 10: Computed correspondence for the Pineapple to Cup task.

Figure 11: Computed correspondence for the Strawberry to Bowl task.

Figure 12: Computed correspondence for the Hanging Tape task.

Figure 13: Computed correspondence for the Wiping with Cloth task.

Figure 14: Trajectory warping for the Pineapple to Bowl task.

Figure 15: Trajectory warping for the Wiping with Cloth task.

Figure 16: Environment states with the flipped bowl (left) or pineapple pushed into shelf (right) that required manual interventions.

Method	Pineapple from Shelf	Pineapple to Bowl	Bowl to Shelf	Bowl from Shelf
Tether	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	0.9 $\pm$ 0.09	1.0 $\pm$ 0.0
KALM	0.6 $\pm$ 0.15	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.5 $\pm$ 0.16

Appendix

Table 2: Policy Comparison with KALM. Success rates and standard error measured over 10 trials.

Method	Pineapple from Shelf	Pineapple to Bowl	Bowl to Shelf
$\pi_{0}$ (10 human demos)	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0
$\pi_{0}$ (50 play-generated demos)	0.1 $\pm$ 0.09	0.2 $\pm$ 0.13	0.1 $\pm$ 0.09
$\pi_{0}$ (100 play-generated demos)	0.3 $\pm$ 0.14	0.5 $\pm$ 0.16	0.1 $\pm$ 0.09
$\pi_{0}$ (all play-generated demos)	0.7 $\pm$ 0.14	0.8 $\pm$ 0.13	0.4 $\pm$ 0.15
$\pi_{0}$ (pre-trained)	0.5 $\pm$ 0.16	0.7 $\pm$ 0.14	0.0 $\pm$ 0.0

Appendix

Table 3: Downstream

\pi_{0}

Finetuning Results. Success rates and standard error measured over 10 trials.

Explore similar work

PlayWorld: Learning Robot World Models from Autonomous Play
Mar 9, 2026Tenny Yin, Zhiting Mei, Zhonghe Zheng +8World Models for Robotics Video World Models
Princeton University
Steertuning Policies from a Few Human Videos
Oct 13, 2026Billy Yan, Isaac Tu, Kelly Lee +8Robot Policy Adaptation Data-Efficient Robot Learning
New York University
Human-like autonomy emerges from self-play and a pinch of human data
Jun 11, 2026Daphne Cornelisse, Julian Hunt, Zixu Zhang +4Self-Play RL Autonomous Driving
NYU Tandon School of Engineering · NYU Courant · Princeton University +2

Mar 9, 2026J/K move · Enter open · S save

PlayWorld: Learning Robot World Models from Autonomous Play

Tenny Yin, Zhiting Mei, Zhonghe Zheng, Miyu Yamane+7

Princeton University

World Models for Robotics Video World Models Action-Conditioned World Models Robot Learning Robotic Manipulation

Open paper AskPDF arXiv

Action-conditioned video models offer a promising path to building general-purpose robot simulators that can improve directly from data. Yet, despite training on large-scale robot datasets, current state-of-the-art video models still struggle to predict physically consistent robot-object interactions that are crucial in robotic manipulation. To close this gap, we present PlayWorld, a simple, scalable, and fully autonomous pipeline for training high-fidelity video world simulators from interaction experience. In contrast to prior approaches that rely on success-biased human demonstrations, PlayWorld is the first system capable of learning entirely from unsupervised robot self-play, enabling naturally scalable data collection while capturing complex, long-tailed physical interactions essential for modeling realistic object dynamics. Experiments across diverse manipulation tasks show that PlayWorld generates high-quality, physically consistent predictions for contact-rich interactions that are not captured by world models trained on human-collected data. We further demonstrate the versatility of PlayWorld in enabling fine-grained failure prediction and policy evaluation, with up to 40% improvements over human-collected data. Finally, we demonstrate how PlayWorld enables reinforcement learning in the world model, improving policy performance by 65% in success rates when deployed in the real world.

Abstract

Explore similar work

PlayWorld: Learning Robot World Models from Autonomous Play

Steertuning Policies from a Few Human Videos

Human-like autonomy emerges from self-play and a pinch of human data