Scaling robot learning is bottlenecked not only by data volume, but also by whether the data targets the policy's failures. Existing robot-free handheld interfaces enable portable in-the-wild data collection, yet remain open-loop: users record demonstrations without feedback from the current policy. Interactive methods such as DAgger provide such feedback by collecting corrective on-policy data, but require deploying imperfect policies on physical robots, making them costly, unsafe, and difficult to scale. We present RoboPocket, a smartphone-based system for robot-free policy iteration. By combining on-device sensing, display, and communication with cloud-side policy inference, RoboPocket provides real-time collection guidance and visualizes the policy's predicted trajectory in augmented reality as AR Visual Foresight. This allows users to identify policy weaknesses and collect targeted corrective data without deploying a physical robot. An asynchronous finetuning pipeline continuously incorporates new corrections and updates the policy within minutes, realizing a DAgger-inspired robot-free correction loop. Across diverse real-world manipulation tasks, we validate RoboPocket as a scalable data engine and show that online corrective data collection improves sample efficiency by up to 2× over scaling offline demonstrations. Project page and videos: https://robo-pocket.github.io.
Figures & tables
Figure 1: Unlike previous workflows (left) relying on prolonged offline feedback loops with physical robots, RoboPocket (right) enables instant policy updates in distributed environments using consumer smartphones. By visualizing the policy’s intent via AR Visual Foresight, users can proactively identify weaknesses and provide corrective data that refines the policy in minutes.
Figure 2: RoboPocket System Design. (a) Hardware Design: RoboPocket combines a consumer smartphone with a low-cost, 3D-printed adaptive gripper that is isomorphic to the Robotiq 2F-85. A fisheye lens expands the visual field of view, while an external gripper-width sensor captures proprioceptive information. (b) Software Interface: The iOS application provides real-time feedback for tracking quality and kinematic validity, visualizes recorded or predicted trajectories in AR, and synchronizes multiple devices for bimanual collection.
Figure 3: Overview of Robot-Free Instant Policy Iteration. (Left) Using AR Visual Foresight, the user identifies policy weaknesses (OOD states) and proactive failures in the real world, which improves the policy instantly. (Right) Collected corrective data is immediately streamed to the Data Serving Node . The Training Server performs online finetuning using RLPD sampling [ 2 ] and pushes updated weights to the Inference Server . The improved policy predictions are streamed back to iPhone in real-time ( <150 ms), enabling continuous, robot-free policy improvement in minutes .
Figure 5: Evaluation Tasks. a) We evaluate our method on four manipulation tasks: Block Sorting, Seasoning Pouring, Towel Folding, and Snack Bagging. b) Mouse Arrangement validates data fidelity and scaling; the four images are keyframes from a single rollout. Evaluation uses diverse novel object pairs, scenes, and initial poses. c) To assess distributed adaptation, 4 users collect data and perform in-the-wild policy iteration across distinct environments for the Block Sorting task.
Figure 6: RoboPocket Localization Accuracy. We measure cumulative 3D Euclidean trajectory error against robot kinematic ground truth.
Figure 7: Data Scaling Trend Validation. On the Mouse Arrangement task, policy performance shows strong power-law correlations with environment-object diversity ( r=−0.962 and r=−0.882 ), and consistently improves with more training data.
Figure 8: Performance Comparison. Across four manipulation tasks, IL + Instant PI improves data efficiency by up to 2 × over pure offline scaling and achieves performance comparable to manual policy iteration without requiring physical robot deployment.
Figure 9: Distributed Policy Iteration. RoboPocket enables users to improve policy performance across four distinct environments with only 12 interactive corrections per scene.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Participant Demographics. The distribution of volunteers based on their prior experience with imitation learning and UMI data collection.
Figure 11: User Study Results. Visualization of user feedback regarding the helpfulness of Real-time Feedback, Virtual Foresight, and the Instant Policy Iteration workflow.
Figure 12: State Coverage Visualization. PCA projection of DINOv2 features comparing the coverage of data collected by non-experts using RoboPocket versus expert data.
Figure 13: Quantitative State Coverage Analysis. Cross-dataset KNN coverage between non-expert and expert data in the PCA feature space.
Figure 14: Training environments for verifying Data Scaling Laws. These environments encompass a wide range of lighting conditions and textures.
Figure 15: Training object pairs for verifying the Data Scaling Laws. We combine mice and mouse pads to obtain a diverse set of object pairs.
Figure 16: Robot Evaluation Setup. The setup includes a movable lifting table, a Flexiv Rizon 4 robot with a Robotiq 2F-85 gripper, and a workstation powered by a mobile power station.
Hyperparameter
Value
Observation Horizon ( Tobs )
1
Action Prediction Horizon ( Tpred )
16
Action Execution Horizon ( Texec )
8
Training Epochs
600
Batch Size
64
Optimizer
AdamW
Appendix
Table 3: Hyperparameters for Policy Training
Hyperparameter
Value
Batch Size
32
Learning Rate
1.0×10−4
Observation Encoder Learning Rate
1.0×10−5
Learning Rate Schedule
Constant
Model Sync Interval N
100 Steps
Appendix
Table 4: Hyperparameters for Robot-free Instant Policy Iteration
Learning from human demonstrations is a reliable way to teach robots new tasks, but the gains from each additional demonstration shrink as the policy improves. Continued improvement can instead come from supervised deployment, where an operator places the objects and intervenes when the policy fails. We ask how to maximize improvement from a fixed budget of supervised episodes on high-precision manipulation tasks with wide ranges of object placements. We observe that failures can concentrate in a small subset of initial states, so uniform collection spends much of the operator's time on states the policy already handles. Mulligan makes the initial-state distribution a decision, starting each round's episodes at observed failures and untried states. To further improve data efficiency, we augment interactive imitation learning with a value function trained on all data, including failures that imitation discards. Across three real-world tasks evaluated on 2,550 held-out, blinded episodes and two simulated tasks, Mulligan outperforms uniform initial-state sampling at matched collection budgets, and combined with value-based action selection, HiL-IDQL+Mulligan, improves final real-task success by 10-34 percentage points. With operator interventions, the human-robot team completes 98% of collection episodes, remaining productive while the policy learns. Videos, code, and data are available at https://mulligan.page/.
Collecting human demonstrations via teleoperation is a common approach for teaching robots task-specific skills. However, when only a limited number of demonstrations are available, policies are prone to entering out-of-distribution (OOD) states due to compounding errors or environmental stochasticity. Existing interactive imitation learning or human-in-the-loop methods try to address this issue by following the Human-Gated DAgger (HG-DAgger) paradigm, an approach that augments demonstrations through selective human intervention during policy execution. Nevertheless, these approaches struggle to balance dexterity and generality: they either provide fine-grained corrections but are limited to specific kinematic structures, or achieve generality at the cost of precise control. To overcome this limitation, we propose the Human-Robot Copilot framework that can leverage a scaling factor for dexterous teleoperation while maintaining compatibility with a wide range of industrial and research manipulators. Experimental results demonstrate that our framework achieves higher performance with the same number of demonstration trajectories. Moreover, since corrective interventions are required only intermittently, the overall data collection process is more efficient and less time-consuming.
Learning generalizable robot manipulation policies requires large-scale and diverse interaction data, yet collecting real-world demonstrations remains costly and difficult to scale. Existing approaches to data collection are either dependent on specific robot hardware that limits crowdsourcing and transferability, or suffer from incomplete annotation and limited behavioral diversity. Inspired by how games sustain long-term human engagement, we explore an alternative paradigm that turns data collection into an engaging gameplay experience and transfers the resulting human manipulation experience to real robots. We present Project Kitchen, a VR-based gamified egocentric data collection platform that elicits diverse, goal-directed manipulation while remaining independent of specific robot embodiments and hardware, making it applicable to broader and potentially large-scale deployment. To bridge the game-to-real gap, we further introduce Game2Policy, which extracts embodiment-invariant affordance cues, including contact points and sub-goal states, from gameplay trajectories. An affordance model is pre-trained on game-collected data and then jointly fine-tuned with downstream policies using only a handful of real-robot demonstrations. Experiments show that Game2Policy improves average success rates by 10.0 points in simulation and 18.3 points on real robots in the few-shot setting. User studies and quantitative analyses further show that Project Kitchen promotes diverse manipulation behaviors and provides an engaging data collection experience. These results demonstrate the potential of gamified virtual environments as a scalable source of manipulation knowledge. The platform and code will be released upon acceptance.
Zheng Li, Liang Zhu, Junzhe Wang +13
The Hong Kong University of Science and Technology (Guangzhou). · The University of Hong Kong. · Tsinghua University.