Scaling robot learning is bottlenecked not only by data volume, but also by whether the data targets the policy's failures. Existing robot-free handheld interfaces enable portable in-the-wild data collection, yet remain open-loop: users record demonstrations without feedback from the current policy. Interactive methods such as DAgger provide such feedback by collecting corrective on-policy data, but require deploying imperfect policies on physical robots, making them costly, unsafe, and difficult to scale. We present RoboPocket, a smartphone-based system for robot-free policy iteration. By combining on-device sensing, display, and communication with cloud-side policy inference, RoboPocket provides real-time collection guidance and visualizes the policy's predicted trajectory in augmented reality as AR Visual Foresight. This allows users to identify policy weaknesses and collect targeted corrective data without deploying a physical robot. An asynchronous finetuning pipeline continuously incorporates new corrections and updates the policy within minutes, realizing a DAgger-inspired robot-free correction loop. Across diverse real-world manipulation tasks, we validate RoboPocket as a scalable data engine and show that online corrective data collection improves sample efficiency by up to 2× over scaling offline demonstrations. Project page and videos: https://robo-pocket.github.io.
Figures & tables
Figure 1: Unlike previous workflows (left) relying on prolonged offline feedback loops with physical robots, RoboPocket (right) enables instant policy updates in distributed environments using consumer smartphones. By visualizing the policy’s intent via AR Visual Foresight, users can proactively identify weaknesses and provide corrective data that refines the policy in minutes.
Figure 2: RoboPocket System Design. (a) Hardware Design: RoboPocket combines a consumer smartphone with a low-cost, 3D-printed adaptive gripper that is isomorphic to the Robotiq 2F-85. A fisheye lens expands the visual field of view, while an external gripper-width sensor captures proprioceptive information. (b) Software Interface: The iOS application provides real-time feedback for tracking quality and kinematic validity, visualizes recorded or predicted trajectories in AR, and synchronizes multiple devices for bimanual collection.
Figure 3: Overview of Robot-Free Instant Policy Iteration. (Left) Using AR Visual Foresight, the user identifies policy weaknesses (OOD states) and proactive failures in the real world, which improves the policy instantly. (Right) Collected corrective data is immediately streamed to the Data Serving Node . The Training Server performs online finetuning using RLPD sampling [ 2 ] and pushes updated weights to the Inference Server . The improved policy predictions are streamed back to iPhone in real-time ( <150 ms), enabling continuous, robot-free policy improvement in minutes .
Figure 5: Evaluation Tasks. a) We evaluate our method on four manipulation tasks: Block Sorting, Seasoning Pouring, Towel Folding, and Snack Bagging. b) Mouse Arrangement validates data fidelity and scaling; the four images are keyframes from a single rollout. Evaluation uses diverse novel object pairs, scenes, and initial poses. c) To assess distributed adaptation, 4 users collect data and perform in-the-wild policy iteration across distinct environments for the Block Sorting task.
Figure 6: RoboPocket Localization Accuracy. We measure cumulative 3D Euclidean trajectory error against robot kinematic ground truth.
Figure 7: Data Scaling Trend Validation. On the Mouse Arrangement task, policy performance shows strong power-law correlations with environment-object diversity ( r=−0.962 and r=−0.882 ), and consistently improves with more training data.
Figure 8: Performance Comparison. Across four manipulation tasks, IL + Instant PI improves data efficiency by up to 2 × over pure offline scaling and achieves performance comparable to manual policy iteration without requiring physical robot deployment.
Figure 9: Distributed Policy Iteration. RoboPocket enables users to improve policy performance across four distinct environments with only 12 interactive corrections per scene.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Participant Demographics. The distribution of volunteers based on their prior experience with imitation learning and UMI data collection.
Figure 11: User Study Results. Visualization of user feedback regarding the helpfulness of Real-time Feedback, Virtual Foresight, and the Instant Policy Iteration workflow.
Figure 12: State Coverage Visualization. PCA projection of DINOv2 features comparing the coverage of data collected by non-experts using RoboPocket versus expert data.
Figure 13: Quantitative State Coverage Analysis. Cross-dataset KNN coverage between non-expert and expert data in the PCA feature space.
Figure 14: Training environments for verifying Data Scaling Laws. These environments encompass a wide range of lighting conditions and textures.
Figure 15: Training object pairs for verifying the Data Scaling Laws. We combine mice and mouse pads to obtain a diverse set of object pairs.
Figure 16: Robot Evaluation Setup. The setup includes a movable lifting table, a Flexiv Rizon 4 robot with a Robotiq 2F-85 gripper, and a workstation powered by a mobile power station.
Hyperparameter
Value
Observation Horizon ( Tobs )
1
Action Prediction Horizon ( Tpred )
16
Action Execution Horizon ( Texec )
8
Training Epochs
600
Batch Size
64
Optimizer
AdamW
Appendix
Table 3: Hyperparameters for Policy Training
Hyperparameter
Value
Batch Size
32
Learning Rate
1.0×10−4
Observation Encoder Learning Rate
1.0×10−5
Learning Rate Schedule
Constant
Model Sync Interval N
100 Steps
Appendix
Table 4: Hyperparameters for Robot-free Instant Policy Iteration