cs.ROMar 10, 2026
SaveSee, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation
Organizations: School of Information Science and Technology, University of Science and Technology of China, Anhui, China. · University of Technology Sydney, Ultimo, NSW, Australia. · Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates. · The University of Hong Kong, Hong Kong, China. · Institute of AI for Industry, Chinese Academy of Sciences, Jiangsu, China. · School of Intelligent Science and Engineering, Harbin Institute of Technology (Shenzhen), Guangdong, China.
Abstract
Measurement of task progress through explicit, actionable milestones is critical for robust robotic manipulation. This progress awareness enables a model to ground its current task status, anticipate verifiable intermediate states, and detect and recover from failures when progress stalls. To embody this capability, we introduce See, Plan, Rewind (SPR), a progress-aware vision-language-action framework that dynamically grounds language instructions into a sequence of spatial subgoals. SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure by monitoring progress against the expected sequence. This closed-loop approach enables robust error correction without requiring additional training data or auxiliary models. Extensive experiments demonstrate the framework's effectiveness, generalization and robustness: SPR outperforms the MolmoAct baseline by 5% on the LIBERO benchmark. On the challenging LIBERO-Plus benchmark with unseen instructions and initial states, SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA, demonstrating superior out-of-distribution robustness.
Figures & tables
Fig. 1 : See-Plan-Rewind framework’s closed-loop execution workflow. Starting from the initial state ( top-left ), the model performs See-Plan reasoning to generate actions (arrow to top-right ), which visualizes the subtask decomposition and trajectory planning. Under normal execution, the loop returns to top-left. When progress monitoring detects failures ( bottom-right ), the Rewind mechanism activates ( bottom-left ), returning the robot to the initial state before resuming the loop. This closed-loop design enables autonomous error recovery through explicit progress awareness.
Fig. 2 : SPR framework overview. Left : Upon receiving the task description and observation, the model performs See-Plan reasoning(Sec. III-A ), which identifies remaining subtasks with 2D spatial coordinates (See) and plans a gripper trajectory to the next subtask waypoint (Plan), then outputs action tokens for execution. Each inference step also updates the state recorder, where denotes the predicted subtask count and the planned 2D trajectory at the current timestep. Right : The Rewind mechanism (Sec. III-C ) examines the state recorder after each step: if no anomaly is detected, the original task description is retained; if sustained anomalies are identified, the task description is switched to a rewind instruction for steps before reverting to normal execution. Data generation in Sec. III-B .
| Method | LIBERO-PLUS | |||||
| Background | Robot | Language | Layout | Light | Avg | |
| OpenVLA [ 30 ] | 25.3% ↓51.2% | 4.1% ↓72.4% | 26.8% ↓49.7% | 31.6% ↓44.9% | 4.4% ↓72.1% | 18.7% ↓57.8% |
| OpenVLA-OFT [ 16 ] | 83.6% ↓14.0% | 30.6% ↓67.0% | 83.6% {}_{\downarrow{\color[rgb]{1,0,0}\uline{{\color[rgb]{0,0,0}14.0\%}}}} | 73.2% ↓24.4% | 91.6% {}_{\downarrow{\color[rgb]{1,0,0}\uline{{\color[rgb]{0,0,0}6.0\%}}}} | 70.6% ↓27.0% |
| OpenVLA-OFT_w [ 16 ] | 92.5% {}_{\downarrow{\color[rgb]{1,0,0}\textbf{2.8\%}}} | 43.7% ↓51.6% | 73.2% ↓22.1% | 72.3% ↓23.0% | 68.2% ↓27.1% | 68.5% ↓26.8% |
| [ 31 ] | 78.5% ↓15.7% | 6.6% ↓87.6% | 61.0% ↓33.2% | 70.4% ↓23.8% | 79.6% ↓14.6% | 56.6% ↓37.6% |
| -fast [ 39 ] | 67.7% ↓17.8% | 24.8% ↓60.7% | 63.3% ↓22.2% | 70.3% {}_{\downarrow{\color[rgb]{1,0,0}\textbf{15.2\%}}} | 73.0% ↓12.5% | 58.4% ↓27.1% |
TABLE I : Out-of-distribution robustness evaluation on LIBERO-Plus benchmark. We report task success rates across five perturbation types, with subscripts indicating performance degradation relative to the original LIBERO test sets. Bold and underlined values denote the best and second-best success rates, respectively. Red bold and red underlined subscripts highlight the smallest and second-smallest performance drops, indicating superior OOD robustness. Our method achieves the highest average success rate (71.8%) with the smallest average degradation (18.8%), demonstrating strong generalization to unseen perturbations.
| LIBERO | |||||
| Method | Spatial | Object | Goal | Long | Avg |
| Diffusion Policy [ 42 ] | 78.3% | 92.5% | 68.3% | 50.5% | 72.4% |
| Octo [ 43 ] | 78.9% | 85.7% | 84.6% | 51.1% | 75.1% |
| OpenVLA [ 30 ] | 84.7% | 88.4% | 79.2% | 53.7% | 76.5% |
| GRAPE [ 8 ] | 88.5% | 92.1% | 83.1% | 57.2% | 80.2% |
| ThinkAct [ 3 ] | 88.3% | 91.4% | 87.1% | 70.9% | 84.4% |
TABLE II : Performance on LIBERO benchmark. We report results for two training configurations: separately-trained models (Ours) fine-tuned individually on each subset, and a jointly-trained model (Ours ∗ ) trained on all four subsets. Bold and underlined values indicate the best and second-best results respectively.
Fig. 3 : Real-robot task setup and subtask decomposition for Tidy up the Table and Push-T . Each task shows the initial scene configuration alongside the model’s subtask decomposition, demonstrating that SPR produces structured and interpretable subtask plans for both pick-and-place and continuous-contact manipulation tasks.
| Real-Robot Tasks | |||
| Parameter | Pick up | Tidy up | Push-T |
| Demonstrations | 100 | 200 | 200 |
| Steps | 5K | 5K | 5K |
| Global Batch Size | 64 | 64 | 64 |
| Vision Encoder | Frozen | ||
| GPUs (A100s) | 4 | ||
TABLE III : Training configuration for real-robot tasks. We collect 100 demonstrations for Pick up the Object and 200 each for Tidy up the Table and Push-T. All three tasks are trained separately with the vision encoder frozen. The action chunk size is 4 during training, but only the first 2 actions are executed at each inference step to enable more responsive closed-loop control.
| LIBERO Task Suite | |||||
| Parameter | Spat. | Obj. | Goal | Long | All |
| Steps | 50K | 50K | 40K | 80K | 20K+10K |
| Global Batch Size | 128 | 128 | 128 | 128 | 1024 |
| Vision Frozen | No | No | No | No | Yes |
| GPUs (A100s) | 32 | ||||
| Input Images | 1 Third-person + 1 Wrist-Mounted | ||||
TABLE IV : Training configuration for LIBERO task suites. The first four columns show separately-trained models fine-tuned on individual subsets with both vision encoder and language model trainable. The last column shows the jointly-trained model with a two-stage process: first training on original MolmoAct data for 20K steps, then on our SPR-annotated data for 10K steps with data sampled at ratio 5:5:4:8 (Spatial:Object:Goal:Long).
Fig. 4 : Task success rate vs. maximum episode length across LIBERO subsets. Progress-aware models continue improving after the baseline plateaus, demonstrating the ability to leverage extended horizons for complex error recovery.
Fig. 5 : Component ablation on LIBERO-Long and LIBERO-Plus variants. Performance progressively improves from w/o Rewind & Semantics (spatial coordinates only) to w/o Rewind (spatial + semantic) to Ours (full SPR with Rewind), validating the contribution of each component.
Fig. 6 : Effect of rewind step count on LIBERO. =3 achieves optimal performance across all subsets.
Fig. 7 : Scaling analysis on real-robot Tidy up the Table with 1–4 objects. The planning visualization (left) compares SPR and MolmoAct on a 3-object trial, where yellow circles denote subtask waypoints and green lines indicate planned trajectories. The success rate curve (right) shows that SPR degrades gracefully as object count increases, while MolmoAct’s coarse-grained planning collapses beyond 2 objects.
| Real-Robot Tasks | |||
| Method | Pick up | Tidy up | Push-T |
| MolmoAct [ 10 ] | 50% | 0% | 0% |
| Ours | 70% | 30% | 40% |
TABLE V : Performance on real-robot tasks. We report success rates for our method and the MolmoAct baseline across three tasks: Pick up the Object, Tidy up the Table (evaluated in the 3-object setting), and Push-T. Additional task configurations and visualizations are provided in the supplementary material. Bold values indicate the best results.
| Method | Spatial | Object | Goal | Long | Avg |
| MolmoAct † | 89.4% | 92.4% | 88.2% | 72.4% | 85.6% |
| w/o Rewind | 92.6% | 91.8% | 92.2% | 81.8% | 89.6% |
| Ours | 92.4% | 93.0% | 94.2% | 82.8% | 90.6% |
TABLE VI : Ablation study on LIBERO benchmark. Both See-Plan capabilities (w/o Rewind) and the complete SPR framework (Ours) outperform the MolmoAct baseline across all subsets.
Fig. 8 : SPR’s error recovery across diverse failure scenarios (red dashed boxes: error states). (a) Dynamic replanning : Recovers from object relocation and environment state changes by updating spatial subtasks. (b) Robustness to suboptimal initial starts : Mitigates challenging initial configurations by rewinding to regain spatial freedom and replanning the approach. (c) Execution failure recovery : Detects OOD states from failed grasps and resets to familiar configurations for retry.
Fig. 9 : Fine-grained spatial subtask decomposition across diverse manipulation tasks. (a) Complex sequential task with multiple interaction types. (b) Robust decomposition under OOD layout configurations. (c) Spatial relationship understanding with relative positioning. (d) Basic pick-and-place with clear semantic grounding. Each subtask pairs semantic descriptions with 2D spatial coordinates, enabling interpretable progress monitoring.
Fig. 10 : Representative failure cases: (a) discrete action tokens limit precision for careful placement tasks, (b) rewind mechanism fails when robot is physically stuck, and (c) model fails to complete task despite successful rewind due to persistent spatial misalignment and action-planning inconsistency.