See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation
Authors: Tingjun Dai, Mingfei Han, Tingwen Du, Zhiheng Liu, Zihao Zhang, Zhihui Li, Salman Khan, Jun Yu, +1 more
Organizations: School of Information Science and Technology, University of Science and Technology of China, Anhui, China. · University of Technology Sydney, Ultimo, NSW, Australia. · Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates. · The University of Hong Kong, Hong Kong, China. · Institute of AI for Industry, Chinese Academy of Sciences, Jiangsu, China. · School of Intelligent Science and Engineering, Harbin Institute of Technology (Shenzhen), Guangdong, China.
Measurement of task progress through explicit, actionable milestones is critical for robust robotic manipulation. This progress awareness enables a model to ground its current task status, anticipate verifiable intermediate states, and detect and recover from failures when progress stalls. To embody this capability, we introduce See, Plan, Rewind (SPR), a progress-aware vision-language-action framework that dynamically grounds language instructions into a sequence of spatial subgoals. SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure by monitoring progress against the expected sequence. This closed-loop approach enables robust error correction without requiring additional training data or auxiliary models. Extensive experiments demonstrate the framework's effectiveness, generalization and robustness: SPR outperforms the MolmoAct baseline by 5% on the LIBERO benchmark. On the challenging LIBERO-Plus benchmark with unseen instructions and initial states, SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA, demonstrating superior out-of-distribution robustness.
Figures & tables
Fig. 1 : See-Plan-Rewind framework’s closed-loop execution workflow. Starting from the initial state ( top-left ), the model performs See-Plan reasoning to generate actions (arrow to top-right ), which visualizes the subtask decomposition and trajectory planning. Under normal execution, the loop returns to top-left. When progress monitoring detects failures ( bottom-right ), the Rewind mechanism activates ( bottom-left ), returning the robot to the initial state before resuming the loop. This closed-loop design enables autonomous error recovery through explicit progress awareness.
Fig. 2 : SPR framework overview. Left : Upon receiving the task description and observation, the model performs See-Plan reasoning(Sec. III-A ), which identifies remaining subtasks with 2D spatial coordinates (See) and plans a gripper trajectory to the next subtask waypoint (Plan), then outputs action tokens for execution. Each inference step also updates the state recorder, where SN denotes the predicted subtask count and TN the planned 2D trajectory at the current timestep. Right : The Rewind mechanism (Sec. III-C ) examines the state recorder after each step: if no anomaly is detected, the original task description is retained; if sustained anomalies are identified, the task description is switched to a rewind instruction for N steps before reverting to normal execution. Data generation in Sec. III-B .
TABLE I : Out-of-distribution robustness evaluation on LIBERO-Plus benchmark. We report task success rates across five perturbation types, with subscripts indicating performance degradation relative to the original LIBERO test sets. Bold and underlined values denote the best and second-best success rates, respectively. Red bold and red underlined subscripts highlight the smallest and second-smallest performance drops, indicating superior OOD robustness. Our method achieves the highest average success rate (71.8%) with the smallest average degradation (18.8%), demonstrating strong generalization to unseen perturbations.
LIBERO
Method
Spatial
Object
Goal
Long
Avg
Diffusion Policy [ 42 ]
78.3%
92.5%
68.3%
50.5%
72.4%
Octo [ 43 ]
78.9%
85.7%
84.6%
51.1%
75.1%
OpenVLA [ 30 ]
84.7%
88.4%
79.2%
53.7%
76.5%
GRAPE [ 8 ]
88.5%
92.1%
83.1%
57.2%
80.2%
ThinkAct [ 3 ]
88.3%
91.4%
87.1%
70.9%
84.4%
TABLE II : Performance on LIBERO benchmark. We report results for two training configurations: separately-trained models (Ours) fine-tuned individually on each subset, and a jointly-trained model (Ours ∗ ) trained on all four subsets. Bold and underlined values indicate the best and second-best results respectively.
Fig. 3 : Real-robot task setup and subtask decomposition for Tidy up the Table and Push-T . Each task shows the initial scene configuration alongside the model’s subtask decomposition, demonstrating that SPR produces structured and interpretable subtask plans for both pick-and-place and continuous-contact manipulation tasks.
Real-Robot Tasks
Parameter
Pick up
Tidy up
Push-T
Demonstrations
100
200
200
Steps
5K
5K
5K
Global Batch Size
64
64
64
Vision Encoder
Frozen
GPUs (A100s)
4
TABLE III : Training configuration for real-robot tasks. We collect 100 demonstrations for Pick up the Object and 200 each for Tidy up the Table and Push-T. All three tasks are trained separately with the vision encoder frozen. The action chunk size is 4 during training, but only the first 2 actions are executed at each inference step to enable more responsive closed-loop control.
LIBERO Task Suite
Parameter
Spat.
Obj.
Goal
Long
All
Steps
50K
50K
40K
80K
20K+10K
Global Batch Size
128
128
128
128
1024
Vision Frozen
No
No
No
No
Yes
GPUs (A100s)
32
Input Images
1 Third-person + 1 Wrist-Mounted
TABLE IV : Training configuration for LIBERO task suites. The first four columns show separately-trained models fine-tuned on individual subsets with both vision encoder and language model trainable. The last column shows the jointly-trained model with a two-stage process: first training on original MolmoAct data for 20K steps, then on our SPR-annotated data for 10K steps with data sampled at ratio 5:5:4:8 (Spatial:Object:Goal:Long).
Fig. 4 : Task success rate vs. maximum episode length across LIBERO subsets. Progress-aware models continue improving after the baseline plateaus, demonstrating the ability to leverage extended horizons for complex error recovery.
Fig. 5 : Component ablation on LIBERO-Long and LIBERO-Plus variants. Performance progressively improves from w/o Rewind & Semantics (spatial coordinates only) to w/o Rewind (spatial + semantic) to Ours (full SPR with Rewind), validating the contribution of each component.
Fig. 6 : Effect of rewind step count N on LIBERO. N =3 achieves optimal performance across all subsets.
Fig. 7 : Scaling analysis on real-robot Tidy up the Table with 1–4 objects. The planning visualization (left) compares SPR and MolmoAct on a 3-object trial, where yellow circles denote subtask waypoints and green lines indicate planned trajectories. The success rate curve (right) shows that SPR degrades gracefully as object count increases, while MolmoAct’s coarse-grained planning collapses beyond 2 objects.
Real-Robot Tasks
Method
Pick up
Tidy up
Push-T
MolmoAct [ 10 ]
50%
0%
0%
Ours
70%
30%
40%
TABLE V : Performance on real-robot tasks. We report success rates for our method and the MolmoAct baseline across three tasks: Pick up the Object, Tidy up the Table (evaluated in the 3-object setting), and Push-T. Additional task configurations and visualizations are provided in the supplementary material. Bold values indicate the best results.
Method
Spatial
Object
Goal
Long
Avg
MolmoAct †
89.4%
92.4%
88.2%
72.4%
85.6%
w/o Rewind
92.6%
91.8%
92.2%
81.8%
89.6%
Ours
92.4%
93.0%
94.2%
82.8%
90.6%
TABLE VI : Ablation study on LIBERO benchmark. Both See-Plan capabilities (w/o Rewind) and the complete SPR framework (Ours) outperform the MolmoAct baseline across all subsets.
Fig. 8 : SPR’s error recovery across diverse failure scenarios (red dashed boxes: error states). (a) Dynamic replanning : Recovers from object relocation and environment state changes by updating spatial subtasks. (b) Robustness to suboptimal initial starts : Mitigates challenging initial configurations by rewinding to regain spatial freedom and replanning the approach. (c) Execution failure recovery : Detects OOD states from failed grasps and resets to familiar configurations for retry.
Fig. 9 : Fine-grained spatial subtask decomposition across diverse manipulation tasks. (a) Complex sequential task with multiple interaction types. (b) Robust decomposition under OOD layout configurations. (c) Spatial relationship understanding with relative positioning. (d) Basic pick-and-place with clear semantic grounding. Each subtask pairs semantic descriptions with 2D spatial coordinates, enabling interpretable progress monitoring.
Fig. 10 : Representative failure cases: (a) discrete action tokens limit precision for careful placement tasks, (b) rewind mechanism fails when robot is physically stuck, and (c) model fails to complete task despite successful rewind due to persistent spatial misalignment and action-planning inconsistency.
Most existing vision-language-action (VLA) models for robotic manipulation lack progress awareness, typically relying on hand-crafted heuristics for task termination. This limitation is particularly severe in long-horizon tasks involving cascaded sub-goals. In this work, we investigate the estimation and integration of task progress, proposing a novel model named {\textbf \vla}. Our technical contributions are twofold: (1) robust progress estimation: We pre-train a progress estimator on large-scale, unsupervised video-text robotic datasets. This estimator achieves a low prediction residual (0.07 on a scale of [0,1]) in simulation and demonstrates zero-shot generalization to unseen real-world samples, and (2) differentiable progress guidance: We introduce an inverse dynamics world model that maps predicted action tokens into future latent visual states. These latents are then processed by the progress estimator; by applying a maximal progress regularization, we establish a differentiable pipeline that provides progress-piloted guidance to refine action tokens. Extensive experiments on the CALVIN and LIBERO benchmarks, alongside real-world robot deployment, consistently demonstrate substantial improvements in success rates and generalization over strong baselines.
Hongyu Yan, Qiwei Li, Jiaolong Yang +1
Peking University · Microsoft Research Asia (MSRA)
Robotic manipulation poses fundamental challenges due to uncertainty, long-horizon execution, and compounding errors, which can easily destabilize execution and lead to task failure. Although recent vision-language-action (VLA) models exhibit strong generalization, they typically lack explicit mechanisms to assess execution stability and to recover when execution deviates from its nominal behavior. In this paper, we propose: (1) two complementary metrics to assess execution quality at runtime, and (2) an agentic reinforcement learning framework that learns to restore effective execution through high-level decision-making rather than directly learning low-level actions. In this framework, an agentic policy reasons over recent execution history and selects among a small set of execution modes to regulate the execution process. Under execution degradation, it triggers appropriate recovery mechanisms to restore the robot to previously visited nominal states, enabling the task to continue. We evaluate the proposed method on the LIBERO benchmark, achieving up to a 13.7% improvement in success rate under standard settings and up to a 39.2% improvement under disturbance settings, demonstrating substantially enhanced execution robustness.
Xiaopeng Zhang, Yueyang Weng, Qi Liu +2
School of Inteligence Science and Engineering, the Harbin Institute of Technology Shenzhen, 518055, China. · Faculty of Robot Science and Engineering, Northeastern University, Shenyang, 110819, China.
We present ProgVLA, a compact vision-language-action (VLA) model designed for reliable robot manipulation under tight compute and memory budgets. The model specifically focuses on efficiently processing long multi-modal sequences by maintaining an explicit representation of task progress over extended horizons. To this end, ProgVLA integrates two key components. First, a multi-modal encoder with a two-stage Perceiver resampling scheme compresses variable-length visual, language, and proprioceptive streams into a fixed set of control-ready context tokens, substantially reducing sequence length while preserving cross-modal grounding. Second, an auxiliary set of progress heads is trained with offline reinforcement learning (RL) objectives to jointly learn critics over normalized remaining-horizon targets. This provides the policy with an internal estimate of task progress and enables advantage- and success-weighted flow-matching imitation learning. On two well-established multi-task robot manipulation benchmarks, a 0.1B-parameter ProgVLA model reaches success rates that are competitive with, and on long-horizon and harder task tiers exceed, substantially larger pretrained baselines. Ablations indicate that the learned context resampler and task-adaptive visual fine-tuning are the largest single contributors, while progress-aware training provides a consistent additional gain that is concentrated on long-horizon and multi-object tasks. We further validate the approach in real-world toy-kitchen environments.