ReEnvision: Future-Guided Selective Action Repair for Long-Horizon Manipulation

Xiyin Zeng; Xuewei Huang; Yi Lu; Weihao Yuan; Hao Wang

cs.ROOct 13, 2026

ReEnvision: Future-Guided Selective Action Repair for Long-Horizon Manipulation

Authors: Xiyin Zeng, Xuewei Huang, Yi Lu, Weihao Yuan, Hao Wang

Organizations: Information Hub, The Hong Kong University of Science and Technology (Guangzhou) · Daimon Robotics · School of Intelligence Science and Technology, Nanjing University

Abstract

Long-horizon robotic manipulation requires robust execution over sequences of interdependent actions. Existing methods typically handle execution deviations by replanning the remaining action chunk or correcting actions step by step as new observations arrive. However, a local deviation may affect only a few actions under distribution shift, yet replanning regenerates the entire chunk from an unfamiliar state, potentially introducing errors into otherwise valid actions. Step-wise correction may react too late, as it does not explicitly anticipate how the deviation affects subsequent actions. We propose ReEnvision, a future-guided selective action repair framework that updates a joint future-action belief using execution feedback. It then learns how the resulting belief revision affects each unexecuted action. The resulting belief revision enables selective repair of only the affected actions within a cached action chunk, preserving valid actions and reserving full replanning for severe deviations. On CALVIN, L-CALVIN, and RLBench, ReEnvision achieves 86.6% 5-step success, improves 10-step success from 56% to 63%, and reaches an average success rate of 88.8%, respectively. On five real-world manipulation tasks, it achieves an average success rate of 82.8%. Controlled repair experiments further demonstrate the benefits of localizing affected actions and preserving valid ones during execution.

Figures & tables

Figure 1: Overview of ReEnvision. ReEnvision maintains a joint future-action belief $h_{t}$ and revises it with new observations. The revision identifies affected actions in the cached chunk, enabling selective repair. Unlike methods that rely on full replanning, ReEnvision repairs only affected actions within the current chunk while preserving unaffected ones.

Figure 2: In ReEnvision, joint future-action belief $h_{t}$ conditions action-chunk generation and is revised after each new observation. The resulting belief revision is used to predict per-action correction, validity, and uncertainty, enabling selective repair of the affected cached actions.

Method	1	2	3	4	5	Avg. Len. $\uparrow$
$\pi_{0.5}^{*}$ ( Intelligence et al., 2025 )	94.8	87.4	78.2	71.7	64.3	3.97
VPP ( Hu et al., 2025 )	96.5	90.9	86.6	82.0	76.9	4.33
REMAC ( Wang et al., 2026a )	96.0	91.4	88.3	84.9	78.4	4.39
PALM ( Liu et al., 2026 )	96.9	93.8	89.3	85.9	82.0	4.48
DeFI ( Zhang et al., 2026 )	97.9	94.2	90.7	87.0	81.2	4.51
VITA ( Ma et al., 2026 )	99.1	94.9	91.2	87.8	84.5	4.57

Table 1: CALVIN ( Mees et al., 2022 ) ABC

\rightarrow

D results. We report the average success rate over 1,000 rollouts and the average number of consecutively completed tasks (Avg. Len.). ReEnvision demonstrates clear improvements over the baselines, particularly on longer-horizon sequences.

Method	Tasks Completed in Sequence										Avg. Len.
	1	2	3	4	5	6	7	8	9	10
VLAS ( Zhao et al., 2025 )	0.88	0.80	0.72	0.58	0.49	0.46	0.34	0.28	0.19	0.16	4.90
RoboVLMs ( Li et al., 2026b )	0.90	0.80	0.78	0.70	0.64	0.60	0.51	0.41	0.36	0.34	6.04
REMAC ( Wang et al., 2026a )	0.94	0.90	0.82	0.74	0.71	0.58	0.54	0.50	0.47	0.41	6.61
DeFI ( Zhang et al., 2026 )	0.98	0.96	0.92	0.86	0.75	0.71	0.59	0.57	0.52	0.44	7.30
Long-VLA ( Fan et al., 2025 )	1.00	1.00	0.98	0.91	0.85	0.82	0.79	0.70	0.63	0.56	8.24

Table 2: Comparison with prior methods on the L-CALVIN ( Fan et al., 2025 ) , using 100 evaluation sequences of 10 consecutive instructions under the ABC

\rightarrow

D setting.

Method	Avg. SR	Close Jar	Insert Peg	Light Bulb	Meat off Grill	Open Drawer	Place Cups	Shape Sorter	Place Wine
HDFlow ( Gireesh et al., 2026 )	87.5	96.0	93.3	88.7	98.7	82.0	56.7	73.3	94.7
TVVE ( Bai et al., 2026 )	86.6	100.0	98.0	86.0	94.0	90.0	54.0	62.0	92.0
SpatialActor ( Shi et al., 2026 )	87.4	94.0	93.3	88.7	98.7	82.0	56.7	73.3	94.7
BridgeVLA ( Li et al., 2025a )	85.7	98.0	80.0	86.0	100.0	98.0	48.0	38.0	90.0
ReEnvision	88.8	98.0	82.0	90.0	100.0	90.0	60.0	50.0	96.0

Table 3: Performance comparison on 18 RLBench ( James et al., 2019 ) manipulation tasks, with 50 evaluation episodes per task. We report task success rates (%) and the average success rate (Avg. SR) across all 900 episodes. Best results are shown in bold.

Method	Success Rate (%)
Method	Pen	Color	Drawer	Towel	Flower	Average
Diffusion Policy ( Chi et al., 2024 )	78	56	78	80	52	68.8
$\pi_{0.5}^{*}$ ( Intelligence et al., 2025 )	90	70	80	94	70	80.8
BridgeVLA ( Li et al., 2025a )	88	70	86	74	68	77.2
DeFI ( Zhang et al., 2026 )	86	68	78	82	66	76.0
ReEnvision	90	76	84	88	76	82.8

Table 4: Real-world manipulation performance on five tasks using an ARX X5 robot. Each method is evaluated over 50 trials per task.

Figure 3: Visualization on a flower placement task. ReEnvision detects affected actions following an unexpected flower drop and selectively repairs the remaining action chunk through belief revision. For visualization, the teacher correction magnitudes $\|\Delta a_{k}^{*}\|_{2}$ are rescaled for readability, with actions exceeding 0.10 on the rescaled scale marked as affected.

Methods	RLBench	CALVIN		L-CALVIN
Methods	Avg. SR $\uparrow$	5-Step $\uparrow$	Avg. Len. $\uparrow$	Avg. Len. $\uparrow$	5-Step $\uparrow$	10-Step $\uparrow$
Baseline	85.7	81.2	4.51	7.30	0.75	0.44
$z_{f}$ only	84.6	84.7	4.58	8.22	0.85	0.55
$z_{a}$ only	88.4	85.2	4.57	8.26	0.80	0.56
w/o belief revision	87.4	82.6	4.54	7.96	0.83	0.50
w/o uncertainty	88.0	85.2	4.60	8.32	0.90	0.61

Table 5: Ablation study of ReEnvision on RLBench, CALVIN, and L-CALVIN.

Methods	RLBench	CALVIN		L-CALVIN
	Avg. SR $\uparrow$	5-Step $\uparrow$	Avg. Len. $\uparrow$	Avg. Len. $\uparrow$	5-Step $\uparrow$	10-Step $\uparrow$
No Repair	85.7	81.2	4.51	7.30	0.75	0.44
Random Mask	83.1	81.6	4.48	7.59	0.80	0.49
Complement Mask	77.4	78.8	4.39	6.85	0.69	0.38
Selective w/ $m^{*}$	88.8	85.9	4.65	8.45	0.91	0.66
Whole Suffix	81.2	80.7	4.50	7.15	0.80	0.43

Table 6: Repair-location control study on RLBench, CALVIN, and L-CALVIN.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Value
Affected fraction ( $m^{*}$ )	0.208
Pred. affected fraction	0.206
Segments per suffix ( $m^{*}$ )	0.821
Pred. segments per suffix	0.850
Mask F1	0.861
Temporal IoU (all)	0.785

Appendix

Table 7: Validation of action-change locality and TCN localization accuracy.

m^{*}

denotes the reference affected-action mask derived from paired policy responses. Non-empty IoU is computed only on suffixes containing at least one affected action.

Severity	Strategy	Recovery Success $\uparrow$	Avg. Len. $\uparrow$	Modified Actions	Replan Rate
Clean	ReEnvision	0.75	8.36	0.08	0.03
Small	No Repair	0.79	7.18	0.00	0.00
Small	Step-wise	0.75	7.56	0.03	0.00
Small	Full Replan	0.68	7.07	1.00	1.00
Small	Oracle Selective	0.93	8.25	0.08	0.05
Small	ReEnvision	0.84	7.93	0.18	0.11

Appendix

Table 8: Controlled deviation recovery on L-CALVIN under perturbations of increasing severity.

Figure 4: ReEnvision’s execution strategy under controlled deviations of increasing severity. Each bar shows the fraction of perturbation cases handled by keeping the cached suffix, selectively repairing it, or triggering full replanning. As deviation severity increases, ReEnvision progressively shifts from suffix preservation toward selective intervention and full replanning.

Component	Latency
Backbone policy	$\sim$ 110 ms
Future–action belief	$\sim$ 8 ms
Belief revision model	$\sim$ 2 ms
Temporal impact model	$\sim$ 1 ms
Mask optimization	$<$ 1 ms
ReEnvision (end-to-end)	$\sim$ 145 ms

Appendix

Table 9: Runtime analysis of ReEnvision. Component latencies are measured individually, while the total latency reports end-to-end inference time per execution step. ReEnvision introduces approximately 32% additional latency over the backbone policy, while the belief revision and selective repair components add only a small computational overhead.

Parameter	Value	Avg. Len. $\uparrow$	5-Step SR (%) $\uparrow$	Replan Rate (%)
$\tau$	$5\times 10^{-5}$	4.50	85.0	8.0
	$\mathbf{1\times 10^{-4}}$	4.62	86.6	2.1
	$2\times 10^{-4}$	4.55	82.0	3.0
$\rho$	0.4	4.59	87.0	12.0
	$\mathbf{0.6}$	4.62	86.6	2.1
	0.8	4.57	86.0	4.0

Appendix

Table 10: Hyperparameter sensitivity on CALVIN. Each parameter is varied independently while the others are fixed at their default values:

\tau=10^{-4}

\rho=0.6

, and

\eta=0.9

. The default configuration is shown in bold.

Explore similar work

See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation
Mar 10, 2026Tingjun Dai, Mingfei Han, Tingwen Du +6Vision-Language-Action Models Language-Conditioned Robot Manipulation
School of Information Science and Technology, University of Science and Technology of China, Anhui, China. · University of Technology Sydney, Ultimo, NSW, Australia. · Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates. +3
ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback
Sep 29, 2026Ahmed Nader Ahmed, Omar Moured, Mughni Irfan Mohammed Abdul +2Robot Policy Adaptation Long-Horizon Robotic Manipulation
Khalifa University, Abu Dhabi, United Arab Emirates. · Sereact GmbH, Stuttgart, Germany.
Commit While Futures Agree: Consequence-Aware Adaptive Action Chunking for Robot Manipulation
Oct 6, 2026Yuyan Li, Yujia Wang, Yusong Huang +8Robotic Manipulation
The Hong Kong University of Science and Technology (Guangzhou) · ANU Intelligent · Victoria University of Wellington +2

Mar 10, 2026J/K move · Enter open · S save

See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation

Tingjun Dai, Mingfei Han, Tingwen Du, Zhiheng Liu+5

School of Information Science and Technology, University of Science and Technology of China, Anhui, China. · University of Technology Sydney, Ultimo, NSW, Australia. · Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates.+3

Vision-Language-Action Models Language-Conditioned Robot Manipulation Robustness of VLA Models Robot Failure Recovery Instruction Generalization in VLA Models

Open paper AskPDF arXiv

Measurement of task progress through explicit, actionable milestones is critical for robust robotic manipulation. This progress awareness enables a model to ground its current task status, anticipate verifiable intermediate states, and detect and recover from failures when progress stalls. To embody this capability, we introduce See, Plan, Rewind (SPR), a progress-aware vision-language-action framework that dynamically grounds language instructions into a sequence of spatial subgoals. SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure by monitoring progress against the expected sequence. This closed-loop approach enables robust error correction without requiring additional training data or auxiliary models. Extensive experiments demonstrate the framework's effectiveness, generalization and robustness: SPR outperforms the MolmoAct baseline by 5% on the LIBERO benchmark. On the challenging LIBERO-Plus benchmark with unseen instructions and initial states, SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA, demonstrating superior out-of-distribution robustness.

Abstract

Explore similar work

See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation

ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback

Commit While Futures Agree: Consequence-Aware Adaptive Action Chunking for Robot Manipulation