Back to the Future: Predicting Long-Horizon Dynamics with Retrospective Correction for Disturbance-Aware Retrieval

Jiwoong Choi; Haoyu Li; Yiwei Lyu

cs.RONov 1, 2026

Back to the Future: Predicting Long-Horizon Dynamics with Retrospective Correction for Disturbance-Aware Retrieval

Authors: Jiwoong Choi, Haoyu Li, Yiwei Lyu

Organizations: Dept. of Aerospace Eng. and Eng. Mechanics, University of Texas at Austin · Dept. of Computer Sci. and Eng., Texas A&M University

Abstract

Retrieving objects from densely packed storage requires anticipating how contact disturbs surrounding objects when limited clearance prevents lifting the target over clutter. However, contact can propagate beyond the target's immediate neighbors, while accumulated prediction errors make learned dynamics unreliable over long planning horizons. We introduce DARE (Disturbance-Aware REtrieval), a geometry-aware relational dynamics framework that uses retrospective correction to improve long-horizon prediction for disturbance-aware retrieval. DARE encodes compact object descriptors and predicts action-conditioned disturbance propagation through object-level message passing. When a sparse observation becomes available, a learned corrector uses the prediction discrepancy to amend an earlier predicted state and reruns the intervening dynamics, propagating the correction through the predicted contact chain. These predictions guide a receding-horizon planner that balances retrieval progress against displacement and rotation of surrounding objects, permitting necessary contact while discouraging harmful disturbance. Experiments in planar and RoboCasa environments evaluate long-horizon prediction, generalization, and robustness for disturbance-aware retrieval in dense clutter. See our project website: https://the-back-to-the-future.github.io

Figures & tables

Fig. 1 : Disturbance-aware retrieval. (a) Collision avoidance leaves the target blocked. (b) Direct retrieval topples a neighboring object. (c) Our method predicts contact-induced disturbance to plan retrieval through controlled contact, preserving stability of surrounding objects.

Fig. 2 : Pipeline Overview: Given object geometry and the current scene state, DARE propagates action-conditioned interactions over an object graph to predict scene evolution under candidate target motions. The resulting rollouts capture direct and indirect disturbances to surrounding objects, enabling the planner to select a retrieval trajectory that balances goal progress against collateral disturbance.

Fig. 3 : Retrospective correction for long-horizon prediction. The corrector combines prior predictions with the observation mismatch at $t_{15}$ (yellow) to estimate a corrected state $\tilde{t}_{11}$ , the first step after the last observation $t_{10}$ . Rerolling from it propagates the correction through the learned dynamics and initializes the next cycle, mitigating error drift .

Fig. 4 : Tabletop dynamics prediction under noise. Outlines in purple, light blue and dark blue, indicate predicted object poses in a held-out scene as the target (yellow) pushes through clutter. Uncorrected predictions drift (top), direct observation updates inherit noise (middle), and noise-trained correction maintains alignment with the objects (bottom).

Fig. 5 : Object Retrieval Visualization Top-down views compare three planners, with target motion marked in orange. Each row shows four evenly spaced execution steps with translucent next-step poses and the executed target path in the final frame.

Fig. 6 : Dynamics prediction. Terminal translation (left) and rotation (right) errors normalized by the no-motion error at each horizon. Rotation is omitted for symmetric pucks.

Method	Obs.	Corr.	Success rate $\uparrow$			Translational disturbance (cm) $\downarrow$			Tilt disturbance ( ∘ ) $\downarrow$			Yaw disturbance ( ∘ ) $\downarrow$
Method	Obs.	Corr.	5 obj.	10 obj.	20 obj.	5 obj.	10 obj.	20 obj.	5 obj.	10 obj.	20 obj.	5 obj.	10 obj.	20 obj.
Direct retrieval	Clean	–	0.60/0.60	0.30/0.20	0.20/0.10	26.9/21.5	46.0/48.4	91.2/69.6	66.5/36.5	120.3/88.9	191.6/105.2	185.8/182.9	292.2/290.3	555.0/573.3
Collision avoidance	Clean	–	0.40/0.50	0.20/0.10	0.10/0.10	4.5/4.2	6.2/4.7	11.9/13.4	3.3/24.3	20.8/17.7	54.2/70.9	43.0/32.0	61.0/64.9	93.2/108.5
Ours w/o dist. obj.	Clean	Off	0.90 / 1.00	0.90 / 1.00	0.90 /0.80	11.1/9.6	35.1/28.4	81.7/84.0	17.3/0.6	84.0/23.7	98.7/115.0	127.7/142.7	286.8/198.1	486.9/538.1
Ours	Clean	Off	0.90 / 1.00	0.90 / 1.00	0.90 / 1.00	9.9/8.5	30.0/33.3	75.1/71.2	15.0/0.5	90.1/21.2	120.2/50.7	113.6/126.1	300.5/223.2	458.0/500.4
Ours w/o dist. obj.	Noisy	Off	0.90/ 1.00	0.40/0.00	0.10/0.00	11.1/11.6	28.0/14.3	87.7/77.5	17.3/18.6	91.8/34.1	181.6/204.2	119.3/148.9	240.6/78.7	458.2/513.3

TABLE I : Retrieval from cluttered compartments. Each numerical entry reports Easy / Hard setup.

Fig. 7 : Effect of joint training. Terminal translation (top) and rotation (bottom) errors with shared pretrained initialization and matched additional optimization steps.

Fig. 8 : Retrospective correction under noise. Translation (top) and rotation (bottom) RMSE over 40-step rollouts versus observation noise $\sigma$ , fixed within each rollout. Shading indicates noise levels beyond the noise training range.

Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.

See all 449 papers

Plan-Conditioned Imitation for Robust Object Retrieval under Self-Occlusion in Dense Clutter
Sep 30, 2026Kowndinya Boyalakuntla, Ajinkya Pawar, Abdeslam Boularias +1Imitation Learning Robotic Manipulation
Rutgers University · Indian Institute of Technology Bombay, India
Back to the Future: Reverse-Consequence Modeling for Long-Horizon Particle Dynamics
Oct 13, 2026Leyan Zhang, Nantian He, Jiajun Ma +2Trajectory Prediction Diffusion Models for Motion Generation
Shanghai Jiaotong University
RetrDex: Efficient Object Retrieval in Cluttered Scenes with a Dexterous Hand
Feb 25, 2025Fengshuo Bai, Yu Li, Jie Chu +5RL for Robotics Robotic Manipulation
Shanghai Jiao Tong University · PKU-PsiBot Joint Lab · Peking University +2

Sep 30, 2026J/K move · Enter open · S save

Plan-Conditioned Imitation for Robust Object Retrieval under Self-Occlusion in Dense Clutter

Kowndinya Boyalakuntla, Ajinkya Pawar, Abdeslam Boularias, Jingjin Yu

Rutgers University · Indian Institute of Technology Bombay, India

Imitation Learning Robotic Manipulation Interactive Imitation Learning Robot Imitation Learning Robotic Object Search

Open paper AskPDF arXiv

Retrieving objects from dense clutter requires rearrangement during which the manipulator can occlude objects while moving them. Repeated arm withdrawals to restore visibility interrupt execution. We introduce TRACE, a plan-conditioned imitation framework for retrieval under self-occlusion. A single unoccluded observation initializes a digital twin, where a privileged teacher generates a fixed nominal rollout. A recurrent student combines local rollout context, partial object observations, and proprioception to select actions that can correct deviations from the prediction. Behavior cloning initializes the student; DAgger refines it with teacher labels on student-visited states. The rollout remains fixed throughout execution, so the deployed student needs neither online teacher queries nor additional simulator rollouts during pushing. On 511 simulation test scenes, TRACE achieves 90.7% success versus 43.4% for nominal replay and 96.7% for the privileged closed-loop teacher. At a matched 26,373-label budget, student-state supervision achieves 87.8% versus 66.7% for expert-only cloning, demonstrating gains beyond additional labels. On a UR5e, TRACE achieves 90.0% success versus 95.0% for the closed-loop teacher, while reducing total execution time from 192.7 s to 67.3 s. It avoids the teacher's 16.8 sensing-related arm retractions per trial during pushing, retaining a final withdrawal for graspability evaluation. Code and data will be released at: https://trace-retrieval.github.io.

Abstract

Long-Horizon Robotic Manipulation

Explore similar work

Plan-Conditioned Imitation for Robust Object Retrieval under Self-Occlusion in Dense Clutter

Back to the Future: Reverse-Consequence Modeling for Long-Horizon Particle Dynamics

RetrDex: Efficient Object Retrieval in Cluttered Scenes with a Dexterous Hand