Non-Markovian Long-Horizon Robot Manipulation via Keyframe Chaining
Authors: Yipeng Chen, Wentao Tan, Lei Zhu, Fengling Li, Jingjing Li, Guoli Yang, Heng Tao Shen
Organizations: Tongji University · University of Technology Sydney · University of Electronic Science and Technology of China · Advanced Institute of Big Data
Existing Vision-Language-Action (VLA) models often struggle to generalize to long-horizon tasks due to their heavy reliance on immediate observations. While recent studies incorporate retrieval mechanisms or extend context windows to handle procedural tasks, they often struggle to capture Non-Markovian dependencies, where optimal actions rely solely on specific past states rather than the current observation. To address this, we introduce Keyframe-Chaining VLA, a framework that extracts and links key historical frames to model long-horizon dependencies. Specifically, we propose an automatic keyframe selector that learns a discriminative embedding space, effectively identifying distinct state transitions. To capture task-critical information, we design a progress-aware query mechanism that dynamically retrieves historical frames based on their temporal relevance to the current execution phase. These selected keyframes are integrated into the VLA as interleaved visual tokens, explicitly grounding the policy in the long-horizon temporal context. Finally, we introduce a suite of four Non-Markovian manipulation tasks built upon the ManiSkill simulator to measure task success rates. Experimental results demonstrate that our method achieves superior performance, effectively tackling robot manipulation tasks characterized by long-horizon temporal dependencies. Code is available at https://github.com/cytoplastm/KC-VLA.
Figures & tables
Figure 1: (a) Unlike standard VLAs ( Kim et al., 2024 ; Team et al., 2024 ; Chi et al., 2025 ) that rely on dense sliding windows or context compression ( Jang et al., 2025 ; Koo et al., 2025 ) , our Keyframe-Chaining VLA explicitly selects sparse semantic keyframes, avoiding computational redundancy. (b) The other three paradigms in (a) suffer from limited temporal receptive fields, failing to capture distant critical cues. In contrast, our KSM-based retrieval achieves a global temporal receptive field, enabling the agent to access long-range historical information and effectively resolve state aliasing. (c) Our framework achieves state-of-the-art performance, significantly outperforming baselines in both ManiSkill memory-dependent tasks and real-world long-horizon deployment.
Figure 2: Overview of the Keyframe-Chaining VLA framework. The architecture operates via two synergistic modules. (Left) Keyframe Selection Module: Adopts a two-stage design where Stage I optimizes a visual encoder via unified metric learning, and Stage II employs a Task-Modulated Query network to compute phase-specific confidence scores st . (Top Right) Greedy Temporal Smoothing: A stabilizing mechanism designed to filter detection jitter. It greedily updates provisional candidates to capture the latest valid state and finalizes the keyframe only after passing a temporal validation window. (Bottom Right) VLA Module: The policy integrates the sparse semantic history of detected keyframes with the current observation to generate actions via Flow Matching.
Figure 3: Overview of the four custom Non-Markovian ManiSkill tasks. Red bounding boxes indicate the ground-truth keyframes utilized for training. Please refer to the Appendix for the detailed definitions of keyframe events for each task.
Model / Configuration
Sampling
Nh
I
Spatial
Temporal
Identity
Counting
Average
π0 ( Black et al., 2024 )
Dense
0
-
2.0
0.0
0.0
60.0
15.5
Diffusion Policy ( Chi et al., 2025 )
1
1
22.0
10.0
0.0
30.0
15.5
GR00T-N1.5 ( Bjorck et al., 2025 ) ( No History )
0
-
20.0
0.0
28.0
16.0
16.0
GR00T-N1.5 ( Short-term )
Dense
1
1
8.0
16.0
30.0
4.0
14.5
2
1
18.0
52.0
12.0
0.0
20.0
3
1
14.0
54.0
0.0
40.0
27.0
Table 1: Quantitative comparison of success rates (%) on non-Markovian tasks. Keyframe-Chaining VLA(ours) achieves robust and superior performance.
KSM Training Paradigm
P (%) ↑
R (%) ↑
F1 (%) ↑
FPR (%) ↓
FNR (%) ↓
w/o Metric Pre-training
90.7
81.7
85.1
9.3
18.3
Joint End-to-End
93.6
79.6
83.4
6.4
20.4
Ours (Two-stage)
97.5
97.5
97.5
2.5
2.5
Table 2: Comparative diagnostic evaluation of KSM paradigms. Metrics are averaged across four tasks and reported in percentage. All metrics are reported in percentage (%). Detailed task-wise results are provided in Appendix B .
Configuration
Spatial
Temporal
Identity
Counting
Average
w/o Prompt Refinement
56.0
96.0
100.0
100.0
88.0
Ours (Context-Aware)
70.0
98.0
100.0
100.0
92.0
Table 3: Ablation on Prompt Refinement. Comparing context-aware prompting against generic instructions.
Figure 4: Real-world Experimental Setup.
Model
Spatial
Temporal
Counting
Identity
Average
(4 stages)
(3 stages)
(2 stages)
(2 stages)
DP
0.0 / 33.8
0.0 / 16.7
5.0 / 35.0
10.0 / 45.0
3.8 / 32.6
GR00T
0.0 / 47.5
0.0 / 30.0
10.0 / 45.0
15.0 / 50.0
6.3 / 43.1
Ours
10.0 / 58.8
50.0 / 75.0
80.0 / 90.0
55.0 / 77.5
48.75 / 75.3
Table 4: Real-world quantitative evaluation (20 trials per task). Results are reported as Success Rate / Completion Rate (%). Completion Rate measures the percentage of completed sub-stages (see Appendix for stage definitions). Our method demonstrates decisive advantages in multi-stage reasoning tasks.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Method
Precision
Recall
F1-Score
Temporal Sequencing
w/o Metric Pre-training
80.3
81.7
81.0
Joint End-to-End
93.3
93.3
93.3
Ours (Two-stage)
100.0
100.0
100.0
Counting & Latency
w/o Metric Pre-training
100.0
62.5
76.9
Joint End-to-End
100.0
40.0
57.1
Ours (Two-stage)
100.0
100.0
100.0
Appendix
Table 5: Detailed Task-wise KSM Diagnostic Metrics (%). We compare Precision (P), Recall (R), and F1-score across three training paradigms. Our method consistently achieves optimal or near-optimal performance across all categories.
Figure 5: The structured system prompt used to condition the VLA policy. The {Language Instruction} placeholder is replaced by the specific natural language command for the current episode (e.g., “Stack the cubes based on their initial vertical order”).
Figure 6: Qualitative rollouts of Keyframe-Chaining VLA in real-world scenarios. Each row corresponds to one of the four Non-Markovian tasks. The snapshots demonstrate the agent’s ability to maintain long-horizon consistency, such as waiting for signal lights (row 3) or tracking swapped objects (row 4).
Vision-Language-Action (VLA) models have demonstrated strong capabilities in robotic manipulation, but their performance degrades significantly in long-horizon tasks due to cumulative error propagation. This limitation largely arises from static feature fusion mechanisms that rely on fixed weights to combine visual, language, and action representations, preventing the model from adapting to different phases of task execution. To address this limitation, we propose S2-VLA, a framework that introduces a State-Space Guided Adaptive Attention (SSGAA) mechanism. SSGAA maintains a belief state that tracks task progression and generates dynamic gating weights to adaptively fuse information from three complementary sources visual features for spatial perception, task intents for high-level task planning, and temporal action sequences for execution consistency. This adaptive fusion allows the model to shift its focus throughout task execution, aligning with the evolving requirements of different task stages. Despite its compact 2B parameter size, S2-VLA consistently outperforms larger 7B-scale models and achieves state-of-the-art performance on long-horizon manipulation benchmarks, including LIBERO and SimplerEnv. highlighting the importance of adaptive feature fusion for long-horizon robotic manipulation.
Zhipeng Xie, Zongyi Han, Xiangyi Wei +3
School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 2State Key Laboratory of Submarine Geoscience, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China
General-purpose robots must master long-horizon manipulation, defined as tasks involving multiple kinematic structure changes (e.g., attaching or detaching objects) in unstructured environments. While Vision-Language-Action (VLA) models offer the potential to master diverse atomic skills, they struggle with the combinatorial complexity of sequencing them and are prone to cascading failures due to environmental sensitivity. To address these challenges, we propose LiLo-VLA (Linked Local VLA), a modular framework capable of zero-shot generalization to novel long-horizon tasks without ever being trained on them. Our approach decouples transport from interaction: a Reaching Module handles global motion, while an Interaction Module employs an object-centric VLA to process isolated objects of interest, ensuring robustness against irrelevant visual features and invariance to spatial configurations. Crucially, this modularity facilitates robust failure recovery through dynamic replanning and skill reuse, effectively mitigating the cascading errors common in end-to-end approaches. We introduce a 21-task simulation benchmark consisting of two challenging suites: LIBERO-Long++ and Ultra-Long. In these simulations, LiLo-VLA achieves a 69% average success rate, outperforming Pi0.5 by 41% and OpenVLA-OFT by 67%. Furthermore, real-world evaluations across 8 long-horizon tasks demonstrate an average success rate of 85%. Project page: https://yy-gx.github.io/LiLo-VLA/.
Yue Yang, Shuo Cheng, Yu Fang +4
University of North Carolina at Chapel Hill · Georgia Institute of Technology · Carnegie Mellon University
Humans perform long-horizon manipulation by retaining knowledge of what earlier actions have established while continuously adapting the motion underway. By contrast, action-chunked vision-language-action (VLA) policies repeatedly replan from the current input at each query. Existing methods preserve either long-term task evidence through memory or short-term motion through action reuse and ensembling, leaving the cross-query handoff incomplete. We introduce ChainVLA, a 1.2B-parameter VLA policy that chains successive queries through a joint and revisable execution state. Progress Context combines a recurrent Working State with sparse event memory to carry observation-derived task progress, while Motion Tail feeds the preceding prediction's unexecuted continuation into state construction and action generation. Together, the two components condition a decoder that regenerates each action horizon under the latest observation, allowing the carried state to guide the next prediction without fixing it. ChainVLA reaches 62.8% average success on RMBench and 98.8% across four LIBERO suites, while removing Motion Tail or Progress Context reduces RMBench success to 11.2% and 3.0%, respectively. These asymmetric ablations are consistent with motion continuity helping preserve the observation stream from which task progress is inferred.
Yuzhi Huang, Weijue Bu, Ziyi Xiong +4
Shenzhen International Graduate School, Tsinghua University · China University of Mining and Technology · Shenzhen Technology University