Long-horizon robot manipulation requires a robot to remember task-relevant information beyond its short observation window. Existing memory-augmented policies typically retrieve relevant information from history and reason over it at execution time, placing a heavy reasoning burden on the high-level model. To address this challenge, we propose PAL (Plan first, Act Later), a hierarchical memory framework for vision-language robot manipulation. It maintains a short-term visual buffer, a semantic-event memory, and an executable subtask\_list. A high-level vision-language model (VLM) converts streaming visual observations into the subtask\_list. At execution time, the VLM selects the currently relevant subtask from the subtask\_list and passes it to a low-level vision-language-action (VLA) policy. To teach the model what to write to memory and when to update it, we introduce a two-stage data generation procedure that first expands sparse update supervision and then uses self-labeling with task-grounded verification to generate training data that better match the state distribution encountered during execution. We evaluate our approach on memory-dependent benchmarks and real-world manipulation tasks. On RoboMME, PAL achieves a 56.05% success rate, outperforming the strongest baseline, FrameSamp-Modul, at 44.51%. In real-world experiments across four manipulation tasks, \pal{} reaches 60.83% overall success, surpassing MemER (45.00%) and FrameSamp-Modul (28.33%). These results show that proactively writing future-relevant subtasks into hierarchical memory improves long-horizon visual reasoning and robot control. Code and videos can be found on our website: https://qinghangliu.github.io/pal-webpage/
Figures & tables
Figure 1: Overview of our pipeline. The high-level VLM incrementally converts streaming visual observations into hierarchical memory and retrieves the currently relevant subtask for execution by the low-level VLA policy.
Figure 2: Streaming memory update.
Figure 3: Dense update-timing supervision. A sparse update annotation is expanded into multiple training samples with visual buffers of different lengths, exposing the model to different levels of evidence around the reference update boundary.
Method
Short-video
Long-video
In-context
Episode-start
AVG
Baselines
π0.5
10.67
28.45
11.72
36.15
17.93±0.61
EventVLA
27.60
19.00
32.50
21.33
26.57±0.13
FrameSamp-Modul
41.87
46.00
27.05
72.96
44.51±0.77
MemER
37.52
28.00
41.00
65.11
42.38±0.33
GPT-6-astra + VLA
31.71
66.00
28.00
56.67
39.75
Table 1: Mean success rates (%) on RoboMME task groups. Short-video and Long-video infer an executable subtask list from conditioning videos of different lengths; In-context adapts it from observations during the episode; and Episode-start has all information needed to infer the complete list at the beginning of the episode. The AVG column reports mean ± standard deviation when multiple evaluation seeds are available.
PatternLock
RouteStick
VideoPlaceOrder
VideoPlaceButton
Variant
Update freq.
Final acc.
Update freq.
Final acc.
Update freq.
Final acc.
Update freq.
Final acc.
Sparse Label
2.68
22.00%
3.34
50.00%
0.00
0.00%
0.00
0.00%
(-14.6%)
(-8.7%)
(-100.0%)
(-100.0%)
Stage 1 Only
4.32
24.00%
5.28
30.00%
1.34
18.00%
1.38
18.00%
(+37.6%)
(+44.3%)
(+34.0%)
(+38.0%)
Update Gate
3.10
78.00%
3.66
82.00%
1.00
80.00%
1.00
78.00%
Table 2: Subtask-list update behavior. For each task, we report the average number of subtask_list updates per episode (Update freq.) and the accuracy of the final list (Final acc.). Parentheses report the relative change in update frequency from ground truth.
Figure 4: Real-world manipulation tasks used to evaluate PAL . The tasks require retaining visual evidence from previous observations to support subsequent manipulation.
Method
PatternLock
PickXtimes
VideoPlaceOrder
VideoUnmask
Overall
FrameSamp-Modul
30.00 ± 10.00
40.00 ± 30.00
13.33 ± 5.77
30.00 ± 0.00
28.33 ± 3.82
MemER
33.33 ± 5.77
0.00 ± 0.00
46.67 ± 30.55
100.00 ± 0.00
45.00 ± 6.61
Ours
40.00 ± 0.00
53.33 ± 15.28
50.00 ± 0.00
100.00 ± 0.00
60.83 ± 3.82
Table 3: Real-world success rates (%, mean ± sample standard deviation over three repeats of each 10-episode evaluation).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Configurations
Values
Network Architecture
Base VLM
Qwen3-VL-4B-Instruct
Low-level policy
Pretrained MemER VLA
Action chunk length
20
Fine-tuning
LoRA on language-model linear layers
Frozen modules
Vision tower and aligner
Appendix
Table 4: Network architecture and Stage 1 training hyperparameters of PAL .
Figure 5: Accepted episodes and normal rollout successes over the 15 self-labeling cycles. Each cycle relabels all 200 episodes of PatternLock and VideoPlaceOrder.
Method
Short-video
Long-video
In-context
Episode-start
AVG
Video Umsk
Video Repick
Video UmskS
Pattern Lock
Route Stick
Move Cube
Insert Peg
Video PlcBtn
Video PlcOrd
Button Umsk
Button UmskS
Pick HighL
Stop Cube
Pick Xtimes
Bin Fill
Swing Xtimes
Baselines
π0.5
20.44
0.44
18.67
2.89
4.67
26.00
1.56
31.11
25.78
22.22
6.67
11.33
6.67
42.89
30.00
35.56
17.93
EventVLA
66.00
30.00
17.60
4.00
27.60
44.00
4.00
22.00
16.00
60.00
12.00
12.00
46.00
48.00
6.00
10.00
26.57
FrameSamp-Modul
32.67
30.44
24.44
53.56
66.67
77.78
7.56
60.00
32.00
25.11
18.22
22.89
42.00
87.33
39.56
92.00
44.51
MemER
81.33
25.33
38.00
16.67
12.00
82.67
6.67
30.00
26.00
72.00
21.33
70.67
0.00
79.33
56.67
59.33
42.38
Appendix
Table 5: Full RoboMME results. Success rates (%) are reported for all 16 tasks, grouped by how the executable subtask list is obtained. The highest score for each task is in bold.
Configurations
Values
Conditioning-video sampling
Every 4 timesteps
Execution action chunk
16
VLM query
Once per action chunk
Visual buffer input
4 uniformly sampled frames
Policy seeds
42,43,44,45,46
Appendix
Table 6: PAL inference and evaluation protocol on RoboMME.
Task
Level 1
Level 2
Level 3
Level 4
PickXtimes
2 successful picks
3 successful picks
4 successful picks
–
VideoPlaceOrder
2 placements
3 placements
4 placements
5 placements
PatternLock
1 movement
2 movements
3 movements
4 movements
VideoUnmask
No difficulty levels
Appendix
Table 7: Difficulty settings for the real-world tasks. PickXtimes uses three levels, while VideoUnmask has no difficulty distinction.
Figure 6: Real-world rollout visualizations of PAL . From top to bottom, the rows show PatternLock, PickXtimes, VideoPlaceOrder, and VideoUnmask, with eight frames sampled along each rollout.
Figure 7: The four conditioning images used in the video-phase prompt above, in timestep order: 349 , 378 , 407 , and 436 .
Figure 8: The four execution images used in the execution-phase prompt above, in timestep order: 58 , 71 , 83 , and 96 .
Department of Automation, BNRist, Tsinghua University, Beijing, China · Department of Computer Science, The University of Hong Kong, Hong Kong, China · Dexmal, Beijing, China +1