Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation

Yansong Shi; Jiange Yang; Xijie Yang; Shaowei Zhang; Yuhan Zhu; Tao Lu; Limin Wang

cs.ROSep 30, 2026

Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation

Authors: Yansong Shi, Jiange Yang, Xijie Yang, Shaowei Zhang, Yuhan Zhu, Tao Lu, Limin Wang

Organizations: School of Information Science And Technology, University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · State Key Lab of Novel Software Technology, Nanjing University · Computer Science and Technology, Zhejiang University · Computer Science and Technology, Shanghai Jiaotong University

Abstract

Recent advances in robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments. However, reliable execution often depends on hidden task states that cannot be determined from current observations alone, making interaction history essential. We introduce $HIDE$ , a benchmark for evaluating manipulation memory under partial observability. HIDE comprises 15 tasks covering repetition counting, historical-state recall, and execution-progress tracking, with randomized initial configurations and decision points where similar observations require different actions depending on prior events. We further propose $SEEK$ , a framework combining three complementary memory mechanisms to retain historical evidence and track execution state. Evaluations reveal substantial limitations in existing policies on HIDE, while memory augmentation improves task success in both simulation and real-world experiments. Individual mechanisms benefit some tasks but can degrade others; their combination achieves the highest average success rate on HIDE among the evaluated configurations. These findings highlight the importance of maintaining internal representations of hidden task states and matching memory design to task-specific information requirements.

Figures & tables

Figure 1: Overview of HIDE and SEEK. HIDE covers three hidden-state requirements and provides longer, more diverse trajectories than RLBench, while SEEK achieves the highest overall performance on the benchmark.

Figure 2: Detailed trajectories for the three HIDE task categories, with step order and repeated/checking operations indicated by colored arrows and labels.

Figure 3: Overview of SEEK. Windowed Context Memory (WCM) retains recent interactions, Persistent Anchor Memory (PAM) retrieves relevant historical evidence, and Stage-Counter Memory (SCM) tracks execution progress. Together, they condition the coarse-to-fine manipulation policy.

		Repetition Counting (RC)						Historical-State Recall (HSR)						Execution-Progress Tracking (EPT)
Method	Avg.	Push Button	Stack Blocks	Change Channel	Stack Cups	Stack Blocks-N	Avg.	Weigh On/Off	Pour Back	Wipe Desk	Light Bulb	Reopen Drawer	Avg.	Search Drawer	Search Boxes	Search Cabinet	Lift & Check	Swap Pegs	Avg.
OpenVLA ( Kim et al., 2024 )	1.6	8	0	4	0	4	3.2	8	0	0	0	0	1.6	0	0	0	0	0	0.0
OpenVLA-OFT Kim et al. (2025)	13.9	12	0	28	0	0	8.0	36	0	68	0	32	27.2	0	4	0	28	0	6.4
$\pi_{0}$ ( Black et al., 2024 )	22.1	40	0	12	0	0	10.4	40	24	92	0	24	36.0	12	28	36	24	0	20.0
$\pi_{0.5}$ ( Physical Intelligence et al., 2025 )	14.4	16	0	4	4	4	5.6	20	8	76	0	20	24.8	4	8	20	32	0	12.8
GR00T-N1.7 ( NVIDIA et al., 2025 )	26.4	12	0	12	4	4	6.4	96	0	84	0	68	49.6	28	16	40	24	8	23.2

Table 1: Performance on HIDE. Success rates (%) on individual tasks and their category averages. The benchmark contains three categories: repetition counting, historical-state recall, and execution-progress tracking. The best and second-best results in each column are bold and underlined , respectively, including ties.

Panel of Figure 4 — Figure 4: Memory component analysis. Category success rates (%) for progressive integration (a) and individual components versus their combination (b). The components show complementary strengths across hidden-state requirements. Radar axes use category-specific scales.

Table 6

Method	Avg.	Push button $\times$ N	Stack N cups (M total)	Clean desk	Search chip
$\pi_{0.5}$	13	0	20	0	32
SAM2Act+	47	32	48	44	64
SEEK	89	76	100	80	100

Table 4: Real-world results. Success rates (%); Avg. is the unweighted mean across tasks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Task name	Language Template	Frames	Keyframes	# of Var.	Variation Type
push_button_times	“push the [color] button [N] times”	144.3	7.8	50	repetition count $\times$ button color
stack_blocks	“stack [N] [color] blocks”	372.8	16.1	60	stack size $\times$ color
change_channel_times	“turn the channel [direction] [N] times”	244.5	12.2	6	direction $\times$ repetition count
stack_cups_new	“stack [N] cup(s) on top of the [color] cup”	208.8	8.4	40	base color $\times$ cup count
stack_blocks_new	“put [N] [color] blocks into the rectangular container”	283.5	11.8	60	placement count $\times$ color
weighing_on_off	“weigh the pepper and put it in another container”	199.0	11.0	2	starting container

Appendix

Table 5: Task Statistics of HIDE. We report the language template, average number of frames, average number of extracted keyframes, number of task variations, and variation type. Square brackets denote variable placeholders in the language templates, with [N] indicating the requested count.

Task	Language Template	Frames	Keyframes	# Var.	Variation Type
push_button_times	“push the [color] button [N] times”	144.3	7.8	3	repetition count $\times$ randomized object placement
stack_cups	“stack [N] cups on the middle cup”	372.8	16.1	4	target stack count $\times$ randomized object placement
wipe_desk_rubbish	“clean the desk”	272.9	9.0	2	rubbish location $\times$ randomized object placement
lift_and_check	“lift the blocks and look for the white piece”	277.3	14.3	3	hidden-piece location $\times$ randomized object placement

Appendix

Table 6: Real-world task statistics. Language templates, trajectory statistics, and variation types for the four real-world tasks.

Figure 5: Illustration of Repetition Counting (RS) tasks.

Figure 6: Illustration of Historical-State Recall (HSR) tasks.

Figure 7: Illustration of Execution-Progress Tracking (EPT) tasks.

Hyperparameter	Stage 1	Stage 2
Number of GPUs	8	8
Batch size (frames)	56	448
Batch size (temporal sequences)	–	64
Frames per sequence	–	7
Gradient accumulation	1	1
Peak learning rate	$7\times 10^{-4}$	$2\times 10^{-4}$

Appendix

Table 7: Training hyperparameters of SEEK on HIDE. Stage 1 learns the memory-free policy; Stage 2 trains the memory-augmented policy from the Stage 1 checkpoint. Batch sizes are global across all GPUs.

Figure 8: Scene of Real-world experiment.

Panel of Figure 9 — Figure 9: Real-world task setups. The four physical tasks used for real-robot evaluation: repeated button pressing, cup stacking, desk cleaning, and lifting blocks to search for a white piece.

Method	Avg. Success $\uparrow$	Avg. Rank $\downarrow$	Close Jar	Drag Stick	Insert Peg	Meat off Grill	Open Drawer	Place Cups	Place Wine	Push Buttons
Image-BC (CNN) ( Jang et al., 2022 )	1.3	14.08	0.0	0.0	0.0	0.0	4.0	0.0	0.0	0.0
Image-BC (ViT) ( Jang et al., 2022 )	1.3	14.25	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
C2F-ARM-BC ( James et al., 2022 )	20.1	13.03	24.0	24.0	4.0	20.0	20.0	0.0	8.0	72.0
HiveFormer ( Guhur et al., 2022 )	45.3	11.06	52.0	76.0	0.0	100.0	52.0	0.0	80.0	84.0
PolarNet ( Chen et al., 2023 )	46.4	10.19	36.0	92.0	4.0	100.0	84.0	0.0	40.0	96.0
PerAct ( Shridhar et al., 2023 )	49.4 $\pm$ 4.3	9.83	55.2 $\pm$ 4.7	89.6 $\pm$ 4.1	5.6 $\pm$ 4.1	70.4 $\pm$ 2.0	88.0 $\pm$ 5.7	2.4 $\pm$ 3.2	44.8 $\pm$ 7.8	92.8 $\pm$ 3.0

Appendix

Table 8: Full Comparisons of Multi-Task Performance on RLBench. We report success rates (%) on 18 RLBench tasks ( James et al., 2020 ) . SAM2Act results reported in the original paper are shown in light gray. An asterisk ( ∗ ) denotes our evaluation using the officially released SAM2Act weights. Their slightly lower performance under our unified evaluation protocol may reflect differences in evaluation environments, dependency versions, and simulation stochasticity. The last two rows report our method without and with memory, respectively. Avg. Rank is recomputed across all listed rows by averaging per-task ranks, with average ranks assigned to ties. Bold and underlined values indicate the best and second-best results in each column across all rows, respectively; ties are marked equally.

Method	Clean	Average	MO-Color	RO-Color	MO-Texture	RO-Texture	MO-Size	RO-Size
PerAct	$34.5\;(0.0)$	$28.7\;(\downarrow 16.6)$	$24.0\;(\downarrow 30.3)$	$31.7\;(\downarrow 14.4)$	$28.8\;(\downarrow 24.2)$	$17.7\;(\downarrow 16.2)$	$33.6\;(\downarrow 14.0)$	$29.3\;(\downarrow 15.2)$
RVT †	$43.6\;(0.0)$	$36.3\;(\downarrow 16.7)$	$26.0\;(\downarrow 40.4)$	$31.3\;(\downarrow 29.3)$	$44.8\;(\uparrow 4.7)$	$41.1\;(\uparrow 4.3)$	$35.3\;(\downarrow 16.3)$	$40.5\;(\downarrow 18.2)$
RVT-2	$67.8\;(0.0)$	$59.5\;(\downarrow 12.3)$	$53.0\;(\downarrow 21.8)$	$59.2\;(\downarrow 9.8)$	$\mathbf{59.7}\;(\downarrow 8.3)$	$\underline{56.7}\;(\downarrow 0.3)$	$60.9\;(\downarrow 5.9)$	$53.4\;(\downarrow 17.2)$
SAM2Act	$64.7\;(0.0)$	$\mathbf{62.3}\;(\downarrow 3.7)$	$\mathbf{65.0}\;(\uparrow 0.4)$	$61.0\;(\downarrow 0.9)$	$\underline{55.4}\;(\downarrow 10.0)$	$50.3\;(\downarrow 9.6)$	$\mathbf{65.4}\;(\downarrow 7.5)$	$\mathbf{58.0}\;(\downarrow 16.2)$
w/o memory (Ours)	$\underline{68.4}\;(0.0)$	$61.5\;(\downarrow 10.1)$	$62.1\;(\downarrow 9.3)$	$\underline{65.2}\;(\downarrow 1.2)$	$52.1\;(\downarrow 11.1)$	$\mathbf{56.8}\;(\downarrow 12.6)$	$\underline{63.1}\;(\downarrow 10.3)$	$\underline{57.3}\;(\downarrow 20.6)$
w/ memory (Ours)	$\mathbf{68.6}\;(0.0)$	$\underline{61.9}\;(\downarrow 9.8)$	$\underline{62.4}\;(\downarrow 9.0)$	$\mathbf{65.7}\;(\downarrow 0.9)$	$52.7\;(\downarrow 11.0)$	$56.6\;(\downarrow 12.9)$	$62.4\;(\downarrow 11.4)$	$\underline{57.3}\;(\downarrow 20.6)$

Appendix

Table 9: The Colosseum results. We compare the selected methods. Entries report success rates (%), with relative changes (%) from the corresponding unperturbed evaluation in parentheses. Non-dagger literature results are aggregated over the perturbation-specific task subsets used in SAM2Act. Relative changes use the Clean means of the same task subsets. Average is the unweighted mean over the 12 individual perturbations, excluding All-Mixed; its relative change is measured against overall Clean. PerAct and RVT are aggregated from the original benchmark paper; RVT-2 is aggregated from the BridgeVLA paper; SAM2Act uses paper-reported results. Both memory variants are our methods.

\dagger

denotes legacy RVT results with different task subsets, retained for reference and excluded from ranking. The best and second-best displayed success rates among the listed non-dagger methods are bold and underlined, respectively; ties share the same rank.

Explore similar work

Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation
Jun 19, 2026Kuancheng Wang, Seungho Yeom, Jinglin Cao +3Robotic Manipulation Policies Visuomotor Policy
UC San Diego
LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks
Apr 18, 2026Xueyao Chen, Jingkai Jia, Tong Yang +3Long-Horizon Manipulation Robotic Manipulation Policies
Fudan University
T $^2$ Mem: Learning Test-Time Memory for Robotics
Sep 29, 2026Yize Liu, Huang Huang, Yining Hong +4Robotic Manipulation Test Time
Stanford University · NVIDIA · University of Michigan, Ann Arbor

Jun 19, 2026, cs.ROJ/K move · Enter open · S save

Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation

Kuancheng Wang, Seungho Yeom, Jinglin Cao, Yuheng Zhi+2

UC San Diego

Robotic Manipulation Policies Visuomotor Policy Contact-Rich Manipulation Objects

Open paper AskPDF arXiv

Long horizon, contact-rich manipulation is inherently partially observable. This is as a single visual observation rarely captures a robot's full action context, including prior attempts, interactions, or progress. Consequently, standard visuomotor policies or vision-language-action models are prone to struggle in such tasks due to a lack of memory. To address this, we introduce Compressed Action Memory Policy (CAMP) based on the insight that a robot's own action history serves as a highly informative, self-supervised signal, enabling the policy to learn a robust, compact history representation. In our approach, we train a memory module to maintain a compressed representation of past actions, forcing it to encode a latent behavioral memory of all the robot's past interactions that can then be used to better contextualize future actions. This allows our approach to implicitly track generalized task progress and learn from failed attempts without any additional supervision, or external oversight. We evaluate CAMP across four real-robot setups and two novel simulation benchmarks: Memory-T-Bench and Memory-Manip-Bench. By demonstrating substantial gains over state-of-the-art baselines, CAMP is, to our knowledge, the first policy to demonstrate substantial success on contact-rich partially observable manipulation tasks purely through learned memory.

Abstract

Explore similar work

Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation

LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks

T2^22Mem: Learning Test-Time Memory for Robotics

T $^2$ Mem: Learning Test-Time Memory for Robotics