Value as Memory: Online Value-Guided Post-training for General Robot Policies

Wenye Yu; Yijin Chen; Kaiyi Xiong; Wenbo Han; Wendi Chen; Yang Jin; Han Xue; Jun Lv

cs.ROOct 15, 2026

Value as Memory: Online Value-Guided Post-training for General Robot Policies

Authors: Wenye Yu, Yijin Chen, Kaiyi Xiong, Wenbo Han, Wendi Chen, Yang Jin, Han Xue, Jun Lv,

Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.

Abstract

Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, while offline post-training methods delay feedback from deployment experience. We present Value as Memory (VaM), a framework that addresses these challenges through value caching and online real-world reinforcement learning (RL). Value caching conditions actions on recent value predictions as task-progress memory, while online updates enable timely policy improvement by combining offline demonstrations, off-policy interventions, and on-policy rollouts for sample-efficient adaptation. As a result, VaM transitions from offline RL warmup to online autonomous learning with sparse rewards. Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of $29.4\%$ with value caching, while achieving a $32.8\%$ gain with online autonomous learning over matched offline training. The evaluated online schedule performs $8\times$ as many policy iterations under the same gradient-step budget, with more than $2\times$ lower recorded time per gradient step with concurrent data collection and training.

Figures & tables

Figure 1: VaM uses online real-world RL and value caching to boost sample efficiency and mitigate partial observability in long-horizon, deformable-object, and contact-rich manipulation.

Figure 2: VaM training pipeline and model architecture. The top panel summarizes the four-stage offline-to-online training pipeline. The bottom panel shows how the distributional value expert turns previously computed value predictions into the value cache that conditions the action expert.

Figure 3: Task illustrations for Box Assembly (top), Laundry Folding (middle), and Magnetic Bus Assembly (bottom).

Figure 4: Across Box Assembly , Laundry Folding , and Magnetic Bus Assembly , value caching yields a mean absolute success-rate gain of

29.4\%

Method	Box Assembly $(\%)$	Laundry Folding $(\%)$	Magnetic Bus Assembly $(\%)$
VaM	$\mathbf{96.7}\,{\color[rgb]{0.5,0.5,0.5}\pm 2.9}$	$\mathbf{88.9}\,{\color[rgb]{0.5,0.5,0.5}\pm 10.2}$	$\mathbf{76.7}\,{\color[rgb]{0.5,0.5,0.5}\pm 5.8}$
VaM (w/o value caching)	$81.7\,{\color[rgb]{0.5,0.5,0.5}\pm 2.9}$	$48.9\,{\color[rgb]{0.5,0.5,0.5}\pm 30.1}$	$43.3\,{\color[rgb]{0.5,0.5,0.5}\pm 5.8}$
VaM (w/o offline RL warmup)	$81.7\,{\color[rgb]{0.5,0.5,0.5}\pm 7.6}$	$60.6\,{\color[rgb]{0.5,0.5,0.5}\pm 18.3}$	$50.0\,{\color[rgb]{0.5,0.5,0.5}\pm 10.0}$
RECAP	$71.7\,{\color[rgb]{0.5,0.5,0.5}\pm 16.1}$	$64.4\,{\color[rgb]{0.5,0.5,0.5}\pm 7.7}$	$43.3\,{\color[rgb]{0.5,0.5,0.5}\pm 5.8}$
VaM (demo-trained value)	$80.0\,{\color[rgb]{0.5,0.5,0.5}\pm 13.2}$	$71.1\,{\color[rgb]{0.5,0.5,0.5}\pm 20.4}$	$40.0\,{\color[rgb]{0.5,0.5,0.5}\pm 10.0}$
VaM (post-warmup value)	$85.0\,{\color[rgb]{0.5,0.5,0.5}\pm 13.2}$	$68.9\,{\color[rgb]{0.5,0.5,0.5}\pm 10.2}$	$50.0\,{\color[rgb]{0.5,0.5,0.5}\pm 10.0}$

Table 1: Success rate over the final three online RL checkpoints. Entries are mean

\pm

sample standard deviation across the final three online RL checkpoints. By replacing on-policy value estimates with counterfactual estimates at inference, the bottom block demonstrates the importance of on-policy values for effective value caching.

Panel of Figure 5 — Figure 5: Value caching supports progress-aware recovery: VaM (left) versus VaM (w/o value caching) (right) on Box Assembly . We illustrate the eye-to-hand camera images, PCA embeddings of the value expert colored by rollout progress, and scalar value predictions.

Method	Box Assembly $(\%)$	Laundry Folding $(\%)$	Magnetic Bus Assembly $(\%)$
VaM	$\mathbf{54.5}$	$\mathbf{86.4}$	$\mathbf{33.3}$
VaM (w/o value caching)	$50.0$	$58.8$	$7.4$
VaM (w/o offline RL warmup)	$25.3$	$72.7$	$27.3$
RECAP	$42.4$	$76.6$	$13.3$

Table 2: Recovery rate during LfE.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Real-robot experimental setup. The illustration summarizes the robot hardware, sensing configuration, human-intervention interface, and task workspaces used in our experiments.

Task	Offline demonstrations (episodes)	Offline RL warmup (episodes)	Online RL rollouts per round (episodes)
Box Assembly	200	100	20
Laundry Folding	90	50	15
Magnetic Bus Assembly	30	40	10

Appendix

Table 3: Task-specific data-collection budgets.

Task	$N$	VaM $(\%)$	VaM (w/o value caching) $(\%)$	Gap $(\%)$
Box Assembly	8	$96.25$	$80.00$	$16.25$
Laundry Folding	3	$88.89$	$48.89$	$40.00$
Magnetic Bus Assembly	4	$75.00$	$37.50$	$37.50$

Appendix

Table 4: Success rate over all available LfE checkpoints.

N

is the number of checkpoints per method.

Task	Method	# recovered / # required	# episodes	Recovery rate $(\%)$
Box Assembly	VaM	24/44	180	$54.5$
	VaM (w/o value caching)	35/70	180	$50.0$
	VaM (w/o offline RL warmup)	20/79	180	$25.3$
	RECAP	36/85	180	$42.4$
Laundry Folding	VaM	38/44	45	$86.4$
	VaM (w/o value caching)	30/51	45	$58.8$

Appendix

Table 5: Raw counts for the blind recovery evaluation.

Task	Setting	Time per gradient step (s)	Total experimental time (hours)
Box Assembly	Offline RL	$4.885\,{\color[rgb]{0.5,0.5,0.5}\pm 0.413}$	6.514
	Online RL	$1.887\,{\color[rgb]{0.5,0.5,0.5}\pm 0.107}$	2.516
Laundry Folding	Offline RL	$5.090\,{\color[rgb]{0.5,0.5,0.5}\pm 0.345}$	7.069
	Online RL	$2.253\,{\color[rgb]{0.5,0.5,0.5}\pm 0.057}$	3.129
Magnetic Bus Assembly	Offline RL	$5.891\,{\color[rgb]{0.5,0.5,0.5}\pm 0.105}$	7.213
	Online RL	$2.571\,{\color[rgb]{0.5,0.5,0.5}\pm 0.160}$	3.061

Appendix

Table 6: Experimental time per gradient step and total experimental time. Time per gradient step is reported as mean

\pm

sample standard deviation.

Hyperparameter	Scope	Task	Value
Value-support atoms $K$	Value expert	All tasks	256
Action horizon $H$ (steps)	Action expert	All tasks	100
Control frequency (Hz)	Deployment	All tasks	100
Progress-label std. dev. $\sigma_{\mathrm{init}}$	Value expert	All tasks	$3/512$
Target-network EMA coefficient $\beta$	Value expert	All tasks	1.0
Value-training stride $d$ (steps)	Value training	All tasks	30

Appendix

Table 7: Hyperparameters for value and action learning.

Panel of Figure 9 — Figure 9: Qualitative comparison of VaM (left) and VaM (w/o value caching) (right) on Laundry Folding (first three rows) and Magnetic Bus Assembly (last two rows). Each example shows eye-to-hand camera images, PCA embeddings of the value expert colored by rollout progress, and scalar value predictions. The dark-blue trace shows the 15-frame mean of scalar value predictions.

Panel of Figure 10 — Figure 10: Policy predictions after losing contact in Laundry Folding . Under identical visual observations, the policy using the original cached values predicts opening the right gripper wider and adjusting its pose to prepare for another attempt to grasp the garment corner, compared with using demo-trained or post-warmup values as the cache.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 283 papers

OnEvoMemory: Evolving Memory through Online Robot Rollouts for Pretrained Robot Policies
Aug 9, 2026Zhongxi Chen, Shenqi ZongRobot Policy Learning Long-Horizon Robotic Manipulation
Shanghai Jiao Tong University · Tsinghua University
Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
Jul 10, 2026Wenke Xia, Pei Ren, Wenbo Yu +10Value Function Estimation Robotic Manipulation
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Beijing Innovation Center of Humanoid Robotics · 3Beijing Forestry University +4
Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement
Sep 19, 2026Jiarui Yang, Jiajin Zhang, Bin Zhu +2Robot Policy Learning RL Post-Training
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University

Aug 9, 2026J/K move · Enter open · S save

OnEvoMemory: Evolving Memory through Online Robot Rollouts for Pretrained Robot Policies

Zhongxi Chen, Shenqi Zong

Shanghai Jiao Tong University · Tsinghua University

Robot Policy Learning Long-Horizon Robotic Manipulation Policy Learning Memory-Augmented Agents

Open paper AskPDF arXiv

Long-horizon robot manipulation requires policies to track completed subtasks and critical interaction events. However, existing memory mechanisms heavily rely on external models or predefined update rules. To address this, we propose OnEvoMemory, a value-guided memory module for pretrained robot policies. It maintains recent context, high-value experiences, and salient transitions, while learning which experiences should be retained from trajectory outcomes. Offline demonstrations initialize the memory prior, whereas successful and unsuccessful online rollouts refine memory selection, helping the policy recognize task-stage transitions and avoid repeating completed subtasks. Experiments on long-horizon manipulation benchmarks show that OnEvoMemory improves the performance of the base VLA policy through both offline initialization and online memory evolution.

Abstract

Robotic RL

Explore similar work

OnEvoMemory: Evolving Memory through Online Robot Rollouts for Pretrained Robot Policies

Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning

Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement