VLM-Guided Experience Replay

Elad Sharony; Tom Jurgenson; Orr Krupnik; Dotan Di Castro; Shie Mannor

cs.LGFeb 2, 2026

VLM-Guided Experience Replay

Authors: Elad Sharony, Tom Jurgenson, Orr Krupnik, Dotan Di Castro, Shie Mannor

Organizations: Technion · ForSight Robotics · Nvidia Research

Abstract

Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample efficiency, high-level planning, and interpretability in reinforcement learning (RL). While prior work has integrated LLMs and VLMs into various components of RL, the replay buffer, a core component for storing and reusing experiences, remains unexplored. We propose addressing this gap by leveraging VLMs to guide the prioritization of experiences in the replay buffer. Our key idea is to use a frozen, pre-trained VLM as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences. Across scenarios, including game-playing and robotics, spanning both discrete and continuous domains, agents trained with our proposed prioritization method achieve 15-57% higher average success rates and improve sample efficiency by 35-55% compared to previous approaches. Project page: https://esharony.me/projects/vlm-rb/

Figures & tables

Figure 1: VLM-RB learns more from the same interaction budget. Mean success across 12 algorithm–task configurations, averaged across 5 seeds. Shading denotes standard error.

Figure 2: System diagram: (1) Data is collected with the current policy $\pi_{k}$ interacting with multiple instances of the environment. (2) Transitions $(s,a,r,s^{\prime})$ are kept in a prioritized replay buffer with a provisional VLM score $\bar{\bm{\mathrm{p}}}$ . (3) Asynchronously, after each insertion, a VLM worker scores the corresponding rendered clip $\tau^{O}$ under a prompt $\mathsf{P}$ and writes the resulting score $\bm{\mathrm{p}}^{\scriptscriptstyle\mathrm{VLM}}$ back to the replay buffer (see Section 3.1 ). (4) The learner samples a minibatch using the mixture distribution $q_{t}$ which interpolates between uniform and VLM-prioritized replay (see Section 3.1 ). (5) The agent policy is updated to $\pi_{k+1}$ .

Figure 3: Temporal context resolves visual ambiguity. From the initial state $o_{i}$ (left), multiple futures are possible. The top sub-trajectory shows a successful grasp, while the bottom shows stagnation. By scoring sub-trajectories rather than single frames, the VLM has sufficient context to distinguish meaningful progress from failure modes.

Figure 4: Frozen VLM scoring qualitatively anticipates learned value. We visualize a single reference episode in MiniGrid/DoorKey-16x16 which contains both goal-directed actions and random noise. Left: Visualizations of two 32-frame sub-trajectories: clip A captures a semantically relevant sequence, while clip B shows a mostly random walk which eventually reaches the goal. Top right: A timeline of ground-truth events; sparse reward is only received at the final goal. Both right panels share the reference-episode timestep on the x-axis. Bottom right: The rose curve shows the frozen VLM score for

L=32

-frame clips. The gray curves show the temporal value difference,

\Delta Q(t;L)=Q(t+\tfrac{L}{2})-Q(t-\tfrac{L}{2})

, calculated from critic checkpoints at increasing training steps (light to dark).

Figure 5: Success depends on alignment with semantic priors. We compare the performance of VLM-RB across frames rendered with Standard visuals (dark circles), Misleading swapped sprites (medium squares), and Abstract textures (light diamonds). The dashed line indicates the ASR of PER . Crucially, the agent’s actual observations and the underlying MDP are identical across all settings; only the visual input to the VLM is altered (see Appendix F.11 for visual samples of these modifications). The significant performance drop in the Misleading and Abstract settings confirms that our method relies on the VLM correctly recognizing specific semantic objects (e.g., keys and doors) to accelerate learning.

		vs. UER		vs. PER
Algorithm	Level	Performance ( $\uparrow$ )	Sample Efficiency ( $\downarrow$ )	Performance ( $\uparrow$ )	Sample Efficiency ( $\downarrow$ )
DQN+IQN (DoorKey)	8 $\times$ 8	0% (1.00/1.00)	22% (139K/178K)	0% (1.00/1.00)	26% (139K/187K)
	12 $\times$ 12	61% (1.00/0.62)	57% (277K/640K)	22% (1.00/0.82)	37% (292K/467K)
	16 $\times$ 16	283% (0.92/0.24)	53% (420K/893K)	92% (0.92/0.48)	51% (305K/622K)
SAC+TD3 (Scene)	3	0% (1.00/1.00)	40% (213K/354K)	0% (1.00/1.00)	20% (213K/266K)
	4	22% (1.00/0.82)	62% (295K/776K)	2% (1.00/0.98)	38% (394K/634K)

Table 1: Improvement with respect to baselines. Performance denotes the mean per-seed best ASR (

M=5

seeds). Sample efficiency tracks the steps required to reach the baseline’s best performance; unreached targets are assigned the 1M-step horizon. Values in parentheses indicate the relative improvement over the baseline. Both metrics are averaged across the aggregated algorithms. The corresponding training curves are provided in Fig. 11 .

Figure 6: VLM-RB overcomes the failure modes of heuristic prioritization in sparse-reward tasks. Aggregated success rates on MiniGrid/DoorKey across grid sizes. While alternative methods (AER, ERO, ReLo) fail as they depend on dense rewards or local similarity metrics, VLM-RB successfully bridges the long-horizon dependencies. Curves show the mean across 5 seeds, with shaded standard errors.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Algorithm 1 VLM-prioritized replay buffer

Network Architecture
Hyperparameter	TD3	SAC
Hidden Dimensions	[512, 512, 512]
Q-Network Layer Norm	True	True
Optimization
Critic Learning Rate	$3\times 10^{-4}$
Actor Learning Rate	$3\times 10^{-4}$	$1\times 10^{-3}$

Appendix

Table 2: Hyperparameters and implementation details for OGBench experiments. Shared parameters are listed once; divergent parameters are compared side-by-side.

Optimization
Hyperparameter	DQN	IQN
Learning Rate	$4\times 10^{-5}$
Batch Size	128
Discount Factor ( $\gamma$ )	0.95
Target Update Frequency	1,000 steps
Target Update Rate ( $\tau$ )	1.0 (Hard Update)

Appendix

Table 3: Hyperparameters and implementation details for MiniGrid experiments. Shared parameters are listed once; divergent parameters are compared side-by-side.

Priority	DoorKey-16x16	Scene-5
UER	0.20 (920K, never)	0.44 (900K, never)
PER (TD-only)	0.76 (330K, 850K)	0.62 (570K, 970K)
VLM-only	1.00 (310K, 410K)	0.88 (470K, 670K)
$\bm{\mathrm{p}}^{\scriptscriptstyle\mathrm{VLM}}\|\delta\|$	1.00 (310K, 410K)	0.94 (320K, 440K)

Appendix

Table 4: Prioritization variants on DoorKey-16x16 /DQN and Scene-5 /SAC (5 seeds per cell). Entries report ASR at 1M steps, followed in parentheses by the steps required to reach the best ASR attained by UER and PER , respectively. “Never” means that the target was not reached within 1M steps.

Edge $\tau$	0.25	0.30	0.35	0.40	0.45	0.50	0.55	0.60	0.65
Bin share (%)	1.45	13.95	10.30	33.05	11.30	10.50	16.00	3.20	0.25
Balanced accuracy	0.500	0.509	0.597	0.662	0.871	0.938	0.927	0.583	0.506

Appendix

Table 5: Offline threshold selection on the corpus. Bin share is the percentage of scores in each bin, indexed by its lower edge.

Score form	ASR@1M	Steps to 0.8
$\tau=0.40$	$0.53\pm 0.27$	$755\mathrm{K}\pm 125\mathrm{K}$ (2/3)
$\tau=0.45$	$1.00\pm 0.00$	$547\mathrm{K}\pm 135\mathrm{K}$
$\tau=0.50$	$1.00\pm 0.00$	$357\mathrm{K}\pm 32\mathrm{K}$
Threshold-free ( $r^{\scriptscriptstyle\mathrm{VLM}}$ )	$1.00\pm 0.00$	$537\mathrm{K}\pm 143\mathrm{K}$

Appendix

Table 6: Downstream VLM-only threshold ablation on DoorKey-16x16 /DQN (3 seeds per configuration). The threshold-free row uses the raw score

r^{\scriptscriptstyle\mathrm{VLM}}

directly as the VLM-only priority. ASR is measured at 1M steps; steps to 0.8 are averaged over solved runs. Uncertainty is SEM over seeds.

		Improvement
$\lambda_{\max}$	Base.	Perf. ( $\uparrow$ )	Sample Eff. ( $\downarrow$ )
Pure priority from start	UER	N/A (0.00/0.24)	N/A (1000K/916K)
Pure priority from start	PER	N/A (0.00/0.80)	N/A (1000K/592K)
0.25	UER	+233.3% (0.80/0.24)	+40.6% (544K/916K)
0.25	PER	+0.0% (0.80/0.80)	-0.3% (594K/592K)
0.50	UER	+316.7% (1.00/0.24)	+62.9% (340K/916K)

Appendix

Table 7: Ablation of mixing schedule (

\lambda_{\max}

) on DoorKey-16x16 with VLM-only priority. Performance denotes the highest average success rate (

M=5

seeds). Sample efficiency tracks the steps required to reach the baseline’s best performance.

Figure 7: MiniGrid/DoorKey-16x16 :

\lambda_{\mathrm{max}}

ablation (5 seeds). “Pure priority from start” disables the warm-up and sets

\lambda_{t}=1

throughout. Left: performance for different

\lambda_{\mathrm{max}}

values. Right:

\lambda_{\mathrm{max}}

value evolution.

Transition class	Buffer share	Frequency / uniform
VLM-positive	$18.8\%$	$1.41\times$
Successful-episode subset	$1.4\%$	$2.17\times$
VLM-negative	$81.1\%$	$0.91\times$
No VLM score yet	$0.1\%$	$0.44\times$

Appendix

Table 8: Realized replay frequencies at the 250K buffer snapshot in Scene-5 /SAC (five seeds). The successful-episode subset is contained within the VLM-positive row. A ratio of

1.0

is uniform replay, and

0.5

is the instantaneous mixture floor. Unscored transitions are newest, so their cumulative ratio may be lower because they have been present for fewer sampling steps.

Model	Load (GiB)	Peak (GiB)	Time (s)	FPS
1B	2.86	3.77	0.46	69.27
3B	6.56 (+130%)	8.16 (+116%)	0.77 (+66%)	41.75 (-40%)
8B	18.25 (+539%)	20.34 (+439%)	2.15 (+366%)	14.88 (-79%)

Appendix

Table 9: Performance comparison of Perception-LM ( Cho et al., 2025 ) models. Values in parentheses denote the relative percentage change compared to 1B. We use an Nvidia RTX 4090 GPU, and run 100 batches of 32 frames.

Figure 8: MiniGrid/DoorKey-16x16 : VLM size ablation (5 seeds)

Model	$\bm{\tau}$	Bal. acc.	AUC	Agreement	$\bm{\rho}$
Perception-LM-1B	0.50	0.94	0.98	100%	1.00
Gemma-4-E4B	0.12	0.88	0.92	86%	0.70
Qwen3-VL-2B	0.89	0.85	0.89	81%	0.73

Appendix

Table 10: VLM-family diagnostics on the offline corpus. Balanced accuracy and AUC use the held-out half; agreement and raw-score Spearman

\rho

use the full corpus relative to Perception-LM-1B.

Model	$\bm{f_{\scriptscriptstyle\mathrm{VLM}}}$	Calibration	ASR@1M
Perception-LM-1B	Raw score	None	$1.00\pm 0.00$
Gemma-4-E4B	Raw score	None	$0.80\pm 0.15$
Gemma-4-E4B	$\mathbf{1}[r^{\scriptscriptstyle\mathrm{VLM}}>0.12]$	Offline threshold	$1.00\pm 0.00$

Appendix

Table 11: VLM-only downstream comparison on DoorKey-16x16 /DQN. The Perception-LM row contains 3 seeds; both Gemma rows contain 5. ASR is measured at 1M steps and uncertainty is SEM over seeds.

Ground-truth definition	Precision	Recall	FPR	FNR
Strict success	0.69	0.99	0.12	0.01
Milestone acquisition	0.92	0.62	0.04	0.38

Appendix

Table 12: Reliability of the default binarized scorer at

\tau=0.5

on the offline corpus.

$\bm{\kappa}$	0.00	0.10	0.25	0.50	0.75
ASR@1M	$1.00\pm 0.00$	$0.96\pm 0.04$	$1.00\pm 0.00$	$0.40\pm 0.24$	$0.44\pm 0.20$
vs. UER (0.20)	$+0.80$	$+0.76$	$+0.80$	$+0.20$	$+0.24$
Worst deficit	$0.0\%$	$0.0\%$	$0.1\%$	$4.0\%$	$21.4\%$

Appendix

Table 13: Controlled clip-label corruption on DoorKey-16x16 /DQN with VLM-only priority (five seeds per configuration; mean ASR at 1M steps

\pm

SEM). The deficit is the worst seed’s terminal uncompensated demotions as a fraction of the cumulative positive-label count.

Prompt	Base	Paraph. A	Paraph. B	Terse	Under.	AUC
Base	–					0.989
Paraphrase A	0.909 / 0.892	–				0.989
Paraphrase B	0.945 / 0.956	0.953 / 0.911	–			0.987
Terse	0.300 / 0.931	0.210 / 0.857	0.254 / 0.933	–		0.983
Underspecified	0.701 / 0.881	0.791 / 0.813	0.747 / 0.909	0.000 / 0.858	–	0.955

Appendix

Table 14: Prompt sensitivity on the offline corpus. Lower-triangular entries are fixed-

0.5

label agreement / raw-score Spearman correlation; the last column is strict-success ROC AUC.

Figure 9: Fig. 4 protocol on LavaCrossing . On a successful episode, the frozen VLM score anticipates the learned value signal around the gap crossing.

Panel of Figure 10 — Figure 10: Samples of the modified visuals. We modify only the frames passed to the VLM for scoring (agent observations and environment dynamics unchanged).

Hardware	PER	VLM-RB	Rel. Speed
NVIDIA A100	111	97	$87\%$
NVIDIA A40	92	81	$88\%$
NVIDIA A4000	76	67	$88\%$

Appendix

Table 15: Training throughput (steps/second) comparison on MiniGrid/DoorKey-16x16 . Higher is better.

Alg.	Task	Baseline	Best ASR	Steps to Base. Best	Time vs. Baseline
			Performance ( $\uparrow$ )	Sample Efficiency ( $\downarrow$ )	Wall-Clock Saving ( $\uparrow$ )
SAC	Scene-3	UER	+0.0% (1.00/1.00)	+50.7% (224K/454K)	+44.7%
	Scene-3	PER	+0.0% (1.00/1.00)	+18.8% (224K/276K)	+9.1%
	Scene-4	UER	+4.2% (1.00/0.96)	+54.2% (378K/826K)	+48.7%
	Scene-4	PER	+4.2% (1.00/0.96)	+52.8% (378K/800K)	+47.1%
	Scene-5	UER	+113.6% (0.94/0.44)	+60.0% (366K/914K)	+55.2%

Appendix

Table 16: Performance denotes the mean per-seed best success rate (

M=5

seeds). Sample efficiency tracks the steps required to reach the baseline’s best performance; unreached targets are assigned the 1M-step horizon. Wall-clock saving estimates the reduction in real training time, accounting for a

12\%

inference overhead per step.

Env Type	Agg. Algorithms	Baseline	Mean Best ASR	Mean Steps to Base Peak
			Performance ( $\uparrow$ )	Sample Efficiency ( $\downarrow$ )
Scene	(SAC + TD3)	UER	+32.6% (0.96/0.73)	+55.2% (291K/650K)
Scene	(SAC + TD3)	PER	+15.1% (0.96/0.84)	+34.6% (372K/568K)
DoorKey	(DQN + IQN)	UER	+57.0% (0.97/0.62)	+51.1% (279K/570K)
DoorKey	(DQN + IQN)	PER	+27.0% (0.97/0.77)	+42.3% (245K/425K)

Appendix

Table 17: Performance denotes the mean per-seed best success rate (

M=5

seeds). Sample efficiency tracks the steps required to reach the baseline’s best performance. Both metrics are averaged across the aggregated algorithms and tasks.

Figure 11: VLM-RB consistently outperforms baselines across continuous and discrete tasks. The plots show aggregated success rates for four algorithms (DQN, IQN, SAC, TD3) on MiniGrid and OGBench domains. Shaded regions indicate standard error across seeds.

Explore similar work

Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
Apr 18, 2026Weiyu Ma, Yongcheng Zeng, Yan Song +4Large Language Model Reinforcement Learning Prioritized Experience Replay
King Abdullah University of Science and Technology (KAUST) · Chinese Academy of Sciences, Institute of Automation (CASIA) · AI Centre, Department of Computer Science, University College London +1
Rollout-Level Advantage-Prioritized Experience Replay for GRPO
Jun 3, 2026Gyeongtae Yoo, Sanghyeok Park, Soohyuk Jang +2Parallel Rollouts Replay
Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in AI, Seoul National University · AIIS, ASRI, INMC, and ISRC, Seoul National University
From Passive Reuse to Active Reasoning: Grounding Large Language Models for Neuro-Symbolic Experience Replay
May 10, 2026Yanan Xiao, Yixiang Tang, Zechen Feng +3Prioritized Experience Replay Hierarchical Reinforcement Learning
1Affiliation 1 · 2Affiliation 2 · 3Affiliation 3

Apr 18, 2026, cs.CLJ/K move · Enter open · S save

Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning

Weiyu Ma, Yongcheng Zeng, Yan Song, Xinyu Cui+3

King Abdullah University of Science and Technology (KAUST) · Chinese Academy of Sciences, Institute of Automation (CASIA) · AI Centre, Department of Computer Science, University College London+1

Large Language Model Reinforcement Learning Prioritized Experience Replay

Open paper AskPDF arXiv

Reinforcement Learning (RL) has achieved impressive success in post-training Large Language Models (LLMs) and Vision-Language Models (VLMs), with on-policy algorithms such as PPO, GRPO, and REINFORCE++ serving as the dominant paradigm. However, these methods discard all collected trajectories after a single gradient update, resulting in poor sample efficiency, particularly wasteful for agentic tasks where multi-turn environment interactions are expensive. While Experience Replay drives sample efficiency in classic RL by allowing agents to reuse past trajectories and prioritize informative ones, directly applying Prioritized Experience Replay (PER) to LLMs fails. The rapid policy evolution of billion-parameter models renders stored priorities stale, causing old high-priority trajectories to dominate sampling long after they have become uninformative. We propose Freshness-Aware PER, which addresses this priority staleness problem by augmenting any PER-based priority with a multiplicative exponential age decay grounded in effective sample size analysis. To the best of our knowledge, Freshness-Aware PER is the first work to successfully apply PER to LLM/VLM reinforcement learning. We evaluate on eight multi-step agentic, reasoning, and math competition tasks with 0.5B, 3B, and 7B models. Freshness-Aware PER significantly outperforms on-policy baselines, achieving +46% on NQ Search, +367% on Sokoban, and +133% on VLM FrozenLake, while standard PER without age decay consistently degrades performance. Our code is publicly available at https://github.com/Vision-CAIR/Freshness-Aware-PER.

Abstract

Explore similar work

Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning

Rollout-Level Advantage-Prioritized Experience Replay for GRPO

From Passive Reuse to Active Reasoning: Grounding Large Language Models for Neuro-Symbolic Experience Replay