VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents

Pengsen Liu; Maosen Zeng; Nan Tang; Kaiyuan Li; Jing-Cheng Pang; Yunan Liu; Yang Yu

cs.LGMar 24, 2026

VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents

Authors: Pengsen Liu, Maosen Zeng, Nan Tang, Kaiyuan Li, Jing-Cheng Pang, Yunan Liu, Yang Yu

Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, China & School of Artificial Intelligence, Nanjing University, China · Computational Intelligence Center (CIC), School of Computer and Artificial Intelligence, Shandong Jianzhu University, Jinan, 250101, China

Abstract

Combining Large Language Models (LLMs) with Reinforcement Learning (RL) enables agents to interpret language instructions more effectively for task execution. However, LLMs typically lack direct perception of the physical environment, which limits their understanding of environmental dynamics and their ability to generalize to unseen tasks. To address this limitation, we propose Visual-Language Knowledge-Guided Offline Reinforcement Learning (VLGOR), a framework that integrates visual and language knowledge to generate imaginary rollouts, thereby enriching the interaction data. The core premise of VLGOR is to fine-tune a vision-language model to predict future states and actions conditioned on an initial visual observation and high-level instructions, ensuring that the generated rollouts remain temporally coherent and spatially plausible. Furthermore, we employ counterfactual prompts to produce more diverse rollouts for offline RL training, enabling the agent to acquire knowledge that facilitates following language instructions while grounding in environments based on visual cues. Experiments on robotic manipulation benchmarks demonstrate that VLGOR significantly improves performance on unseen tasks requiring novel optimal policies, achieving a success rate over 24% higher than the baseline methods.

Figures & tables

Figure 1: An example comparison of imaginary rollouts generated by KALM ( Pang et al., 2024 ) and our VLGOR. Relying solely on language knowledge, KALM does not explicitly model the spatial states of multiple objects. In contrast, VLGOR generates more realistic rollouts with smoother and physically plausible object motions.

Figure 2: Overall framework of the proposed VLGOR method. It consists of two stages: rollout generation for synthesizing imaginary rollouts from novel instructions, and skill acquisition for learning policies via offline reinforcement learning.

Figure 3: Illustration of BAGEL fine-tuning optimization via vertical data flow and rollout generation via horizontal data flow.

Figure 4: Success rates (%) of different methods on unseen tasks with varying difficulty levels. The top row reports success rate on the CLEVR-Robot environment, while the bottom row reports success rate on the Meta-World environment. Results are averaged over the last five checkpoints, with error bars indicating half the standard deviation across three random seeds.

Figure 5: Comparison of imaginary rollouts generated by KALM ( Pang et al., 2024 ) and the proposed VLGOR.

Number of samples	CLEVR-Robot				Meta-World
Number of samples	PSNR $\uparrow$	SSIM $\uparrow$	MED $\downarrow$	Success rate $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	MED $\downarrow$	Success rate $\uparrow$
5000	19.63 dB	0.7426	0.1762	22.53%	17.41 dB	0.6941	0.2015	16.15%
10000	22.33 dB	0.7915	0.1261	28.63%	21.25 dB	0.7315	0.1548	20.56%
10000	23.75 dB	0.8072	0.0933	32.95%	22.18 dB	0.7558	0.1334	23.14%
15000	23.94 dB	0.8123	0.0884	33.71%	22.64 dB	0.7612	0.1285	23.57%

Table 1: Effect of the number of samples used for VLM finetuning. PSNR, SSIM, and MED evaluate the quality of generated rollouts, while Success Rate measures instruction execution. Lower MED is better, while higher values are preferred for the other metrics.

Figure 6: Success rate curves on unseen counterfactual tasks. The x -axis denotes training epochs, while the y -axis denotes success rates of instruction completion in offline RL. Shaded regions represent half of the standard deviation over three random seeds.

Method	CLEVR-Robot			Meta-World
Method	Rephrasing	Counterfactual	Hard	Rephrasing	Counterfactual	Hard
Baseline	25.3	15.2	1.2	13.6	5.9	3.8
w/o target	40.6	36.3	14.3	21.8	22.4	13.1
w/o behavior	39.8	35.6	13.1	20.3	20.9	11.8
Our VLGOR	41.7	39.3	19.3	23.6	25.6	17.8

Table 2: Impact of different counterfactual prompts as instructions on task success rates (%).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Hyper-parameter	Setting
Diffusion timesteps	70
Text temperature	0.3
Classifier-free guidance interval	[0.0, 1.0]
Maximum thinking tokens	1024
VAE preprocess size	$256\times 256$
ViT preprocess size	$224\times 224$

Appendix

Table 3: VLM Hyper-parameter Configuration.

Hyper-parameter	Setting
Number of training epochs	50
Batch size	128
Learning rate	$3\times 10^{-4}$
Minimum learning rate	$1\times 10^{-6}$
Learning rate schedule	Cosine annealing
Stream fit	Yes

Appendix

Table 4: Offine RL Hyper-parameter Configuration.

Original Instruction	Type	Generated Counterfactual Instruction
Move the green ball toward the right.	Counterfactual Target	Move the blue ball toward the right.
Move the green ball toward the right.	Counterfactual Behavior	Move the green ball toward the left .

Appendix

Table 5: Examples of generated counterfactual instructions in the CLEVR-Robot environment. Red and blue fonts indicate the target and behavior, respectively.

Original Instruction	Type	Generated Counterfactual Instruction
Move the gripper to the target.	Counterfactual Target	Move the gripper to the left of the target position .
Move the gripper to the target.	Counterfactual Behavior	Move the gripper to the target region along a straight-line trajectory .

Appendix

Table 6: Examples of generated counterfactual instructions in the Meta-World environment.

Figure 7: Visualization of rollouts generated by different instructions.

Figure 8: Success rates (%) of different methods in the Meta-World environment.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 298 papers

Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents
Apr 15, 2026Angelo Moroncelli, Roberto Zanetti, Marco Maccarini +1RL for Robotics Data-Efficient Robot Learning
University of Applied Science and Arts of Southern Switzerland, Department of Innovative Technologies, IDSIA-SUPSI, Lugano, Switzerland · Università della Svizzera Italiana, Faculty of Informatics, Lugano, Switzerland · Politecnico di Milano, Mechanical Department, Milano, Italy
Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigation
Mar 27, 2026Dongsheng Yang, Yinfeng Yu, Liejun WangVision-Language Navigation Vision-Language Grounding
VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning
Oct 14, 2026Diyuan Shi, Yiqi Tang, Hongyin Zhang +2Test-Time Adaptation of VLA Models In-Context RL
Zhejiang University · Westlake University

Apr 15, 2026J/K move · Enter open · S save

Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents

Angelo Moroncelli, Roberto Zanetti, Marco Maccarini, Loris Roveda

University of Applied Science and Arts of Southern Switzerland, Department of Innovative Technologies, IDSIA-SUPSI, Lugano, Switzerland · Università della Svizzera Italiana, Faculty of Informatics, Lugano, Switzerland · Politecnico di Milano, Mechanical Department, Milano, Italy

RL for Robotics Data-Efficient Robot Learning Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation, but scaling to long-horizon tasks with sparse or imperfect rewards remains difficult due to inefficient exploration and poor credit assignment. Vision-Language-Action (VLA) models leverage large-scale multimodal pretraining to provide generalist, task-level reasoning, but current limitations hinder their direct use in fast and precise manipulation. In this paper, we propose Vision-Language-Action Jump-Starting (VLAJS), a method that bridges sparse VLA guidance with on-policy RL to improve exploration and learning efficiency. VLAJS treats VLAs as transient sources of high-level action suggestions that bias early exploration and improve credit assignment, while preserving the high-frequency, state-based control of RL. Our approach augments Proximal Policy Optimization (PPO) with a directional action-consistency regularization that softly aligns the RL agent's actions with VLA guidance during early training, without enforcing strict imitation, requiring demonstrations, or relying on continuous teacher queries. VLA guidance is applied sparsely and annealed over time, allowing the agent to adapt online and ultimately surpass the guiding policy. We evaluate VLAJS on six challenging manipulation tasks: lifting, pick-and-place, peg reorientation, peg insertion, poking, and pushing in simulation, and validate a subset on a real Franka Panda robot. VLAJS consistently outperforms PPO and distillation-style baselines in sample efficiency, reducing required environment interactions by over 50% in several tasks. Real-world experiments demonstrate zero-shot sim-to-real transfer and robust execution under clutter, object variation, and external perturbations.

Abstract

Offline RL

Explore similar work

Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents

Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigation

VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning