Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models
Organizations: Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, School of Computer and Information Technology, Shanxi University · College of Intelligence and Computing, Tianjin University
Abstract
Exploration remains a major challenge for reinforcement learning (RL) agents in Craftax-style environments due to sparse rewards and long horizons. Large language models (LLMs) can guide exploration by generating high-level goals, but methods using models without explicit task adaptation often rely on carefully designed prompts and may produce inconsistent or contextually inappropriate goals. We introduce CCGP-2M, a multimodal goal-preference dataset constructed from interaction trajectories in Craftax-Classic, and propose a vision-language model (VLM)-guided RL framework that learns environment-specific goal guidance from this dataset. The framework consists of two stages: offline VLM fine-tuning and online goal-guided RL. In the offline stage, we align the VLM with goal preferences using Kullback-Leibler Token-level Direct Preference Optimization (KLTDPO), which combines token-level KL regularization with distributionally robust reweighting. Experiments on Craftax-Classic show that our method collects diamonds earlier than state-of-the-art baselines, achieves the highest score and reward, and attains the highest success rates on most achievements.
Figures & tables
| Statistic | Random | partial | Total |
| Environment steps | M | M | M |
| Trajectories | |||
| Observations | |||
| Observations with diamond |
| Method | Score (%) | Reward |
| PPO [ 25 ] | ||
| ELLM [ 7 ] | ||
| AdaRefiner [ 33 ] | ||
| Causal-aware LLMs [ 3 ] | ||
| SGRL (Qwen3-Coder-30B- A3B-Instruct) [ 21 ] | ||
| SGRL (DeepSeek-R1) [ 21 ] |
Goal-Conditioned RL
Goal-conditioned reinforcement learning trains agents to reach specified states or outcomes, enabling one policy to solve many tasks.
See all 101 papers