cs.LGNov 19, 2026

Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models

Authors: Yajie Qi, Wei Wei, Lin Li, Lijun Zhang, Jianye Hao

Organizations: Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, School of Computer and Information Technology, Shanxi University · College of Intelligence and Computing, Tianjin University

Abstract

Exploration remains a major challenge for reinforcement learning (RL) agents in Craftax-style environments due to sparse rewards and long horizons. Large language models (LLMs) can guide exploration by generating high-level goals, but methods using models without explicit task adaptation often rely on carefully designed prompts and may produce inconsistent or contextually inappropriate goals. We introduce CCGP-2M, a multimodal goal-preference dataset constructed from interaction trajectories in Craftax-Classic, and propose a vision-language model (VLM)-guided RL framework that learns environment-specific goal guidance from this dataset. The framework consists of two stages: offline VLM fine-tuning and online goal-guided RL. In the offline stage, we align the VLM with goal preferences using Kullback-Leibler Token-level Direct Preference Optimization (KLTDPO), which combines token-level KL regularization with distributionally robust reweighting. Experiments on Craftax-Classic show that our method collects diamonds earlier than state-of-the-art baselines, achieves the highest score and reward, and attains the highest success rates on most achievements.

Figures & tables

Explore similar work

CardsList
  1. GIFT: Goal-Injected Fine-Tuning for Efficient Manipulation Policy Adaptation

    Sep 7, 2026Xiaoyuan Fang, Shuo Feng, Yuxuan Wang +3Goal-Conditioned Robotic ManipulationVision-Language-Action Models

  2. RM-VLM: From Natural-Language Task Descriptions to Vision-Grounded Reward Machines

    Nov 5, 2026Roko Parac, Christos Ziakas, Leo Ardon +1Reward MachinesReinforcement Learning

  3. Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training

    Sep 14, 2026Yuanhao Yue, Qianli Ma, Chengyu Wang +3Multimodal Large Language ModelsCurriculum RL