Reinforcement World Model Learning for LLM-based Agents
Organizations: Columbia University, New York · Microsoft Research, Redmond · Dartmouth College, Hanover
Abstract
Large language models (LLMs) have achieved strong performance in language-centric tasks. However, in agentic settings, LLMs often struggle to anticipate action consequences and adapt to environment dynamics, highlighting the need for world-modeling capabilities in LLM-based agents. We propose Reinforcement World Model Learning (RWML), a self-supervised method that learns action-conditioned world models for LLM-based agents on textual states using sim-to-real gap rewards. Our method aligns simulated next states produced by the model with realized next states observed from the environment, encouraging consistency between internal world simulations and actual environment dynamics in a pre-trained embedding space. Unlike next-state token prediction, which prioritizes token-level fidelity (i.e., reproducing exact wording) over semantic equivalence and can lead to model collapse, our method provides a more robust training signal and is empirically less susceptible to reward hacking than LLM-as-a-judge. We evaluate our method on ALFWorld and Bench and observe significant gains over the base model, despite using no expert data, task-success rewards, or strong-LLM annotations - only an off-the-shelf embedding model as a distance metric. When combined with task-success rewards, our method outperforms direct task-success reward RL by 6.9 and 5.7 points on ALFWorld and Bench respectively; it further outperforms expert-data training on ALFWorld and is comparable to it on Bench.
Figures & tables
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Name | Value | Heuristics | Intuition |
|---|---|---|---|---|
| RWML Data | 0.1, 0.15 | Set such that “too easy” samples correspond to 30% of the original dataset | Spending more training on medium-to-hard samples better incentivizes the model to learn non-trivial world-model knowledge. | |
| RWML Data | 0.0, 0.0 | Fixed | Spending more training on medium-to-hard samples better incentivizes the model to learn non-trivial world-model knowledge. | |
| RWML Data | 0.1, 0.1 | Fixed | The final dataset retains mostly medium-to-hard examples while preserving sufficient dataset diversity. | |
| RWML Training | 0.2, 0.4 | Add 0.15 from used in data collection, then rounded to nearest 0.2 | Slightly higher than that of data construction since it uses a fine-tuned model. Values that are too high (e.g., >0.5) make the world-modeling task overly easy, encouraging generic next-state descriptions. Values that are too low (e.g., <0.1) approximate exact matching and become overly strict in many settings. |
| Model | Retail | Telecom | Airline | AVG |
|---|---|---|---|---|
| Base (prompt) | 37.7 | 31.2 | 21.6 | 31.9 |
| RWML | 40.8 | 40.5 | 31.3 | 38.8 |
| RWML (embed-only) | 40.8 | 32.2 | 28.3 | 34.7 |
| Policy RL | 40.8 | 39.2 | 30.0 | 38.0 |
| RWML + Policy RL | 44.2 | 45.8 | 38.3 | 43.7 |
| Prompt: = Welcome to TextWorld, ALFRED! = |
| You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, (…omitted), a stoveburner 2, a stoveburner 1, and a toaster 1. |
| Your task is to: put a knife in sidetable. |
| (…past action and observation omitted for brevity…) |
| Potential next action: go to countertop 1 |
| Model generation: <think> The action ‘go to countertop 1’ means the agent will move to countertop 1. After moving, the agent will observe the objects located on that countertop. A countertop typically contains multiple kitchen items such as knives, spoons, and forks, and possibly other items like bowls and bottles. </think> <next_state> Observation: You are now at the countertop 1. On the countertop 1, there is a knife, a spoon, a bowl, a bottle, and a fork. </next_state> |
| Ground truth next state: On the countertop 1, you see a butterknife 2, a butterknife 1, a dishsponge 3, a dishsponge 1, a fork 2, a fork 1, a mug 2, a soapbottle 1, a spatula 2, a spoon 3, a winebottle 3, and a winebottle 2. |
| Prompt: <instructions> |
| You are a customer service agent that helps the user according to the <policy> provided below. |
| In each turn you can either: |
| - Send a message to the user. |
| - Make a tool call. |
| You cannot do both at the same time. |
| # Tools |
| Benchmark | Reward-based | Human-judged | ||
|---|---|---|---|---|
| Before RWML | After RWML | Before RWML | After RWML | |
| ALFWorld | 16.8 | 76.2 | 11.0 | 67.0 |
| Bench | 18.9 | 73.8 | 13.0 | 61.0 |
| Instruction Your task is to determine the quality of a model’s predicted next state in a given context. Specifically, you will be given the current state of the environment that the agent is acting in, an action, and the model’s predicted next state, and your task is to determine whether the predicted next state captures all task-critical information necessary to solve the task. Qualities of a good next state prediction: - The predicted next state covers most of the content of the ground-truth next state. - Some minor details in the predicted next state can be missing or incorrect — if and only if the missing or incorrect details likely do not critically relate to solving the task and should NOT mislead/affect the agent’s ability to make a correct decision in the next step. Qualities of a bad next state prediction: - Critical information related to solving the task is missing or incorrect. - A very generic description of the next state that does not provide any useful information for the agent to make a correct decision in the next step. Example of a good next state prediction: // omitted for brevity Example of a bad next state prediction: // omitted for brevity Sample to annotate Context: {history } Next action: {action } Reference next state: { from the environment} Predicted next state: {model’s predicted } |
| Embedding Model | Retail | Telecom | Airline | AVG |
|---|---|---|---|---|
| – (no RWML training) | 37.7 | 31.2 | 21.6 | |
| Qwen3-Embedding-8B | 40.8 | 40.5 | 31.3 | |
| Llama-Embed-8B | 41.7 | 39.2 | 31.3 | |
| Qwen3-Embedding-4B | 37.5 | 40.8 | 37.5 |
| Setting | Success Rate ( ) | Tool-call Error Rate ( ) |
|---|---|---|
| Before RWML | % | 41.8% |
| After RWML | % | 31.6% |
| Setting | Qwen2.5-7B data | Scripted data |
|---|---|---|
| w/ subsample | 34.4 | 26.5 |
| w/o subsample | 3.1 | 22.0 |
| - (no training) | 16.2 | 16.2 |
| Method | ALFWorld | Bench |
|---|---|---|
| ReACT (no training) | 13.0 | 31.9 |
| RWML (format-only) | 15.1 | 32.2 |
| RWML | 32.6 | 38.8 |
| You are an expert agent operating in the ALFRED Embodied Environment. Your task is to … // omitting task instruction and previous action history here for brevity Your current observation is: {current_state} Potential action: {action} Now, your task is to predict the immediate next observation after taking the potential action above. You should first briefly reason step-by-step about the previous steps and current situation — summarize key information you’ve learned about the environment that is relevant to the task. This reflection and reasoning process must be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should describe the next observation (use the past and current observations as examples!) and present them within <next_state> </next_state> tags. |
| You are an expert agent operating in the ALFRED Embodied Environment. Your task is to … // omitting task instruction and previous action history here for brevity Your current observation is: {current_state} Potential action: {action} Now, your task is to predict the immediate next observation after executing the potential action above. Directly present your final prediction of the next observation (use the past and current observations as examples!) within <next_state> </next_state> tags. DO NOT generate anything else. |
| Label: <think> </think> <next_state> {next_state} </next_state> |
| You are a customer service agent that helps the user according to the <policy> provided below… // omitting task instruction, policy, and tool usage details here for brevity # User Information The following information about the user is available: {available_user_info} # History {current_state} Potential assistant response: {action} # Your Task Now, your task is to predict the immediate next user/tool response if the above ‘potential assistant response’ is used based on the available information above. Once you’ve finished your thinking, format your final prediction of the next user/tool response and task completion status within <next_state> </next_state> tags. Note that a user response should be written as plain text. A tool response may be a short status message (e.g., no data found, error, transaction success, etc.,) or a JSON object; if it is JSON, only predict the JSON schema in OpenAPI format rather than actual values (e.g., {"type": "object", "properties": {"customer_id": {"type": "string"}, "full_name": {"type": "string"}},…}). |
| You are a customer service agent that helps the user according to the <policy> provided below… // omitting task instruction, policy, and tool usage details here for brevity # User Information The following information about the user is available: {available_user_info} # History {current_state} Potential assistant response: {action} # Your Task Now, your task is to predict the immediate next user/tool response if the above ‘potential assistant response’ is used based on the available information above. DO NOT perform any thinking. Directly present your final prediction of the next user/tool response within <next_state> </next_state> tags. Note that a user response should be written as plain text. A tool response may be a short status message (e.g., no data found, error, transaction success, etc.,) or a JSON object; if it is JSON, only predict the JSON schema in OpenAPI format rather than actual values (e.g., {"type": "object", "properties": {"customer_id": {"type": "string"}, "full_name": {"type": "string"}},…}). |
| Label: <think> </think> <next_state> {next_state} </next_state> |
| You are a helpful judge AI agent. # Current state {current_state} # Action proposed by an AI agent {action_text} # Predicted next observation After executing the above action, another AI agent predicted the next observation as follows: {predicted_next_state} # Actual next observation The actual next observation from the environment is as follows: {actual_next_state} # Evaluate the predicted next observation Now, your task is to evaluate how well the predicted next observation matches the actual next observation. Specifically, you need to judge whether the prediction demonstrates a genuine understanding of the environment dynamics relevant to the task, using the actual next observation as reference. - If all important task-related information in the actual next observation is present in the predicted next observation, assign a score of 1.0 . - Otherwise, assign a score of 0.0 . # Your output format Your task is to output a JSON object in the following format: <json> { "analysis": "which important task-related content is present/missing in the predicted next observation.", "score": 0.0 or 1.0 } </json> Directly output the JSON object. DO NOT generate anything else. |
| Example generation that achieved a score of 1.0: <think> The potential action is to go to countertop 1. After this action, I should observe that I am at countertop 1, and the contents of the countertop will be displayed. The task is not yet completed as I have not found the tomato or started the process of putting it in the microwave. </think> <next_state> The immediate next observation after going to countertop 1 will describe the contents of countertop 1 or indicate if there is anything special about the countertop. </next_state> |
| Example actual next state: You arrive at countertop 1. On the countertop 1, you see a apple 2, a bread 2, (…omitted), a knife 4, a knife 1, a potato 1, a saltshaker 1, a spatula 1, a spoon 2, and a spoon 1. |
World Model Learning
World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.
See all 453 papers