Large language models (LLMs) have achieved strong performance in language-centric tasks. However, in agentic settings, LLMs often struggle to anticipate action consequences and adapt to environment dynamics, highlighting the need for world-modeling capabilities in LLM-based agents. We propose Reinforcement World Model Learning (RWML), a self-supervised method that learns action-conditioned world models for LLM-based agents on textual states using sim-to-real gap rewards. Our method aligns simulated next states produced by the model with realized next states observed from the environment, encouraging consistency between internal world simulations and actual environment dynamics in a pre-trained embedding space. Unlike next-state token prediction, which prioritizes token-level fidelity (i.e., reproducing exact wording) over semantic equivalence and can lead to model collapse, our method provides a more robust training signal and is empirically less susceptible to reward hacking than LLM-as-a-judge. We evaluate our method on ALFWorld and τ2 Bench and observe significant gains over the base model, despite using no expert data, task-success rewards, or strong-LLM annotations - only an off-the-shelf embedding model as a distance metric. When combined with task-success rewards, our method outperforms direct task-success reward RL by 6.9 and 5.7 points on ALFWorld and τ2 Bench respectively; it further outperforms expert-data training on ALFWorld and is comparable to it on τ2 Bench.
Figures & tables
Figure 1: We propose RWML as a scalable, self-supervised method to improve the world modeling ability of LLM-based agent by learning from next-states, prior to downstream policy RL which learns from task-success reward.
Figure 2: Overview of RWML. Given a target model πθ , we first collect training data for RWML by using πθ to gather rollouts (s0,a0,s1,a1,...sT) with the environment, and then convert these rollouts into ⟨s≤t,at,st+1⟩ triplets for all t , after subsampling “too easy” samples defined in Equation 1 . We then train πθ to reason as a world model via GRPO, using lightweight reward functions (e.g., embedding-based cosine similarity) to compare the predicted s^t+1 with the real st+1 .
Figure 3: Comparing parameter change ratios per layer across models trained with different algorithms. We find WM SFT-trained models shows significantly more parameter change compare to RWML and Policy RL, potentially contributing to model forgetting in Section 3.3 .
Figure 4: RWML training with different base models on τ2 Bench.
Figure 5: After RWML, models produce more accurate and efficient decisions by leveraging its improved knowledge of the environment.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Name
Value
Heuristics
Intuition
RWML Data
τd
0.1, 0.15
Set such that “too easy” samples correspond to ∼ 30% of the original dataset
Spending more training on medium-to-hard samples better incentivizes the model to learn non-trivial world-model knowledge.
RWML Data
τeasy
0.0, 0.0
Fixed
Spending more training on medium-to-hard samples better incentivizes the model to learn non-trivial world-model knowledge.
RWML Data
p
0.1, 0.1
Fixed
The final dataset retains mostly medium-to-hard examples while preserving sufficient dataset diversity.
RWML Training
τd
0.2, 0.4
Add 0.15 from τd used in data collection, then rounded to nearest 0.2
Slightly higher than that of data construction since it uses a fine-tuned model. Values that are too high (e.g., >0.5) make the world-modeling task overly easy, encouraging generic next-state descriptions. Values that are too low (e.g., <0.1) approximate exact matching and become overly strict in many settings.
Appendix
Table A2 : Hyperparameters introduced in RWML. Value represents value used in ALFWorld, τ2 Bench, respectively.
Model
Retail
Telecom
Airline
AVG
Base (prompt)
37.7
31.2
21.6
31.9
RWML
40.8
40.5
31.3
38.8
RWML (embed-only)
40.8
32.2
28.3
34.7
Policy RL
40.8
39.2
30.0
38.0
RWML + Policy RL
44.2
45.8
38.3
43.7
Appendix
Table A3 : Ablating the ROUGE term in rWM on τ2 Bench (Qwen3-8B). “RWML (embed-only)” uses embedding-based reward for both user and tool-use responses. The embedding-only variant still significantly improves over the base model.
Prompt: = Welcome to TextWorld, ALFRED! =
You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, (…omitted), a stoveburner 2, a stoveburner 1, and a toaster 1.
Your task is to: put a knife in sidetable.
(…past action and observation omitted for brevity…)
Potential next action: go to countertop 1
Model generation: <think> The action ‘go to countertop 1’ means the agent will move to countertop 1. After moving, the agent will observe the objects located on that countertop. A countertop typically contains multiple kitchen items such as knives, spoons, and forks, and possibly other items like bowls and bottles. </think> <next_state> Observation: You are now at the countertop 1. On the countertop 1, there is a knife, a spoon, a bowl, a bottle, and a fork. </next_state>
Ground truth next state: On the countertop 1, you see a butterknife 2, a butterknife 1, a dishsponge 3, a dishsponge 1, a fork 2, a fork 1, a mug 2, a soapbottle 1, a spatula 2, a spoon 3, a winebottle 3, and a winebottle 2.
Appendix
Table A5 : Example output generated by the RWML-trained model on ALFWorld. The model first reasons about the action’s effect within <think></think> tags, then predicts the next state within <next_state></next_state> tags. Here, the model predicts a semantic description of the next state rather than surface-level object IDs. The outputs in the <next_state> tags and the ground truth are used to compute rWM(s^t+1,st+1) during RWML.
Prompt: <instructions>
You are a customer service agent that helps the user according to the <policy> provided below.
In each turn you can either:
- Send a message to the user.
- Make a tool call.
You cannot do both at the same time.
# Tools
Appendix
Table A6 : Example output generated by the RWML-trained model on τ2 Bench. The model first reasons about the user’s likely follow-up response within <think></think> tags, then predicts the next state within <next_state></next_state> tags. In this stochastic environment, the model reasons about a distribution of plausible user responses (highlighted in blue ) before predicting one likely next state. The outputs in the <next_state> tags and the ground truth are used to compute rWM(s^t+1,st+1) during RWML.
Benchmark
Reward-based
Human-judged
Before RWML
After RWML
Before RWML
After RWML
ALFWorld
16.8
76.2
11.0
67.0
τ2 Bench
18.9
73.8
13.0
61.0
Appendix
Table A7 : Next-state prediction accuracy (%) before and after RWML training. Reward-based : fraction of 512 unseen samples per benchmark that receive rWM=1 under the benchmark-specific reward function. Human-judged : fraction of 100 held-out samples per model per benchmark that blinded human annotators judge to capture all task-critical information (see Section F.2 ).
Instruction Your task is to determine the quality of a model’s predicted next state in a given context. Specifically, you will be given the current state of the environment that the agent is acting in, an action, and the model’s predicted next state, and your task is to determine whether the predicted next state captures all task-critical information necessary to solve the task. Qualities of a good next state prediction: - The predicted next state covers most of the content of the ground-truth next state. - Some minor details in the predicted next state can be missing or incorrect — if and only if the missing or incorrect details likely do not critically relate to solving the task and should NOT mislead/affect the agent’s ability to make a correct decision in the next step. Qualities of a bad next state prediction: - Critical information related to solving the task is missing or incorrect. - A very generic description of the next state that does not provide any useful information for the agent to make a correct decision in the next step. Example of a good next state prediction: // omitted for brevity Example of a bad next state prediction: // omitted for brevity Sample to annotate Context: {history s≤t } Next action: {action at } Reference next state: { st+1 from the environment} Predicted next state: {model’s predicted s^t+1 }
Appendix
Table A8 : Instructions given to human annotators for the blinded evaluation of predicted next states ( Section F.2 ). Each sample page additionally shows one example of a good and one example of a bad prediction, followed by the context, the reference next state, and the predicted next state. The identity of the model that produced each prediction is hidden from the annotators.
Embedding Model
Retail
Telecom
Airline
AVG
– (no RWML training)
37.7
31.2
21.6
31.9±2.9
Qwen3-Embedding-8B
40.8
40.5
31.3
38.8±2.5
Llama-Embed-8B
41.7
39.2
31.3
38.5±2.5
Qwen3-Embedding-4B
37.5
40.8
37.5
37.0±2.1
Appendix
Table A9 : RWML performance on τ2 Bench with different embedding models.
Setting
Success Rate ( ↑ )
Tool-call Error Rate ( ↓ )
Before RWML
3.1±0.0 %
41.8%
After RWML
5.6±1.3 %
31.6%
Appendix
Table A10 : Preliminary RWML results on Terminal-Bench using Qwen3-8B with Kimi-CLI.
Setting
Qwen2.5-7B data
Scripted data
w/ subsample
34.4
26.5
w/o subsample
3.1
22.0
- (no training)
16.2
16.2
Appendix
Table A11 : RWML ablation on ALFWorld with high-quality scripted rollouts vs. Qwen2.5-7B’s own rollouts. With clean data, removing subsampling no longer collapses performance.
Method
ALFWorld
τ2 Bench
ReACT (no training)
13.0
31.9
RWML (format-only)
15.1
32.2
RWML
32.6
38.8
Appendix
Table A12 : Isolation control for the next-state prediction objective. “RWML (format-only)” is trained on the same interaction data with the same training budget as RWML, but its reward only checks whether the output is well-formed, i.e., the next-state prediction reward is removed entirely. We use Qwen2.5-7B-Instruct on ALFWorld and Qwen3-8B on τ2 Bench.
Figure A1 : Parameter change analysis across datasets and training variants. Top: layer-wise weight change ratios by transformer layer. Bottom: main module-wise change ratios aggregated across layers.
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to … // omitting task instruction and previous action history here for brevity Your current observation is: {current_state} Potential action: {action} Now, your task is to predict the immediate next observation after taking the potential action above. You should first briefly reason step-by-step about the previous steps and current situation — summarize key information you’ve learned about the environment that is relevant to the task. This reflection and reasoning process must be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should describe the next observation (use the past and current observations as examples!) and present them within <next_state> </next_state> tags.
Appendix
Table A13 : Prompt for RWML in ALFWorld, given a current state st and an action at . Final output within the <next_state> </next_state> tags is used to compare with the next state in rWM(s^t+1,st+1) during RL.
You are an expert agent operating in the ALFRED Embodied Environment. Your task is to … // omitting task instruction and previous action history here for brevity Your current observation is: {current_state} Potential action: {action} Now, your task is to predict the immediate next observation after executing the potential action above. Directly present your final prediction of the next observation (use the past and current observations as examples!) within <next_state> </next_state> tags. DO NOT generate anything else.
Table A14 : Prompt for WM SFT in ALFWorld, given a current state st and an action at . The label is the next state st+1 . Since SFT requires target tokens and our transition data does not contain ground-truth reasoning traces, we train WM SFT with empty <think> </think> tags followed by the next-state label. We find this direct-prediction setup gives the best performance for both WM SFT and the subsequent Policy RL stage.
You are a customer service agent that helps the user according to the <policy> provided below… // omitting task instruction, policy, and tool usage details here for brevity # User Information The following information about the user is available: {available_user_info} # History {current_state} Potential assistant response: {action} # Your Task Now, your task is to predict the immediate next user/tool response if the above ‘potential assistant response’ is used based on the available information above. Once you’ve finished your thinking, format your final prediction of the next user/tool response and task completion status within <next_state> </next_state> tags. Note that a user response should be written as plain text. A tool response may be a short status message (e.g., no data found, error, transaction success, etc.,) or a JSON object; if it is JSON, only predict the JSON schema in OpenAPI format rather than actual values (e.g., {"type": "object", "properties": {"customer_id": {"type": "string"}, "full_name": {"type": "string"}},…}).
Appendix
Table A15 : Prompt for RWML in τ2 Bench, given s≤t and an action at . Final output within the <next_state> </next_state> tags is used to compare with the next state in rWM(s^t+1,st+1) during RL.
You are a customer service agent that helps the user according to the <policy> provided below… // omitting task instruction, policy, and tool usage details here for brevity # User Information The following information about the user is available: {available_user_info} # History {current_state} Potential assistant response: {action} # Your Task Now, your task is to predict the immediate next user/tool response if the above ‘potential assistant response’ is used based on the available information above. DO NOT perform any thinking. Directly present your final prediction of the next user/tool response within <next_state> </next_state> tags. Note that a user response should be written as plain text. A tool response may be a short status message (e.g., no data found, error, transaction success, etc.,) or a JSON object; if it is JSON, only predict the JSON schema in OpenAPI format rather than actual values (e.g., {"type": "object", "properties": {"customer_id": {"type": "string"}, "full_name": {"type": "string"}},…}).
Table A16 : Prompt for WM SFT in τ2 Bench, given s≤t and an action at . The label is the next state st+1 . Since SFT requires target tokens and our transition data does not contain ground-truth reasoning traces, we train WM SFT with empty <think> </think> tags followed by the next-state label. We find this direct-prediction setup gives the best performance for both WM SFT and the subsequent Policy RL stage.
You are a helpful judge AI agent. # Current state {current_state} # Action proposed by an AI agent {action_text} # Predicted next observation After executing the above action, another AI agent predicted the next observation as follows: {predicted_next_state} # Actual next observation The actual next observation from the environment is as follows: {actual_next_state} # Evaluate the predicted next observation Now, your task is to evaluate how well the predicted next observation matches the actual next observation. Specifically, you need to judge whether the prediction demonstrates a genuine understanding of the environment dynamics relevant to the task, using the actual next observation as reference. - If all important task-related information in the actual next observation is present in the predicted next observation, assign a score of 1.0 . - Otherwise, assign a score of 0.0 . # Your output format Your task is to output a JSON object in the following format: <json> { "analysis": "which important task-related content is present/missing in the predicted next observation.", "score": 0.0 or 1.0 } </json> Directly output the JSON object. DO NOT generate anything else.
Example generation that achieved a score of 1.0: <think> The potential action is to go to countertop 1. After this action, I should observe that I am at countertop 1, and the contents of the countertop will be displayed. The task is not yet completed as I have not found the tomato or started the process of putting it in the microwave. </think> <next_state> The immediate next observation after going to countertop 1 will describe the contents of countertop 1 or indicate if there is anything special about the countertop. </next_state>
Example actual next state: You arrive at countertop 1. On the countertop 1, you see a apple 2, a bread 2, (…omitted), a knife 4, a knife 1, a potato 1, a saltshaker 1, a spatula 1, a spoon 2, and a spoon 1.
Appendix
Table A17 : Example prompt for LLM-as-a-judge in our ablation studies. In general, we find LLM-as-a-judge is unreliable and can award high scores to predictions that are not meaningful. “predicted_next_state” are contents extracted from the <next_state></next_state> tags in the generated response.
Reinforcement learning (RL) improves large language model (LLM) agents by teaching them which actions lead to high rewards, but provides little supervision on what those actions do to the environment. World modeling (WM) can fill this gap, yet existing approaches often require separate simulators, extra training stages, or additional inference-time computation. We observe that on-policy RL rollouts already contain the needed signal: each transition pairs an action with its resulting next observation. Based on this observation, we propose PaW, a Policy and World modeling co-training framework that adds auxiliary WM supervision to the same policy during RL, without changing the inference paradigm. To make auxiliary WM supervision informative and stable, PaW introduces three components: action-entropy-based WM data selection, noise-tolerant WM loss, and reward-adaptive loss balancing. Experiments on three agentic task benchmarks show consistent improvements over strong RL baselines across models and RL algorithms. These results suggest that standard RL rollouts are a practical source of WM supervision for language-agent training.
Ning Lu, Baijiong Lin, Shengcai Liu +9
Southern University of Science and Technology · Hong Kong University of Science and Technology · Hong Kong University of Science and Technology (Guangzhou) +2
How do LLM agents come to both understand environments they act in and master tasks set within them? Through controlled experiments combining world-model training (next-state prediction) and policy training (reward maximization), we investigate this question. We dissect the resulting models through their additive parameter updates. Geometrically, we find effective world-model updates are low-rank and share an input-feature subspace with policy updates while writing to nearly orthogonal output directions, whether trained separately or sequentially. However, we find that, in projection interventions, the sequential update induces more robustness than separate policy RL when removing the world model's leading input directions, suggesting that it has learned alternative input pathways. Behaviorally, we find the sequentially trained agent explores a wider range of states and actions. Based on this, we ask: does policy training preserve world knowledge as well as it could? We probe this with training-free merging built on the geometrically motivated input basis plus an online world-model loss during policy RL, and show both improve over the untreated baseline. Our findings suggest world knowledge and task-directed ability can be learned in geometrically complementary forms, and that future post-training pipelines should consider how best to engineer the interface between them.
A world model predicts environment dynamics based on current observations and actions, serving as a core cognitive mechanism for reasoning and planning. In this work, we investigate how world modeling based on language models can further push the boundaries of general agents. (i) We first focus on building foundation models for agentic environment simulation. We introduce Qwen-AgentWorld-35B-A3B and Qwen-AgentWorld-397B-A17B, the first language world models capable of simulating agentic environments covering 7 domains via long chain-of-thought reasoning. Leveraging more than 10M environment interaction trajectories of 7 domains in real-world environments, we develop Qwen-AgentWorld through a three-stage training pipeline: CPT injects general-purpose world modeling capabilities from the state transition dynamics and augmented professional corpora, SFT activates next-state-prediction reasoning, and RL sharpens simulation fidelity through a tailored framework with hybrid rubric-and-rule rewards. To evaluate language world models, we present AgentWorldBench, a comprehensive benchmark constructed from real-world interactions of 5 frontier models on 9 established benchmarks. Empirical results demonstrate that Qwen-AgentWorld significantly outperforms existing frontier models. (ii) Beyond foundation models, we further investigate two complementary paradigms through which world modeling enhances general agents. First, as a decoupled environment simulator, Qwen-AgentWorld supports scalable and controllable simulation of thousands of real-world environments for agentic RL, yielding gains that surpass real-environment training alone. Second, as a unified agent foundation model, world-model training acts as a highly effective warm-up that improves downstream performance across 7 agentic benchmarks. Code: https://github.com/QwenLM/Qwen-AgentWorld