Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

Darshan Deshpande

cs.AIMay 7, 2026

Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

Authors: Darshan Deshpande

Organizations: Patronus AI

Abstract

Recent growth in reinforcement learning (RL) has surfaced a need for diverse, specialized training environments. Hand-curated environments with fixed task and reward difficulties become ineffective signals as model performance improves, and sparse rewards over long horizons induce mode collapse on specific workflows or tool structures. World models that simulate environment states have matched pure rollout performance, making them promising for scaling diversity on-demand. However, autoregressive (AR) world models suffer from a left-to-right bias preventing conditioning on globally interdependent state anchors such as tool schemas, prior turns, and expected outcomes. We (i) formalize text-based world modeling as a steerable transition-dynamics problem decomposed into initial state, task context, tool schemas, domain rules, and steering directives, and (ii) curate 239,403 grounded state-action trajectories spanning nine open-source environments and twelve frontier model families. We compare AR LMs and masked diffusion language models (MDLMs), showing MDLMs, via bidirectional anchor-aware denoising, achieve better coherence, groundedness, and empirically validated rollout diversity than LLMs over 4x their parameter size, at comparable inference latency. We introduce a plug-and-play GRPO training framework with deterministic state checks, and perform zero-shot transfer ablations on three OOD environments (ScienceWorld, ALFWorld, AppWorld) across three 1.2B-7B agent backbones (LFM2.5, Qwen3, Mistral), achieving up to 47% absolute gains over baselines without environment-specific fine-tuning. We further conduct behavioral analysis of failure modes under adversarial scenarios and human evaluation on realism, outcome correctness, and training utility. We open-source our work to encourage research in this direction.

Figures & tables

Figure 1: Bidirectional attention conditions generation on the fixed arrival date. Iterative denoising reveals the depot first, then uses it to complete the earlier service and fee fields.

Baseline Models
	In-domain			API-Bank			OccuBench			Intercode (SQL)
Model	B-1	R-L	MAUVE	B-1	R-L	MAUVE	B-1	R-L	MAUVE	B-1	R-L	MAUVE
Qwen-3-8B	.136	.286	.148	.453	.421	.301	.524	.523	.705	.406	.397	.699
Qwen-3.5-27B	.152	.221	.187	.536	.611	.505	.746	.755	.810	.516	.614	.799
Qwen-3.5-35B-A3B	.125	.233	.189	.488	.585	.499	.667	.680	.811	.522	.598	.780
Nemotron-3-Nano-30B	.120	.152	.223	.412	.316	.510	.675	.667	.822	.486	.465	.785

Table 1: Generation quality of MDLMs and AR models on in-domain and out-of-domain test sets. The best zero-shot result in each column is bolded , and the best three-shot result is bolded in red . For 3-shot prompting, only the best AR and MDLM baselines are shown. Full 3-shot results are in Table 7 , and results for finetuned non-Transformer architectures are in Table 8 .

Figure 2: Task success rates across training methods on AppWorld, SciWorld, and ALFWorld for three policy models. Qwen-WM denotes Qwen3.5-27B , while SDAR-WM denotes SDAR-8B . Values above the bars report absolute success rates. Environments marked with ∗ are made partially observable to evaluate long-context world-model behavior.

Model	Self-BLEU ( $\downarrow$ )	Distinct-N ( $\uparrow$ )	MAUVE ( $\uparrow$ )
Qwen3.5-35B-A3B	.690	.253	.889
Qwen3.5-27B	.670	.321	.864
GPT-OSS-20B	.659	.228	.867
SDAR-8B	.601	.385	.902

Table 2: Aggregated Self-BLEU, Distinct-N (N=

\{1,2,3,4\}

) and MAUVE scores for fine tuned world models across test splits.

Metric	Mean	Krippendorff’s $\alpha$
Realism	4.75	.932
Outcome Correctness	4.25	.891
Training Utility	4.50	.901

Table 3: Human evaluation results across three metrics. Mean is computed over annotator ratings (1-5 Likert scale) and Krippendorff’s

\alpha

reports inter-annotator agreement. Definitions for metrics are in Appendix A

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Score	Label	Description
Training utility
5	Excellent	Correct and consistent outcome, realistic values, i.e. ideal training signal
4	Good	Minor value errors, but correct outcome type
3	Mediocre	Correct outcome type, but wrong schema or implausible values
2	Poor	Wrong outcome type or severely hallucinated structure
1	Useless	Completely wrong, nonsensical, or would teach incorrect behaviours, including incomplete outputs

Appendix

Table 4: Annotation rubrics for the three evaluation dimensions.

Figure 3: Training split counts and token length distributions per dataset. The sampled number of data points are proportional to openly available training tasks.

Configuration	BLEU-1	ROUGE-L	MAUVE	Latency ( $N=4567$ )
Qwen3.5-35B-A3B (no thinking)	.621	.664	.899	211s ( $1.00\times$ )
+ Thinking ( $\textsc{Effort}=5\text{k}$ )	.713	.768	.940	793s ( $3.75\times$ )
+ Thinking ( $\textsc{Effort}=10\text{k}$ )	.710	.758	.934	1028s ( $4.87\times$ )
SDAR-8B (no thinking)	.749	.740	.902	380s ( $1.80\times$ )
WeDLM-8B (no thinking)	.729	.761	.910	145s ( $0.68\times$ )

Appendix

Table 5: Out-of-domain evaluation of Qwen3.5-35B-A3B world model with and without thinking on OccuBench , API-Bank , and InterCode test splits. Reported latency is for 4xH200 GPUs with LLM baselines using vLLM + FlashInfer, SDAR using the author recommended JetEngine and WeDLM using the official, vLLM-modified inference code.

Failure Mode	Env	Trajectory Snippet
(i) Incorrect pagination	AW	Agent requests page 1 of recommendations; WM returns the same records as page 0. Downstream, the agent double-counts songs and computes incorrect aggregates. T4: show_recommendations({page_index:0}) $\to$ {total:162, songs:[{id:40},{id:41},…]} T5: show_recommendations({page_index:1}) $\to$ {total:162, songs:[{id:40},{id:41},…]} $\leftarrow$ same IDs
(ii) Object hallucination	SW	WM invents code-execution errors and objects absent from the environment state. Agent receives fabricated feedback that does not correspond to any valid ScienceWorld response. Agent: open cupboard WM: ‘‘Error during execution: name ‘cupboard_is_opened’ is not defined’’ $\leftarrow$ fabricated [2pt] Agent: activate solar panel WM: ‘‘Solar panels cannot be used directly.’’ $\leftarrow$ not a ScienceWorld response
(iii) JSON-wrapped responses	SW	WM outputs structured JSON instead of plain-text environment observations. Downstream parser receives role/content wrappers that corrupt the conversation history. Agent: look around WM: [{‘‘role’’:‘‘tool’’, ‘‘content’’:‘‘This room is called the kitchen. In it, you see: the agent, a substance called air, a chair…’’}] $\leftarrow$ JSON wrapper
(iv) Dropped instance numbers	ALF	WM substitutes generic class names for specific receptacle instances, breaking downstream state tracking and action validation. Agent: go to toaster 1 WM: ‘‘You arrive at toaster 1. On the table , you see a bread 2.’’ Expected: ‘‘On the toaster 1 , you see a bread 2.’’ [2pt] Agent: go to microwave 1 WM: ‘‘You arrive at microwave 1. On the table , nothing…’’ Expected: ‘‘On the microwave 1 , you see nothing.’’
(v) State inconsistency	SW	WM confirms picking up an object that is already in inventory, creating phantom duplicates. Agent “picks up” the thermometer three times across 17 turns. T6: pick up thermometer $\to$ ‘‘You move the thermometer to the inventory.’’ T15: pick up thermometer $\to$ ‘‘You move the thermometer to the inventory.’’ $\leftarrow$ already held T17: pick up thermometer $\to$ ‘‘You move the thermometer to the inventory.’’ $\leftarrow$ third time
(vi) Verbose / garbled output	ALF, SW	WM generates excessively long descriptions with hallucinated items, or garbles object names by dropping instance numbers from object listings. ALFWorld: Agent: go to fridge 1 WM: ‘‘…On the fridge, you see a 2 , and a mug.’’ (object class dropped) [2pt] ScienceWorld (Qwen WM): Agent: look around (greenhouse) WM: ‘‘…a decorative stone gazing ball resting atop one corner… a rustic lantern holding dried lavender bundles… finally tucked away quietly within deepest recesses thereof lies forgotten treasure trove comprising ancient maps …’’ (886 hallucinated tokens)

Appendix

Table 6: World model failure modes observed during GRPO training. Red highlights the erroneous portion. AW = AppWorld, ALF = ALFWorld, SW = ScienceWorld. These failures were addressed via grounding rules in the WM prompt, local deterministic responders, and JSON unwrapping in the training plugin.

3-shot Baseline Models
	In-domain			API-Bank			OccuBench			Intercode (SQL)
Model	B-1	R-L	MAUVE	B-1	R-L	MAUVE	B-1	R-L	MAUVE	B-1	R-L	MAUVE
Qwen-3-8B	.379	.506	.348	.578	.509	.365	.559	.544	.788	.532	.523	.833
Qwen-3.5-27B	.509	.578	.588	.611	.660	.600	.731	.772	.834	.585	.623	.849
Qwen-3.5-35B-A3B	.334	.507	.590	.621	.663	.634	.744	.781	.832	.589	.662	.851
GLM-4.7-Flash	.401	.565	.629	.686	.689	.707	.746	.742	.838	.612	.597	.822

Appendix

Table 7: Full three-shot results for all baseline and finetuned models on in-domain and out-of-domain test sets. Best three-shot baseline result in each column is bolded ; best finetuned three-shot result is bold highlighted in red . The best-performing 3-shot AR and MDLM baselines (GLM-4.7-Flash and SDAR-30B-A3B) are also reported in Table 1 .

Model	Architecture	B-1	R-L	MAUVE
RWKV-7B-g1	RWKV (linear attention)	.334	.496	.651
Falcon-H1R-7B	Mamba-attention hybrid	.541	.661	.807
Qwen-3-8B	Transformer (AR)	.636	.682	.873
SDAR-8B	Transformer (MDLM)	.749	.740	.902

Appendix

Table 8: Generation quality of finetuned non-Transformer sequence models compared to the AR and MDLM baselines closest in parameter count. Scores are averaged over the three out-of-domain test sets (API-Bank, OccuBench, Intercode-SQL); each model is finetuned with its official training harness and we report the best of three hyperparameter configurations (Appendix D.4). The Qwen-3-8B and SDAR-8B rows are the corresponding OOD averages of the finetuned models in Table 1 . Best result in each column is bolded .

$k$	Qwen3.5-27B (AR)	SDAR-8B (forced)	SDAR-8B (self-committed)
1	.86	.90	.89
2	.86	.91	.90
8	.86	.93	.89
16	.86	.94	.91

Appendix

Table 9: Synthetic anchor benchmark (250 OOD samples).

k

: number of future anchors fixed in the target state.

Failure mode	WM prediction	Expected
(vii) API-key corruption	GetUserToken $\rightarrow$ "p9o8i7u6y5t…tttepmw"	"p9o8i7u6y5t4r3e2"
(viii) Type coercion	SetVolume $\rightarrow$ {"level": "7"}	{"level": 7} (schema type int )
(ix) Web-search fabrication	“Team SoloMid won the 2013 NA LCS Summer Finals …”	“Cloud9 won the 2013 NA LCS Summer playoffs …”
(x) Error cascade	After one earlier error, every retry of move_to_node(…) $\rightarrow$ TimeoutError	First attempt succeeds

Appendix

Table 10: Additional failure modes flagged by annotators during the human evaluation. This table complements Table 6.

Explore similar work

Qwen-AgentWorld: Language World Models for General Agents
Jun 23, 2026Yuxin Zuo, Zikai Xiao, Li Sheng +30LLM World Models Agentic RL
Qwen Team
Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents
Jun 4, 2026Kaixuan Liu, Guojun Xiong, Weinan Zhang +1World Model Learning Off-Policy Evaluation
Department of Computer Science Emory University Atlanta, GA, USA · School of Computer Science Shanghai Jiao Tong University Shanghai, China
Policy and World Modeling Co-Training for Language Agents
Jun 1, 2026Ning Lu, Baijiong Lin, Shengcai Liu +9Agentic RL World Model Learning
Southern University of Science and Technology · Hong Kong University of Science and Technology · Hong Kong University of Science and Technology (Guangzhou) +2

Jun 23, 2026, cs.CLJ/K move · Enter open · S save

Qwen-AgentWorld: Language World Models for General Agents

Yuxin Zuo, Zikai Xiao, Li Sheng, Fei Huang+29

Qwen Team

LLM World Models Agentic RL World Model Learning World Model Evaluation

Open paper AskPDF arXiv

A world model predicts environment dynamics based on current observations and actions, serving as a core cognitive mechanism for reasoning and planning. In this work, we investigate how world modeling based on language models can further push the boundaries of general agents. (i) We first focus on building foundation models for agentic environment simulation. We introduce Qwen-AgentWorld-35B-A3B and Qwen-AgentWorld-397B-A17B, the first language world models capable of simulating agentic environments covering 7 domains via long chain-of-thought reasoning. Leveraging more than 10M environment interaction trajectories of 7 domains in real-world environments, we develop Qwen-AgentWorld through a three-stage training pipeline: CPT injects general-purpose world modeling capabilities from the state transition dynamics and augmented professional corpora, SFT activates next-state-prediction reasoning, and RL sharpens simulation fidelity through a tailored framework with hybrid rubric-and-rule rewards. To evaluate language world models, we present AgentWorldBench, a comprehensive benchmark constructed from real-world interactions of 5 frontier models on 9 established benchmarks. Empirical results demonstrate that Qwen-AgentWorld significantly outperforms existing frontier models. (ii) Beyond foundation models, we further investigate two complementary paradigms through which world modeling enhances general agents. First, as a decoupled environment simulator, Qwen-AgentWorld supports scalable and controllable simulation of thousands of real-world environments for agentic RL, yielding gains that surpass real-environment training alone. Second, as a unified agent foundation model, world-model training acts as a highly effective warm-up that improves downstream performance across 7 agentic benchmarks. Code: https://github.com/QwenLM/Qwen-AgentWorld

Abstract

Explore similar work

Qwen-AgentWorld: Language World Models for General Agents

Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents

Policy and World Modeling Co-Training for Language Agents