Dyad: Extending Large Language Models with Native Typed Decision-Making

Yundaichuan Zhan; Weishi Wang; Wenbiao Liu; Daniel Dahlmeier; Chengwei Qin; Juncheng Li; Fredrik D. Johansson; Zhongqi Yue

cs.LGSep 28, 2026

Dyad: Extending Large Language Models with Native Typed Decision-Making

Authors: Yundaichuan Zhan, Weishi Wang, Wenbiao Liu, Daniel Dahlmeier, Chengwei Qin, Juncheng Li, Fredrik D. Johansson, Zhongqi Yue

Organizations: Zhejiang University · SAP · Central South University · The Hong Kong University of Science and Technology (Guangzhou) ·

Abstract

We study how to build more capable general-purpose agents by extending large language models (LLMs) with native typed decision-making. We introduce Dyad, an architecture that augments a pretrained LLM with an environment-conditioned action encoder that embeds each candidate action description in parallel, then scores these embeddings against the LLM's internal state to yield a distribution over typed actions. By factorizing decision-making into representations of the evolving interaction state and environment-specific action semantics, Dyad introduces an inductive bias for learning reusable representations while keeping action scoring efficient even as the action space grows. We investigate two complementary reinforcement learning settings driven by environment interaction. With the LLM frozen, training the action encoder alone achieves consistent gains across four unseen environments, enabling modular adaptation without modifying any LLM parameters. Jointly optimizing both components outperforms conventional RL post-training across diverse interactive tasks and model scales, including a 3.80% average absolute gain on ALFWorld with a 9B model, while improving general knowledge, reasoning, and coding.

Figures & tables

Figure 1: Conceptual comparison of System 1, System 2, and Dyad for reasoning and acting.

Figure 2: Dyad architecture. State and environment-conditioned action representations are computed independently. Generating the interaction token $v_{\mathrm{act}}$ triggers typed action selection; otherwise, the pretrained LM head generates language tokens.

Approach	Logit for $a\in\mathcal{A}_{t}$	Action interface
Joint action-set encoder	$F_{a}(e,h_{t},\mathcal{A}_{t})$	Joint scoring
AR encoder	$F(e,h_{t},a)$	Independent scoring
AR Single-token	$\mathbf{w}_{\operatorname{id}(a)}^{\top}F(e,h_{t},\mathcal{A}_{t})$	Fixed token identifiers
Dyad	$f(e,h_{t})^{\top}g(e,a)$	Compositional scoring

Table 1: Conceptual comparison of methods for typed decision-making.

F

denotes a model jointly processing its specified inputs, and

\operatorname{id}(a)

maps an available action to a vocabulary identifier.

Method	ALFWorld	WebShop
Base model	45.66	33.43
+ GRPO	85.01	73.44
+ Dyad-GRPO	90.90	75.17
+ GiGPO	89.32	81.52
+ Dyad-GiGPO	91.02	83.65

Table 2: In-domain performance of Qwen3.5-9B. Evaluations use 3 seeds.

	DIVE				CodeGym
Method	ALF	WebShop	$\tau^{2}$	SWE	ALF	WebShop	$\tau^{2}$	SWE
Qwen3.5-4B	23.45	28.54	61.05	43.21	23.45	28.54	61.05	43.21
+ GRPO	29.71	30.42	60.52	45.67	32.18	33.65	64.10	47.98
+ Dyad-GRPO	31.52 +1.81	31.22 +0.80	64.54 +4.02	44.28 −1.39	33.65 +1.47	35.71 +2.06	65.26 +1.16	45.02 −2.96
+ GiGPO	28.22	29.54	61.84	45.02	31.09	27.97	63.48	46.64
+ Dyad-GiGPO	33.07 +4.85	32.45 +2.91	61.86 +0.02	47.10 +2.08	31.72 +0.63	31.21 +3.24	64.78 +1.30	49.61 +2.97

Table 3: Cross-environment performance. Agents are evaluated without further training. Superscripts show absolute changes (%) from the corresponding RL baseline. ALF denotes ALFWorld. Evaluations use 3 seeds.

Figure 3: General capabilities. Dashed lines indicate zero change relative to each model without additional training. Evaluations are on 3 seeds.

Figure 4: Training dynamics. Curves show validation means for Qwen3.5-9B on CodeGym; shading indicates

\pm 1

standard deviation.

Method	ALFWorld	WebShop	$\tau^{2}$	SWE
Qwen3.5-4B	23.45	28.54	61.05	43.21
+ Dyad w/ freezing	27.75 $+4.30$	31.47 $+2.93$	63.88 $+2.83$	43.78 $+0.57$
+ Dyad w/o freezing	31.52	31.22	64.54	44.28
Qwen3.5-9B	45.66	33.43	62.17	44.56
+ Dyad w/ freezing	48.82 $+3.16$	35.00 $+1.57$	64.23 $+2.06$	45.50 $+0.94$
+ Dyad w/o freezing	53.82	36.31	64.88	46.20

Table 4: Frozen-LLM adaptation. Dyad is trained with GRPO on DIVE. Evaluations use 3 seeds.

Method	Easy	Standard	Hard	Overall	Latency	Output tokens
Jev 1.13	100.00	96.79	70.88	85.01	$0.67^{\dagger}$	—
Qwen3.5-4B (w/o reasoning)	100.00	96.41	56.32	77.89	0.32	—
Qwen3.5-4B (w/ reasoning)	98.76	95.62	59.88	79.10	1.02	102.82
Qwen3.5-4B + Dyad (w/o reasoning)	100.00	97.88	67.07	83.52	0.31	—
Qwen3.5-4B + Dyad (w/ reasoning)	100.00	98.72	72.08	86.18	1.35	120.44

Table 5: JevBench results. Accuracy (%) on the public subset. Best accuracy and latency (s) values are bold.

\dagger

Jev latency comes from published API measurements. Benchmark and measurement details are provided in Appendix B.3 .

Figure 5: Comparison with Jev. Qwen3.5-4B serves as the zero-shot reference on ALFWorld.

Figure 6: Ablation studies. CodeGym validation reward with Qwen3.5-4B, GRPO, and attention pooling. Both panels reuse the Dyad (full) curve; the horizontal dashed line denotes the alignment-only reference.

Projector	Validation reward
Mean pooling	41.23
MLP-weighted pooling	48.20
Attention pooling	55.92

Table 6: Action pooling ablation.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	ALFWorld		WebShop
Input format	Length	Success rate	Length	Success rate
MCP-style (default)	2665.57	33.88	396.50	34.26
Natural language	132.64	31.01	151.50	29.87

Appendix

Table 7: Action encoder input formats. Qwen3.5-4B with Dyad-GRPO is trained on CodeGym and evaluated on ALFWorld and WebShop. Length denotes the mean number of description tokens.

		Records
Split	Cases	MCP-style	Natural language	Total
Train	1,050	1,050	1,050	2,100
Validation	140	140	140	280
Test	140	140	140	280
Total	1,330	1,330	1,330	2,660

Appendix

Table 8: Alignment dataset. Each case has paired MCP-style and natural-language records.

Environment	# Tasks	Avg. tools / actions
DIVE	1,499	40.87
CodeGym	80,130	6.48
ALFWorld	3,827	14
WebShop	11,687	2
$\tau^{2}$ -bench	278	15.41
SWE	500	1

Appendix

Table 9: Environment statistics. Environments cover training and transfer evaluation.

Interaction budgets
Environment	ALFWorld	WebShop	DIVE	CodeGym
Max prompt length (tokens)	2,048	4,096	32,768	5,120
Max interaction steps	50	15	30	80
Memory context window (steps)	2	2	2	2
Max response length (tokens/turn)	512	512	512	386
Sampling and batching

Appendix

Table 10: Agentic RL hyperparameters. Settings correspond to joint post-training.

Panel of Figure 7 — Figure 7: Prompt template for DIVE. It specifies how the agent calls tools and submits its final answer.

Panel of Figure 8 — Figure 8: Prompt template for CodeGym. The agent issues one function call at a time and waits for feedback before continuing.

Panel of Figure 9 — Figure 9: Prompt template for ALFWorld. The agent selects an admissible action based on the current observation and interaction history.

Panel of Figure 10 — Figure 10: Prompt template for WebShop. The agent uses the shopping instruction and current observation to select its next action.

Panel of Figure 11 — Figure 11: Prompt template for $\tau^{2}$ -bench. The agent follows the domain policy when using tools and communicating with the customer.

Panel of Figure 12 — Figure 12: Prompt template for SWE-bench Verified. GiGPO-style context with mini-swe-agent’s Bash interface.

	ALFWorld			WebShop
Model / Method	MMLU-Pro	HMMT	LiveCodeBench	MMLU-Pro	HMMT	LiveCodeBench
Qwen3.5-4B	69.34	36.26	47.39	69.34	36.26	47.39
+ GRPO	68.08	30.33	46.21	67.93	33.30	46.62
+ Dyad-GRPO	68.96	33.36	47.02	69.02	34.76	47.11
+ GiGPO	67.62	32.46	46.78	67.51	33.05	46.06
+ Dyad-GiGPO	68.10	36.33	47.52	69.53	35.16	47.89

Appendix

Table 11: General capability retention. Models are post-trained on ALFWorld or WebShop.

Figure 13: Additional training dynamics on CodeGym. Validation reward, response length, and environment interactions for Qwen3.5-4B (top) and Qwen3.5-27B (bottom).

Figure 14: Training dynamics on DIVE. Models are jointly post-trained on DIVE.

Language Model-Based Control

Language model-based control uses language models to interpret goals, choose actions, and steer AI systems or physical processes.

See all 172 papers

PriorZero: Bridging Language Priors and World Models for Decision Making
May 12, 2026Junyu Xiong, Yuan Pu, Jia Tang +1LLM-Guided RL Latent World Models
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Nanjing University of Aeronautics and Astronautics +1
Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making
Jul 15, 2026Amirhosein Ghasemabadi, Ruichen Chen, Bahador Rashidi +1Multi-Agent LLM Systems Language Model-Based Control
1ECE Department, University of Alberta · 2Huawei Technologies Canada Co., Ltd.
Training Advisors for LLM Agents from Task Outcomes
Oct 7, 2026Sergei Polezhaev, Barys Liskavets, Ori Press +1LLM Agent Training RL for LLM Agents
Nebius AI

May 12, 2026J/K move · Enter open · S save

PriorZero: Bridging Language Priors and World Models for Decision Making

Junyu Xiong, Yuan Pu, Jia Tang, Yazhe Niu

University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Nanjing University of Aeronautics and Astronautics+1

LLM-Guided RL Latent World Models RL Fine-Tuning Model-Based RL

Open paper AskPDF arXiv

Leveraging the rich world knowledge of Large Language Models (LLMs) to enhance Reinforcement Learning (RL) agents offers a promising path toward general intelligence. However, a fundamental prior-dynamics mismatch hinders existing approaches: static LLM knowledge cannot directly adapt to the complex transition dynamics of long-horizon tasks. Using LLM priors as fixed policies limits exploration diversity, as the prior is blind to environment-specific dynamics; while end-to-end fine-tuning suffers from optimization instability and credit assignment issues. To bridge this gap, we propose PriorZero, a unified framework that integrates LLM-derived conceptual priors into world-model-based planning through a decoupled rollout-training design. During rollout, a novel root-prior injection mechanism incorporates LLM priors exclusively at the root node of Monte Carlo Tree Search (MCTS), focusing search on semantically promising actions while preserving the world model's deep lookahead capability. During training, PriorZero decouples world-model learning from LLM adaptation: the world model is continuously refined on interaction data to jointly improve its dynamics, policy, and value predictions, its value estimates are then leveraged to provide fine-grained credit assignment signals for stable LLM fine-tuning via alternating optimization. Experiments across diverse benchmarks, including text-based adventure games in Jericho and instruction-following gridworld tasks in BabyAI, demonstrate that PriorZero consistently improves both exploration efficiency and asymptotic performance, establishing a promising framework for LLM-empowered decision-making. Our code is available at https://github.com/opendilab/LightZero.

Abstract

Language Model-Based Control

Explore similar work

PriorZero: Bridging Language Priors and World Models for Decision Making

Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making

Training Advisors for LLM Agents from Task Outcomes