SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents

Wenyi Yu; Siyin Wang; Terumi Chiba; Xianzhao Chen; Xiaohai Tian; Jun Zhang; Lu Lu; Chao Zhang

cs.CLSep 28, 2026

SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents

Authors: Wenyi Yu, Siyin Wang, Terumi Chiba, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Chao Zhang

Organizations: Tsinghua University · ByteDance

Abstract

Full-duplex speech large language models (LLMs) enable low-latency, natural voice interaction. However, real-world agents must also use tools and perform deliberative reasoning-operations whose variable latency and computational cost conflict with the stringent timing requirements of real-time conversation. To reconcile these demands, we propose SALMONN-duo, an adaptive dual-system voice agent inspired by dual-process theories of cognition. SALMONN-duo separates real-time interaction from deliberative computation by pairing an always-on, fast-thinking full-duplex speech LLM (system 1) with a powerful asynchronous slow-thinking LLM agent (system 2). Beyond handling real-time interaction, system 1 learns when to answer directly and when to delegate, remaining responsive during backend execution and seamlessly integrating returned information into the ongoing dialogue without exposing tool traces or losing conversational context. Evaluations on single-turn spoken question answering (QA) and multi-turn conversations demonstrate that adaptive delegation substantially improves accuracy on knowledge-intensive and multi-hop reasoning questions, while knowledge-boundary-aware training avoids unnecessary system 2 invocations. On a customized version of $τ$ -Voice, SALMONN-duo further demonstrates its ability to complete environment-grounded, policy-constrained tasks through multi-turn interactions in realistic business scenarios. Finally, cost-aware reinforcement learning further enhances the trade-off between task performance and backend usage across the QA and conversation tasks, while improving task success and response safety on $τ$ -Voice with an acceptable increase in the delegation rate.

Figures & tables

Figure 1: Illustration of SALMONN-duo completing a user task through adaptive tool use, real-time handling of complex conversational dynamics (e.g., user barge-in), and seamless delivery of backend results to the user.

Figure 2: SALMONN-duo comprises two collaborative systems: System 1 interacts with the user in a full-duplex manner, adaptively invokes an asynchronous system 2, and naturally incorporates the information provided by system 2 into the ongoing conversation to fulfill the user’s goals.

Routing	TriviaQA			HotpotQA			HaluEval
Structure	%Acc.	AUROC	%ARA.	%Acc.	AUROC	%ARA.	%Acc.	AUROC	%ARA.
delegation head ℓ=16	56.9	0.661	98.1	17.7	0.639	93.1	20.4	0.737	93.8
delegation head ℓ=24	55.5	0.677	97.8	16.6	0.661	92.8	19.3	0.691	92.0
delegation head ℓ=32	54.5	0.684	96.6	13.6	0.619	93.5	15.5	0.741	90.8
delegation token	68.5	0.745	98.6	23.6	0.691	94.5	31.1	0.732	93.6

Table 1: Performance comparison of different delegation mechanisms on spoken QA. Accuracy (%Acc.) measures answer correctness without system 2, while Answer-Reference Agreement Rate (%ARA.) measures the agreement with system 2 references under forced invocation.

Model (Base LM Size)	LlamaQ.		WebQ.		TriviaQA		HotpotQA		HaluEval
Model (Base LM Size)	%Acc.	%Del.	%Acc.	%Del.	%Acc.	%Del.	%Acc.	%Del.	%Acc.	%Del.
GLM-4-Voice (9B)	64.7		32.2		39.1		12.6		21.2
Kimi-Audio (7B)	79.3		70.2		62.1		28.9		40.9
Step-Audio-2-mini (8B)	73.0		53.1		39.6		15.5		14.7
Qwen2.5-Omni (7B)	79.3		62.2		58.1		24.3		24.3
Qwen3-Omni-A3B-Ins. (30B)	82.3		69.3		72.2		30.8		33.5

Table 2: Spoken QA performance of half-duplex and full-duplex speech LLMs and full-duplex voice agents. Accuracy (%Acc.) measures answer correctness when the model can decide whether to request a reference, if supported; Delegation Rate (%Del.

\downarrow

) measures the proportion of cases in which the model requests a reference. Underlined numbers are values reported in the original paper of each model. The performance of GPT-4o references used by voice agents is shown in gray .

Model (Base LM Size)	%Acc.				%Del.
Model (Base LM Size)	turn 3	turn 4	turn 5	Avg.	%Del.
Qwen2.5-Omni (7B)	35.6	36.3	33.5	35.2
Qwen3-Omni-A3B-Ins. (30B)	46.0	42.7	45.0	44.5
MoshiRAG (7B)	55.5	51.7	56.0	54.3	95.6
SALMONN-duo (8B)	59.2	52.9	56.5	56.2	61.9

Table 3: Performance on multi-turn daily conversation. Accuracy (%Acc.) and Delegation Rate (%Del.

\downarrow

) are reported.

Model	TriviaQA		HotpotQA		HaluEval		MultiTurn
Model	AUROC	%ARA.	AUROC	%ARA.	AUROC	%ARA.	AUROC	%ARA.
SFT	0.745	98.6	0.691	94.5	0.732	93.6	0.730	93.6
GRPO ${}_{r_{\text{del, cor}}=0.65}$	0.744	99.3	0.715	95.8	0.722	93.6	0.738	94.1
GRPO ${}_{r_{\text{del, cor}}=0.80}$	0.757	99.1	0.714	95.2	0.748	94.1	0.754	95.3
GRPO ${}_{r_{\text{del, cor}}=0.90}$	0.749	99.1	0.721	95.8	0.726	94.3	0.734	94.9

Table 4: Impact of cost-aware GRPO on single-turn and multi-turn knowledge-based QA performance. Answer-Reference Agreement Rate (%ARA.) measures the agreement with system 2 references under forced invocation. We compare different values of

r_{\text{del, cor}}

while fixing

r_{\text{dir, cor}}=1

and

r_{\text{dir, incor}}=r_{\text{del, incor}}=0

Figure 3: The impact of cost-aware GRPO on accuracy–delegation trade-offs of SALMONN-duo on single- and multi-turn knowledge-based question answering tasks.

Model	Pass@1				Safety		Cost
Model	Airline	Retail	Telecom	Overall	Hal. $\downarrow$	Pol. $\uparrow$	%Del. $\downarrow$	%Effective Del. $\uparrow$
GPT-realtime-1.5	55.0	72.5	60.0	64.0	0.55	0.68
SFT	30.0	72.5	55.0	57.0	0.83	0.56	48.1	87.9
SFT-always S2	60.0	67.5	65.0	65.0	0.78	0.68	100.0	70.6
GRPO for pass@1	35.0	75.0	55.0	59.0	0.77	0.61	57.6	82.9
GRPO $+r_{\text{safety}}$	35.0	75.0	65.0	63.0	0.62	0.75	71.8	78.7

Table 5: Performance on the customized

\tau

-Voice. Safety scores are episode-level metrics that incorporate both the hallucination rate (Hal.

\downarrow

) in model responses and the rate of compliance with domain-specific policies (Pol.

\uparrow

). For cost, we report the proportion of model response turns that invoke system 2 (%Del.

\downarrow

). Among these turns, we define those in which system 2 actually uses tools as effective delegations and report the corresponding effective delegation rate (%Effective Del.

\uparrow

Model	%Acc.
Model	Order	Correctness
Reference		76.8
SFT	81.9	77.2
cost-aware GRPO	87.8	74.7

Table 6: Performance on handling user interruptions, including adherence to the expected response order and accuracy on interrupting questions. Results obtained using single-turn QA are shown in gray for reference.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Knowledge boundary aware multi-turn conversation generation pipeline.

Requested order	Result arrival	Expected order
Original-first	Before	Original-first
Original-first	After	New-first
New-first	Before	New-first
New-first	After	New-first

Appendix

Table 7: Expected response order under user interruptions. Result arrival indicates whether system 2 returns its result before or after the user finishes the interrupting question.

Routing	LlamaQ.			WebQ.			MultiTurn
Structure	%Acc.	AUROC	%ARA.	%Acc.	AUROC	%ARA.	%Acc.	AUROC	%ARA.
delegation head ℓ=16	74.0	0.709	96.7	56.7	0.696	93.9	22.2	0.709	91.9
delegation head ℓ=24	74.3	0.701	97.0	57.7	0.683	92.0	23.0	0.704	91.6
delegation head ℓ=32	71.0	0.666	96.7	54.3	0.691	93.6	19.8	0.727	90.9
delegation token	77.0	0.764	98.3	65.2	0.718	93.6	35.0	0.730	93.6

Appendix

Table 8: Performance comparison of different delegation mechanisms on Llama Questions, Web Questions and MultiTurn. Accuracy (%Acc.) measures answer correctness without system 2, while Answer-Reference Agreement Rate (%ARA.) measures the agreement with system 2 references under forced invocation.

Model	LlamaQ.		WebQ.
Model	AUROC	%ARA.	AUROC	%ARA.
SFT	0.764	98.3	0.718	93.6
GRPO ${}_{r_{\text{del, cor}}=0.65}$	0.763	97.3	0.702	92.4
GRPO ${}_{r_{\text{del, cor}}=0.80}$	0.776	98.0	0.719	93.8
GRPO ${}_{r_{\text{del, cor}}=0.90}$	0.799	97.0	0.715	93.9

Appendix

Table 9: Impact of cost-aware GRPO on the performance of Llama Questions and Web Questions. Answer-Reference Agreement Rate (%ARA.) measures the agreement with system 2 references under forced invocation. We compare different values of

r_{\text{del, cor}}

while fixing

r_{\text{dir, cor}}=1

and

r_{\text{dir, incor}}=r_{\text{del, incor}}=0

Model	LlamaQ.	WebQ.	TriviaQA	HotpotQA	HaluEval
MoshiRAG δ=1.5s	82.7	73.2	85.4	53.9	47.3
MoshiRAG δ=2.0s	74.1	70.7	78.0	48.2	42.6
MoshiRAG δ=2.5s	62.0	64.8	63.9	38.4	33.9
VoiceChat δ=1.5s	68.3	45.0	38.7	10.7	10.3
VoiceChat δ=2.0s	68.3	45.0	38.8	10.9	10.0
VoiceChat δ=2.5s	68.0	45.0	38.3	9.7	9.4

Appendix

Table 10: %Accuracy on spoken QA under varying system 2 invocation latencies

\delta

User text	LlamaQ.		WebQ.		TriviaQA		HotpotQA		HaluEval
User text	ref.	ans.	ref.	ans.	ref.	ans.	ref.	ans.	ref.	ans.
Ground-truth	88.0	79.3	80.8	71.6	93.2	80.4	63.2	46.0	53.6	43.9
ASR	87.0	79.3	76.7	71.4	91.3	80.3	54.2	40.3	57.7	47.7
ASR Word Error Rate (%)	0.71		2.85		3.30		4.47		10.59

Appendix

Table 11: Impact of ground-truth versus ASR-transcribed user text on backend-generated references (ref.) and frontend-generated final answers (ans.). ASR word error rates are also reported.

Figure 5: SALMONN-duo can use the Codex backend to launch tasks that gather up-to-date information and compile it into documents.

Figure 6: SALMONN-duo can use the Codex backend to review and modify code.

Explore similar work

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengFull-Duplex Speech Models Speech Language Models
Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Sep 18, 2026Jagadeesh Balam, Travis Bartley, Edresson Casanova +52Full-Duplex Speech Models Tool Invocation
Wookie
AdaptDuplex: from static to adaptive full-duplex spoken dialogue
Sep 24, 2026Zhiyang Zhou, Yingxin Shang, Zhou Wang +9Full-Duplex Speech Models Dialogue
Zuoyebang Education Technology

Jun 12, 2026, cs.CLJ/K move · Enter open · S save

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

Qingkai Fang, Shoutao Guo, Yang Feng

Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · Key Laboratory of AI Safety, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Full-Duplex Speech Models Speech Language Models Autoregressive Language Models Chatbots

Open paper AskPDF arXiv

Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.

Abstract

Explore similar work

BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities

AdaptDuplex: from static to adaptive full-duplex spoken dialogue