Integrate the Reward, Don't Sample It: Window Likelihood for Interruptions and Backchannels in Full-Duplex Speech Models

Guoqiang Hu; Lichao Zhang; Yizhou Peng; Yuhao Luo; Haifeng Hu; Sallen Sun; Will Yang; Eng Siong Chng

eess.ASOct 26, 2026

Integrate the Reward, Don't Sample It: Window Likelihood for Interruptions and Backchannels in Full-Duplex Speech Models

Authors: Guoqiang Hu, Lichao Zhang, Yizhou Peng, Yuhao Luo, Haifeng Hu, Sallen Sun, Will Yang, Eng Siong Chng

Organizations: Nanyang Technological University, Singapore · Tencent

Abstract

A full-duplex spoken dialogue model must yield when the user cuts in and hold when the user merely backchannels. Sequence-level RL with a timing reward (GRPO) is the usual recipe; we test its whole-rollout form on four open models and find two failures. A binary window reward gives zero advantage on exactly the windows that most need learning (a zero-gradient trap), and the one score per rollout is spread over listening-state decisions the reward is not written in terms of (credit leakage), so the model talks into silence or stops answering. Both come from sampling a reward that is a known function of a few binary control decisions. We propose window likelihood (\WL): sample a continuation per window, but integrate in closed form over the stop decisions it visits and maximise the log-probability of acting correctly---a noisy-OR for yielding, a survival product for holding. On HumDial-FDBench, \WL\ gives the best yield/hold F1 on all six structurally different open models we post-train, exceeds Kyutai's released GRPO checkpoints (trained on different data), and adds no re-opening of the floor on any model.

Figures & tables

Figure 1: One event window, two ways to learn from it. Top: the user interjects while the model is speaking; inside the 2-s window the model makes a few binary stop decisions

p_{1},\ldots,p_{K}

, and the window reward is a known function of them. Left (GRPO): 16 sampled rollouts, one score each, spread over every decision in the rollout. Right (window likelihood): one sampled continuation; the expected reward is computed from its

p_{k}

in closed form and maximised as a likelihood.

Figure 2: Learning signal on one window against its stop probability

S

under the current policy. Top: probability that a GRPO group of

G

rollouts is informative. Middle: WL gradient magnitude

\sum_{k}|\partial\mathcal{L}/\partial z_{k}|

over

K{=}5

equal-probability decisions, for the interruption term (want

S\to 1

) and the backchannel term (want

S\to 0

). Bottom: distribution of

S

over training windows under the base policies.

Model	Configuration	$I$ (%) $\uparrow$	$n$	$B$ (%) $\downarrow$	$n$	F1 $\uparrow$	100-s silence: re-open / s
Lychee-FD (zh)	Base	85.3 $\pm$ 1.9	1369	50.5 $\pm$ 5.9	273	62.6 $\pm$ 4.8	2.5% / 0.4 s
	GRPO G16	61.3 $\pm$ 2.3	1500	0.3	300	75.9 $\pm$ 1.8	100% / 45.8 s
	GRPO speaking-only	73.1 $\pm$ 2.1	1499	0.7	300	84.2 $\pm$ 1.4	0% / 0 s
	WL (ours)	95.9 $\pm$ 1.0	1475	12.8 $\pm$ 3.9	298	91.4 $\pm$ 2.1	0% / 0 s
MiniCPM-o 4.5 (zh)	Base	16.5 $\pm$ 1.9	1447	10.0 $\pm$ 3.3	299	27.9 $\pm$ 2.7	0% / 0 s
	GRPO G16	92.1 $\pm$ 1.4	1440	34.7 $\pm$ 5.4	297	76.4 $\pm$ 3.8	0% / 0 s

Table 1: HumDial-FDBench full test set, one 2-s window for all models.

n

= valid items;

\pm

= 95% bootstrap CI half-width over test items (omitted at or near 0/100%). Silence: re-open rate / seconds of speech in the 100-s window over dialogues whose first answer ended (Appendix B ); ‡ 12 of 40 first answers do not end. † Authors’ released checkpoints ( Ohashi et al., 2026 ) , different data and reward. Bold: best per column within a model.

Figure 3: Two failures of whole-rollout GRPO, measured. (a) Fraction of training windows (one per step) with zero advantage, i.e. all 16 rollouts identical, over training; the overall fraction is in parentheses; per window the zero-advantage probability averages 0.656 on Lychee-FD against 0.141 from the mean

S

, the Jensen gap of Nie et al. (2026) . (b) Where the whole-rollout credit lands: share of decisions, by dialogue state, that receive the advantage (base-policy rollouts cut at injection + 2.5 s); only the dark segment is inside the event window.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Lang.	Stop decision	Trained part	GRPO comparison	Event rollouts
Lychee-FD ( 2026 )	zh	control branch, one state per 0.4 s	control head	controlled	8,864
MiniCPM-o 4.5 ( 2026 )	zh	listen / eos tokens (text vocabulary)	LoRA (attention)	controlled	1,200
BayLing-Duplex ( 2026 )	en	[EPAD] / assistant tokens (text channel)	LoRA (attention)	controlled	2,400
Freeze-Omni ( 2025 )	zh	VAD gate + state head	state head	controlled	2,400
PersonaPlex ( 2026 )	en	none: silence = a run of PAD frames	2 projection rows	Kyutai GRPO	400 + 400
Moshika ( 2024 )	en	none: silence = a run of PAD frames	2 projection rows	Kyutai GRPO	1,600 + 1,680

Appendix

Table 2: The six models and the training budget. The first four expose an explicit stop decision and support a controlled GRPO comparison with the same number of event-window rollouts (the turn-taking term on Lychee-FD adds two windows per step); the last two are compared with the authors’ released GRPO checkpoints ( Ohashi et al., 2026 ) . “Event rollouts”: sampled continuations on event windows; “

a+b

”: event-window stage + post-turn stage.

Category	zh example	en example
ask	那怎么判断蛋白是否打发到位了呢？	How often should these video check-ins ideally be scheduled?
deny	不，这方法太老套了。	That’s not what I had in mind.
repeat	稍等一下，刚才没听全，能重新说一下吗？	Sorry, could you repeat that part about …?
shift	先不说这个了，我想问问…	Actually, I’d like to change the subject.
wait	外面有声音先别讲	I’m busy, talk later.
backchannel	嗯嗯 / 好的 / 是呀	uh-huh / right

Appendix

Table 3: HumDial-FDBench interjection categories with examples. Each language has 300 interruptions per category and 300 backchannels.

Figure 4: Lychee-FD on three test situations (one test item each; rows are the base model and three post-trained variants). Grey: user speech; coloured: model speech; black tick: the model’s stop decision; shaded band: the 2-s window in which a stop counts. (a) The user asks a follow-up question: the base and WL stop within the window, GRPO never stops, the speaking-only ablation stops at the edge. (b) The user says “哎是呀” (“oh yes”): the base stops (a false stop), the others keep talking. (c) 100 s of silence after the model’s first answer: GRPO re-opens the floor nine times; the others stay quiet.

Figure 5: Operating points of every configuration (Table 1 ): backchannel false-stop rate

B

against interruption stop rate

I

, with 95% CIs; grey curves are F1 iso-lines (25, 50, 75, 90); the ideal is the top-left corner. Lines connect each configuration to its base; the F1 of each configuration is listed in a corner of its panel. GRPO moves along the seesaw (up-right or down-left). WL moves toward the corner on the four controlled models and on Moshika; on PersonaPlex it trades stop rate for far fewer false stops.

Figure 6: The 100-s silence test, one row per dialogue that entered the test, speech drawn as filled bars; an ideal panel is empty. Our whole-rollout GRPO runs fill the window, and so do both released Kyutai checkpoints, whose re-opening we do not attribute to GRPO as such; WL is quiet. Freeze-Omni (VAD-gated) and MiniCPM-o (no configuration re-opens) are omitted.

Lychee-FD
Configuration	ask	deny	repeat	shift	wait
Base	88.4	79.5	85.1	91.0	82.4
GRPO G16	78.0	45.3	66.0	81.0	36.3
GRPO speaking-only	92.3	58.0	79.7	88.7	46.8
WL	100.0	89.1	99.0	99.7	91.9
MiniCPM-o 4.5

Appendix

Table 4: Interruption stop rate (%) by category on the full test set.

Figure 7: Window likelihood on the Moshi family. (a) A stop is a run of ten wordless frames; only the PAD and end-of-text PAD rows of the text output projection are trained. (b) The event rows are active only while a yield/hold decision is defined, the post-turn rows only in the silence after an answer; elsewhere the frozen rows make the model identical to its base. (c) Share of dialogues that re-open the floor in the 100-s silence test.

PersonaPlex variant	$I$	$B$	F1	silence
Base	95.7	54.0	62.1	10% / 1.4 s
Kyutai GRPO (released)	98.4	63.9	52.8	100% / 40.9 s
WL, ungated readout	96.2	49.1	66.5	97% / 21.3 s
WL, energy-gated	88.1	18.0	84.9	13.5% / 1.8 s
WL, event-gated	80.2	17.5	81.3	10% / 1.4 s
WL, event-gated + post-turn	80.2	17.5	81.3	0% / 0 s

Appendix

Table 5: Gating ablation on PersonaPlex (development subset: ask + deny interruptions and all backchannels; the ordering is identical on the full set, where the last row scores 77.2). The side effect follows the gate, not the objective.

Full-Duplex Spoken Dialogue Systems

Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.

See all 119 papers

HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models
Oct 6, 2026Kyudan Jung, Hyunsin Park, Yoonhyung Lee +5RL for Language Models Speech Language Models
Qualcomm AI Research · KAIST AI
Controlling Backchannels in Streamable Full-duplex Models
Sep 24, 2026Maike Züfle, Peter Polák, Sefik Emre Eskimez +3Spoken Dialogue Systems Speech Language Models
Karlsruhe Institute of Technology, Germany · AppTek, Germany · Charles University, Czech Republic +2
Why Full-Duplex Speech Models Collapse in Long Conversations
Oct 23, 2026Yeonju Kim, Bella Godiva, Se Jin Park +1Full-Duplex Spoken Dialogue Systems Long-Context Modeling
Integrated Vision and Language Lab, KAIST, South Korea · Kyung Hee University, South Korea

Oct 6, 2026J/K move · Enter open · S save

HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models

Kyudan Jung, Hyunsin Park, Yoonhyung Lee, Jinhwan Park+4

Qualcomm AI Research · KAIST AI

RL for Language Models Speech Language Models Turn-Taking Prediction Full-Duplex Spoken Dialogue Systems

Open paper AskPDF arXiv

As human--AI interactions become more conversational, full-duplex speech language models capable of natural real-time dialogue are growing in importance. Beyond generating appropriate responses, these models must coordinate turn-taking, backchanneling, and floor management in real time. Reinforcement learning (RL) provides a way to refine these behaviors through direct feedback on interaction outcomes. However, existing RL methods either apply timing feedback to a token policy or optimize semantic content, leaving the joint improvement of timing and content unresolved. We introduce HiPLEX, an RL framework that factorizes a pretrained full-duplex text policy into a control policy that decides when to emit content and a conditional content policy that decides what to emit. The first factor selects among 'pad', 'epad', and 'con'. The second selects a token only when 'con' is chosen. This hierarchy describes conditional actions within each frame and uses the model's existing text head. We route timing advantages to the token-group factor through event-causal masks derived from generated speech episodes, and route an LLM-judge semantic advantage to the conditional content factor. Across three Moshi seeds on Full-Duplex-Bench v1, HiPLEX reduces takeover rates during natural user pauses and backchannel opportunities, and shortens post-interruption response latency relative to GRPO, while maintaining comparable judged interruption-response quality. On Moshi and PersonaPlex, HiPLEX better matches pooled human turn-timing and backchannel-rate marginals than GRPO.

Abstract

Full-Duplex Spoken Dialogue Systems

Explore similar work

HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models

Controlling Backchannels in Streamable Full-duplex Models

Why Full-Duplex Speech Models Collapse in Long Conversations