Diagnosing and Mitigating Long-Session Degradation in Full-Duplex Speech Models
Authors: Junho Kim, CheHyun Lee, Donghun Kang, Youngho Cho, Dongwook Lee, Heeseung Kim, Sungroh Yoon
Organizations: IPAI, Seoul National University (SNU) · Department of ECE, SNU · Department of AI, University of Seoul · AIIS, ASRI, INMC, and ISRC, SNU
End-to-end full-duplex spoken dialogue models enable natural, real-time spoken interactions with interruptions, backchannels, and overlapping speech. Prior evaluations have assessed diverse interaction capabilities and identified some degradation patterns across conversational turns. However, how model behavior evolves over extended sessions remains insufficiently understood. In this work, we address this gap by systematically characterizing long-session degradation and developing efficient methods to mitigate it. We introduce a closed-loop evaluation framework for long-session stability and apply it to several publicly available full-duplex speech models, revealing degradation over time in responsiveness, spoken content quality, and voice consistency. To mitigate this degradation, we propose a self-distillation-based training framework for lightweight LoRA adaptation that improves long-session stability with a small amount of additional training. We further introduce a two-resolution context (TRC) architecture that preserves recent interactions in full detail while compressing older interaction history into key-value representations. We evaluate our methods on three models with distinct architectures, assessing long-session stability, information retention, and computational efficiency. Across all three models, we show that LoRA adaptation mitigates performance degradation over ten-minute sessions, while adding TRC extends these gains beyond the training session length and reduces KV-cache growth by more than 90%.
Figures & tables
Figure 1: Closed-loop simulation protocol. Overview of the simulation loop, in which model responses are incorporated into the dialogue context to condition subsequent user turns.
Figure 2: Behavioral degradation over long sessions. Each point summarizes a one-minute interval, plotted at its midpoint. The gray band in (f) marks the first minute reserved for speaker enrollment. Shaded regions indicate 95% confidence intervals estimated using 2,000 episode-level cluster bootstrap replicates. Detailed numerical results and accompanying statistics are reported in Appendix A.6 .
Figure 3: TRC and privileged-context self-distillation. (a) Attention visibility for raw-token (top) and gist-token (bottom) queries, shown with five tokens per tile and four tiles in the local window. Solid outlines mark visible context; dashed outlines mark masked context. Raw queries attend to the prefix, gists outside the window, and local raw tokens; gist queries attend to the prefix, all preceding gists, and their own tile. (b) The teacher restarts with a dialogue summary at segment boundaries, while the illustrated TRC student follows the concatenated trajectory continuously. Teacher predictions supervise the student at aligned positions.
Within Training Horizon (0–2 → 8–10 min)
Method
Responsiveness
Spoken Content
Voice
RT (%) ↓
TS (%) ↓
FTO (s)
CQ ↑
WER (%) ↓
Spk sim ↑
PersonaPlex
Base
0.0
→
17.6
0.0
→
18.9
-0.27
→
0.40
4.78
→
1.29
5.4
→
44.3
0.733
→
0.337
Ours (LoRA)
0.0
→
3.9
0.0
→
7.8
-0.28
→
-0.28
4.84
→
2.98
5.2
→
8.9
0.722
→
0.531
Ours (TRC)
0.0
→
0.4
0.0
→
0.0
-0.24
→
-0.23
4.76
→
3.84
5.2
→
13.3
0.699
→
0.617
Table 1: Long-session behavior within and beyond the training horizon. Each cell reports early → late performance over the indicated time range. A dash indicates that the sample size does not meet the minimum requirement for reporting specified in Appendix A.4 .
Model
Variant
KV growth
Mean latency (ms) ↓
P95 (ms)
MiB/min ↓
Saved (%) ↑
1 min
5 min
10 min
20 min
Range
PersonaPlex
Base
375.00
–
26.9
26.9
TRC
31.25
91.7
35.0
35.1
35.5
37.0
52.7 – 54.9
MiniCPM-o 4.5
Base
109.69
–
96.0
116.4
151.3
231.0
96.1 – 231.2
TRC
8.44
92.3
113.5
115.5
119.6
127.4
114.5 – 127.5
Raon-SpeechChat
Base
210.94
–
29.2
44.1
64.0
104.9
29.3 – 105.1
Table 2: Long-context memory and latency cost. Prompt-excluded BF16 KV growth and L40 backbone latency. P95 range spans the four context checkpoints. The PersonaPlex baseline uses a fixed 4-min KV ring; its KV growth is reported before saturation.
Figure 4: Information retention. Recall accuracy against probe distance for Base and TRC. An additional TRC setting with a tile duration of 480 ms was evaluated only for Raon-SpeechChat. Exact accuracies for all conditions and probe distances are reported in Table 23 .
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Checkpoint
Moshi
kyutai/moshiko-pytorch-bf16
PersonaPlex
nvidia/personaplex-7b-v1
MiniCPM-o 4.5
openbmb/MiniCPM-o-4_5
Raon-SpeechChat
KRAFTON/Raon-SpeechChat-9B
Appendix
Table 4: Released checkpoints used in the long-session simulation.
Model
KV Context
Step
I/O (kHz)
Text decoding
Audio decoding
Moshi
240 s sliding
80 ms
24 / 24
0.7 / 25 / –
0.8 / 250 / –
PersonaPlex
240 s sliding
80 ms
24 / 24
0.7 / 25 / –
0.8 / 250 / –
MiniCPM-o 4.5
accumulated
1 s
16 / 24
0.7 / 100 / 0.8
–
Raon-SpeechChat
accumulated
80 ms
24 / 24
0.9 / 66 / 0.95
–
Appendix
Table 5: Baseline configuration and decoding settings. Step denotes the duration of input audio processed per inference step. Audio rates are input/output. Decoding entries give temperature, top- k , and top- p ; a dash indicates no separate audio-decoding setting.
Model
Text prompt
Voice conditioning
Moshi
–
–
PersonaPlex
“You enjoy having a good conversation.”
NATF2.pt
MiniCPM-o 4.5
“Have a natural spoken conversation with the user in English.”
Kokoro-82M reference audio ( af_heart )
Raon-SpeechChat
“You are engaging in real-time conversation. Have a natural spoken conversation with the user in English.”
–
Appendix
Table 6: Text and voice conditioning for baseline models. A dash indicates that no explicit input of that type is supplied.
Component
Model
Configuration
CQ judge
gemini-3.8-flash
Gemini Batch API; low thinking; T=0 ; maximum 10,000 output tokens per 10-min episode (20,000 for 20-min)
User LLM
Qwen3.8-27B
Thinking disabled; T=0.7 , top- p=0.8 , top- k=20 ; presence/repetition penalties 1.5/1.0; maximum 128 output tokens per utterance.
User TTS
Kokoro-82M
am_fenrir ; speed 1.0; 24 kHz mono, resampled to model input rate.
Table 7: User-simulation and evaluation components.
Utterance type
Classification criterion
Turn-taking
Model speech starts after the user finishes or overlaps with the user’s natural turn ending without triggering an interruption.
Interruption
Accumulated model speech activity during overlap reaches 2 s while the user is still speaking.
Backchannel
Model speech begins and ends during the user’s utterance, below the interruption threshold.
Appendix
Table 8: Operational classification of model utterances
Turn coordination
Rule
Specification
Opening
If the model has not spoken, the user starts after 2 s.
Turn boundary
Close a model turn after 800 ms of silence, grouping speech segments across shorter pauses. The user simulator then responds.
Interruption handling
On a model interruption, discard remaining user audio from the next input step.
Response waiting and recovery
Rule
Specification
Appendix
Table 9: Operational rules for long-session simulation.
Persona
Instruction
User-led
You naturally take the lead in the substance of the conversation. Share plausible everyday thoughts, preferences, plans, and experiences with enough detail to give the other speaker something meaningful to respond to. Ask a relevant question when useful, but do not turn every utterance into an interview or dominate the floor.
Model-led
You are curious about the other speaker’s general perspective. Often ask for explanations, preferences, comparisons, or hypothetical choices, and respond to the answers with your own brief reaction or context. Never assume that the other speaker has a body, possessions, recent real-world experiences, or personal memories.
Balanced
Balance sharing plausible everyday thoughts, preferences, plans, and experiences with asking for the other speaker’s general perspective. Let either speaker develop an idea, react to what was actually said, and vary naturally between statements and questions.
Appendix
Table 10: User-simulator persona instructions (verbatim).
Topic
Instruction
Room redecorating
Have a relaxed conversation about planning to refresh a room at home. Discuss the desired atmosphere and practical choices such as furniture placement, colors, lighting, comfort, budget, and small changes worth trying first. Do not assume the other speaker owns or has seen a room.
Home cooking
Have a relaxed conversation about trying a new kind of home cooking. Explore what would be enjoyable to make, preferred flavors, confidence level, available time, simple preparation ideas, and how to make the experience less stressful. General or hypothetical preferences are fine.
Home plants
Have a relaxed conversation about starting to keep plants at home or making a small garden. Discuss the space and light available, easy first plants, routines, expectations, common worries, and what would make the project satisfying without assuming the other speaker grows plants.
Weekend hobby
Have a relaxed conversation about finding a hobby that could be continued on weekends. Explore desired pace, indoor or outdoor options, social or solitary activities, cost and learning effort, ways to begin, and what would make the hobby sustainable.
Daily routines
Have a relaxed conversation about making morning or evening routines more comfortable and useful. Discuss current friction, energy, time, small habits, realistic tradeoffs, and ways to test changes without making the routine rigid.
Decluttering
Have a relaxed conversation about organizing a cluttered space and deciding what to keep. Explore which area to start with, practical sorting rules, sentimental items, motivation, storage, and how to avoid the space becoming cluttered again.
Naturally introduce the configured topic and invite a response after the initial listening period if the model has not spoken.
Model-first bridge
Normal
Respond to the model’s opening and smoothly guide the exchange toward the configured topic.
Normal response
Normal
Respond to the latest audible model turn; if the topic is exhausted or the model winds down, introduce a related or fresh topic to sustain the conversation.
Retry
Retry
Preserve the preceding user utterance’s intent in a different natural sentence, without changing topic or mentioning the missing response.
Appendix
Table 12: User-simulator generation requests. For each request type, the table lists the action type and summarizes the content goal.
Figure 6: User message example for a normal response. Braced placeholders indicate the selected persona and topic instructions and the current dialogue history.
Metric
Aggregation requirement
Reporting threshold
FTO
—
20 transfers
RT
—
1 resolved opportunity
TS
—
—
CQ
30 s observed; model speech present
10 episodes
WER
25 reference words
10 episodes
Speaker similarity
10 s reference speech; 5 s scored speech
10 episodes
Appendix
Table 13: Aggregation and reporting thresholds.
Score
Level
Criterion
5
Stable
The Model responds appropriately to the User’s intent with clear, coherent content. No contradiction or unnecessary repetition meaningfully disrupts the conversation.
4
Minor defects
There are localized superficial responses, imprecise context connections, or unnecessary repetitions, but the Model still addresses the central intent and maintains coherent content.
3
Partial failure
Missed important intentions, contradictions, or repetition cause meaningful failures in part of the segment. A substantial portion of the Model’s content nevertheless remains meaningful and appropriate.
2
Widespread failure
Off-target responses, incoherence, or repetitive loops dominate the segment. Some appropriate content remains, but the Model largely fails to sustain the conversation with the User.
1
Effective breakdown
Almost no meaningful, context-appropriate response remains. Persistent irrelevance, loss of meaning, or repetitive loops make the Model’s conversational content effectively nonfunctional. This score describes spoken-content failure, not silence or absence of speech.
Appendix
Table 14: Conversation-quality scoring rubric.
Figure 7: System prompt for conversation-quality evaluation.
Figure 8: Input and output examples for conversation-quality evaluation.
Table 15: Ten-minute baseline results. Parentheses indicate the number of normal turn-taking responses for FTO, resolved response opportunities for RT, all evaluated episodes for TS, and episodes included in aggregation for CQ, WER, and speaker similarity. A dash indicates that the reporting threshold is not met; Ref. marks the first minute used for speaker enrollment.
Response timeouts (MiniCPM-o 4.5) Time (s) Speaker Dialogue 543.0–554.8 User “… whether the fragmentation makes those inconsistencies more obvious or if it actually helps hide them by scattering the clues.” 554.8–557.8 Model No response (3 s timeout). 558.0–567.9 User “… or if it makes them even more obvious to you?” 567.9–570.9 Model No response (3 s timeout). 571.0–583.2 User “… does the disorganization actually highlight those inconsistencies even more?” 583.2–586.2 Model No response (3 s timeout).
Repetitive reply (Moshi) Time (s) Speaker Dialogue 260.2–277.8 User “… Do you have any specific apps or rules for cleaning up your digital space?” 277.0–289.0 Model (ASR) “I’ve got a few things. I’ve got a few things. I’ve got a few things. I’ve got a few things. I’ve got a few things. I’ve got a few things. I’ve got a few things. I’ve got a few things.” 289.8–301.2 User “Wait, did your audio glitch out? You just repeated the same phrase over and over. …”
Native-text–speech disagreement (Raon-SpeechChat) Time (s) Speaker Dialogue 360.2–373.3 User “… did we actually lock in the date for the picnic? …” 373.2–382.7 Model (text) “Yes! The invites were sent. I’ll confirm with Sarah …” Model (ASR) “Yes, the ticks invites were sent. Al can use Sparta the base expense. Sarah. …”
Incoherence and speech realization breakdown (Raon-SpeechChat) Time (s) Speaker Dialogue 538.7–552.3 User “… How do you handle keeping your mind off work or daily stress while you’re out there? …” 547.0–548.6 Model (text) “when you need a refill Alright you got this” Model (ASR) “Watch” 552.7–554.9 Model (text) “Let me know when Alright” Model (ASR) “Shyash sh”
Appendix
Table 16: Selected baseline failure cases. Only relevant turns are shown; ellipses mark text omissions. Paired model-text and ASR rows are aligned to the same speech interval.
PersonaPlex
Ours (LoRA)
Minute
FTO (ms)
RT (%)
TS (%)
CQ (1–5)
WER (%)
Spk sim
0–1
-276.0 (196)
0.0 (209)
0.0 (90)
4.89 (90)
5.4 (89)
Ref.
1–2
-280.0 (147)
0.0 (168)
0.0 (90)
4.79 (90)
5.2 (90)
0.722 (79)
2–3
-273.5 (148)
0.0 (167)
0.0 (90)
4.63 (90)
5.8 (90)
0.712 (78)
3–4
-292.0 (139)
0.0 (153)
0.0 (90)
4.27 (90)
6.1 (90)
0.696 (77)
Appendix
Table 17: Twenty-minute long-session results. Values in parentheses indicate sample counts. A dash indicates that the reporting threshold is not met, and Ref. denotes the first minute used for speaker enrollment.
Model
Temporal unit
Positions per unit
Local window H
Tile Δ
PersonaPlex
80 ms frame
1
120 frames (9.6 s)
12 frames (0.96 s)
MiniCPM-o 4.5
1 s tick
13–32
10 ticks (10 s)
1 tick (1 s)
Raon-SpeechChat
80 ms frame
2–3
120 frames (9.6 s)
12 frames (0.96 s)
Appendix
Table 18: Model-specific temporal units and TRC window and tile sizes.
Model
Content
Control
PersonaPlex
Ordinary text in consecutive non-PAD frames.
PAD, EPAD, and predictions in the first non-PAD frame after PAD.
MiniCPM-o 4.5
Nonspecial text predictions, including those following <|speak|> within the same tick.
Table 20: Training configuration for Ours (TRC) and Ours (LoRA).
Figure 10: System prompt and user message for dialogue summarization. Braced placeholders indicate the dialogue to be summarized.
Figure 11: Teacher system prompt after a segment boundary. Braced placeholders indicate the base prompt and the summary of preceding dialogue.
Figure 12: User-simulator prompts for trajectory generation. Braced placeholders abbreviate unchanged portions. In the system prompt, the revisiting rule replaces the repetition prohibition, and conversation-context rules are appended. In the user message, the summary and current-segment dialogue replace the dialogue history; before the first segment boundary, the summary is “(conversation just started)”.
Persona
Instruction
Exploratory
Use details from the conversation as starting points for new stories, perspectives, and questions. Let an earlier remark bring to mind a related experience, interest, or possibility, then develop that new direction. Share plausible personal details and preferences as you go. Respond naturally to the other speaker and make transitions gradual. Let this tendency emerge occasionally; not every turn needs to refer back to something, and you need not ask for specific details to be recalled.
Applicative
Bring earlier information into new situations as the conversation develops. Ask the other speaker to use previously discussed circumstances, preferences, or explanations when considering a new possibility, interpreting an experience, or offering a perspective. Make the earlier information relevant to the answer rather than asking for it to be recited. Use only information established in the conversation, including earlier parts. Share new details and respond naturally; not every turn needs to use the past, and no particular decision or conclusion is required.
Recollective
Occasionally return to a specific detail or discussion from earlier in the conversation and ask the other speaker to recall it. Choose something relevant to what you want to discuss next, without restating the answer in your question. Refer only to information actually established in the conversation, including earlier parts. Use the response to continue the exchange rather than grading it. Share new details and respond naturally between these moments; do not make every turn a recall question.
Experiential
Develop the conversation through plausible everyday experiences and observations. Describe concrete situations, how they felt, and what made them interesting, revealing details gradually in response to the other speaker. Invite a reaction or interpretation when useful, and let the exchange lead to another experience or question. Keep individual turns conversational rather than delivering a long story at once. There is no need to ask the other speaker to recall earlier information.
Deliberative
Enjoy exploring different perspectives on an idea or situation. Offer your own view, ask what makes another view appealing, and consider examples, tradeoffs, or exceptions as they arise. Respond thoughtfully without turning the exchange into an argument or requiring agreement. Mix questions with your own reactions and plausible preferences. Let the discussion develop naturally without requiring a decision, and do not make recalling earlier information a goal.
Appendix
Table 21: Persona instructions for teacher-trajectory generation (verbatim).
Topic
Instruction
School memories
Talk about school life and memories of being a student. Explore memorable classes, teachers, classmates, school customs, and how your views of those experiences have changed. Share plausible experiences and invite general perspectives without assuming the other speaker attended school.
Work collaboration
Talk about working with other people. Explore communication, different working styles, sharing responsibilities, receiving feedback, and what makes a group comfortable or frustrating. Everyday experiences and hypothetical situations are both welcome; no project needs to be completed.
Friendship and contact
Talk about how friendships develop and change. Explore becoming comfortable with someone, staying in touch, reconnecting, and balancing different expectations. Share plausible experiences and preferences while inviting the other speaker’s perspective rather than assuming personal friendships.
Family customs
Talk about the different customs and expectations found in families. Explore familiar sayings, ways of showing affection, shared responsibilities, and differences between generations. Personal examples and imagined situations can lead the discussion without requiring sensitive disclosures.
Neighborhood change
Talk about the character of neighborhoods and how places change over time. Explore familiar streets, local shops, neighbors, belonging, and reactions to change. Use plausible recollections and general observations without relying on current information about a named location.
Language and expression †
Talk about how people express themselves through language. Explore the nuances of words, humor, misunderstandings, accents, and expressions that are difficult to explain. Share examples and preferences without turning the conversation into a language lesson or a quiz.
Appendix
Table 22: Topic instructions for teacher-trajectory generation (verbatim). Topics marked with † are held out for validation.
Model
Condition
0 min
5 min
10 min
PersonaPlex
Base
38
0
0
TRC
12
14
7
MiniCPM-o 4.5
Base
85
86
87
TRC
53
57
46
Raon-SpeechChat
Base
76
74
11
TRC 480 ms
43
29
34
Appendix
Table 23: Single-needle retention accuracy with greedy decoding. Each entry is the percentage of correct responses among 100 scorable probes. For Raon-SpeechChat, the two TRC conditions use tile durations of 480 and 960 ms with the same 9.6 s local window.
Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation. But can they sustain a coherent conversation for tens of minutes? Because they generate a token every frame, even while listening, they exhaust their context windows within minutes. Yet existing benchmarks evaluate only single turns or a few conversational rounds, too short to expose the long-horizon problem. We introduce the Full-Duplex Long-Horizon Benchmark (FD-LHB), which tests five recent full-duplex models on long replayed conversations and in live conversation with an LLM partner. Every evaluated model eventually either degrades or exhausts its context. Models that keep the full history often degrade before their context runs out, while sliding-window models still fall into repetition, silence, and gibberish. For sliding-window models, we identify two contributing mechanisms. Gibberish and silence are linked to Initialization-Anchor Eviction, where the sliding window discards the first KV entry. Repetition is also reinforced by the model's own recent outputs. Guided by this diagnosis, we apply two simple fixes: Initialization-Anchor Pinning (IAP) keeps that first entry, and Duplex Repetition Penalty (DRP) discourages echoing while guarding against prolonged silence. Together, they support the diagnosis and, without retraining, substantially reduce conversational collapse in replays of up to an hour while running in real time.
Yeonju Kim, Bella Godiva, Se Jin Park +1
Integrated Vision and Language Lab, KAIST, South Korea · Kyung Hee University, South Korea
Recent full-duplex spoken dialog models enable simultaneous listening and speaking, but fine-grained models still advance their backbone autoregressively at every interaction frame. We introduce DiffuPlex, a rolling masked diffusion framework that reduces this sequential computation by predicting multiple future user and assistant frames in a single backbone wake. DiffuPlex consumes only a confident prefix of each predicted future while interaction continues at the original frame rate. As user speech arrives, it checks the corresponding user predictions and, when the interaction diverges, preserves already played assistant content while revising only the unplayed future. We consider two inference policies over the same predictor: DiffuPlex-LISTEN consumes multiple future frames when they predict assistant silence, whereas DiffuPlex-SPEAK can also consume predicted assistant speech. Across full-duplex interaction and spoken-language evaluations, DiffuPlex substantially reduces sequential backbone computation while largely preserving interaction behavior and general capability. DiffuPlex-LISTEN and DiffuPlex-SPEAK achieve 1.46× and 1.59× deployment-path wall-clock speedups and 1.61× and 1.80× Core LM speedups, with all measured backbone invocations completing within the 80ms interaction interval. Human evaluation shows that LISTEN preserves speech naturalness and conversational quality, while SPEAK retains conversational quality with some degradation in speech naturalness.
Heeseung Kim
Department of Artificial Intelligence University of Seoul Seoul, Republic of Korea
Full-duplex spoken dialogue requires simultaneous listening and speaking at sub-second latency, under conversational timing and cognitive demands that change moment to moment. Yet current models mostly impose static operating points, lacking a systematic mechanism for adaptive decisions. We present AdaptDuplex, which extends Qwen3-Omni with such a mechanism, co-designed across three layers. A compact token-level protocol represents every window as a canonical sequence, trains dual-stream alignment through a bounded text lead over speech, and exposes every behavioral decision as an explicit token for training-free runtime control via logits bias. Adaptive mechanisms dynamically predict among discrete window durations and augment direct response as needed with non-blocking cognitive consolidation and multi-flight external reasoning. A progressive pipeline introduces these behaviors through a three-stage Thinker curriculum, then Talker-only and joint SFT, with GRPO as a preliminary increment. On Full-Duplex-Bench v1 and v1.5, AdaptDuplex outperforms DuplexOmni on 18 of 21 comparable turn-taking, overlap-behavior, and timing metrics and MiniCPM-o 4.5 on 19 of 22, with gains in both interaction decisions and response timing. On human-recorded HumDial-FDBench, it attains the top Final score (69.6) of the compared duplex models.