ReFrame: Silence-Aware Compute Reclamation for Full-Duplex Speech Models
Organizations: Tsinghua University · The Hong Kong University of Science and Technology (Guangzhou) · ScaleX-IO · Sun Yat-sen University
Abstract
Full-duplex speech models execute dense model inference for every audio frame throughout a live interaction. This uniform framewise schedule creates sustained GPU load across dialogue states whose computation has strong, reusable temporal structure. ReFrame formulates full-duplex inference as a silence-aware computation problem. Mutual Silence Speculation (MSS) operates when both participants are silent: it evaluates multi-frame Temporal continuations, verifies text and hidden-state stability, and applies batched Mimi audio-token correction. Listening Frame Folding (LFF) operates while the user speaks and the agent listens: it processes already-arrived user frames through parallel Temporal execution and frame-batched audio-token generation while accepting only the verified prefix. Frame-Budget Reinvestment (FBR) assigns the released frame intervals to resumable external retrieval and language-model workers, then injects timely references through Moshi's conditioning interface. We instantiate ReFrame on five full-duplex models spanning Moshi-family, phase-structured, and flattened autoregressive generation. Across the complete 1,225-conversation FDBench, ReFrame reduces measured GPU execution time by 18.80-31.01% and retains 96.92-100% non-PAD text agreement on five different models. On HaluEval tests, FBR increases the number of correct generated spoken answers from 86 to 205 under the same auxiliary-service budget. ReFrame turns recurring dialogue structure into lower inference cost and higher end-to-end service capability.
Figures & tables
| Model | Method | Time saved | Share | Text | Exact | |
|---|---|---|---|---|---|---|
| Moshiko | MSS | 10.86% | 46.49% | 4.69 | 99.418% | 527/1225 |
| LFF | 10.92% | 40.27% | 2.23 | 100% | 1224/1225 | |
| Joint | 22.09% | 86.75% | – | 99.418% | 527/1225 | |
| Moshika | MSS | 10.96% | 46.52% | 3.62 | 99.331% | 505/1225 |
| LFF | 11.31% | 40.14% | 2.26 | 99.996% | 1224/1225 | |
| Joint | 22.65% | 86.67% | – | 99.328% | 504/1225 |
| Execution policy | Time saved | Non-PAD text agreement | CB difference |
|---|---|---|---|
| Fixed repeat, strict verification | 1.84 | 100.00 | 0.00 |
| VAD bypass | 24.27 | 85.18 | 26.73 |
| MSS | 10.86 | 96.92 | 5.35 |
| Whole-block replay | 2.13 | 100.00 | 0.00 |
| LFF | 10.92 | 100.00 | 0.00 |
| Joint | 22.09 | 96.92 | 5.35 |
| Dwell | GPU | Text | A. | ||
|---|---|---|---|---|---|
| 4 | 16 | 10.40% | 3.458 | 99.268% | 0.819% |
| 6 | 16 | 10.72% | 3.624 | 99.320% | 0.743% |
| 8 | 16 | 10.97% | 3.742 | 99.203% | 0.835% |
| 6 | 12 | 11.39% | 3.739 | 99.190% | 0.874% |
| 6 | 20 | 9.47% | 3.658 | 99.332% | 0.748% |
| GPU | Mean onset delay | ||
|---|---|---|---|
| 2 | 7.25% | 1.533 | 0.93 frames |
| 4 | 11.25% | 2.264 | 2.78 frames |
| 6 | 11.64% | 2.755 | 4.62 frames |
| 8 | 10.68% | 3.088 | 6.45 frames |
| Metric | Dense | Joint |
| Questions with actual reference injection | 112 | 356 |
| Correct generated spoken answers | 86 | 205 |
| Spoken-answer accuracy (%) | 17.20 | 41.00 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Abbr. | FDBench category | Conversations |
|---|---|---|
| CPH | Candor Pause Handling | 216 |
| CTT | Candor Turn Taking | 119 |
| IBC | ICC Backchannel | 55 |
| SPH | Synthetic Pause Handling | 137 |
| SUI | Synthetic User Interruption | 200 |
| BGS | Background Speech | 100 |
| Model | CPH | CTT | IBC | SPH | SUI | BGS | TTO | UBC | UI |
|---|---|---|---|---|---|---|---|---|---|
| Moshiko | 26.37 | 24.33 | 26.19 | 17.33 | 14.50 | 26.80 | 22.79 | 32.07 | 20.09 |
| Moshika | 27.57 | 27.12 | 22.95 | 19.01 | 14.02 | 27.23 | 23.17 | 30.06 | 22.01 |
| Moshika-RL | 25.33 | 24.01 | 25.42 | 20.41 | 13.40 | 16.01 | 14.94 | 18.99 | 16.99 |
| BayLing-Duplex | 58.35 | 44.98 | 63.94 | 62.16 | 13.19 | 7.18 | 7.16 | 9.28 | 14.19 |
| PersonaPlex | 10.50 | 20.42 | 4.42 | 11.91 | 18.62 | 36.14 | 32.73 | 45.96 | 23.09 |
| Task | Text | CB | WER | O@3 | SNR |
|---|---|---|---|---|---|
| CPH | 100.00 | 0.00 | 0.00 | 100.00 | 103.40 |
| CTT | 99.84 | 0.24 | 1.48 | 99.33 | 99.60 |
| IBC | 100.00 | 0.00 | 0.00 | 100.00 | 104.41 |
| SPH | 100.00 | 0.00 | 0.00 | 100.00 | 103.91 |
| SUI | 99.20 | 0.96 | 5.22 | 97.33 | 84.64 |
| BGS | 99.57 | 0.40 | 3.31 | 97.91 | 91.98 |
| Task | Text | CB | WER | O@3 | SNR |
|---|---|---|---|---|---|
| CPH | 100.00 | 0.00 | 0.00 | 100.00 | 104.16 |
| CTT | 99.80 | 0.33 | 1.93 | 99.35 | 95.74 |
| IBC | 100.00 | 0.00 | 0.00 | 100.00 | 104.86 |
| SPH | 100.00 | 0.00 | 0.00 | 100.00 | 104.62 |
| SUI | 98.77 | 1.32 | 5.86 | 97.06 | 77.81 |
| BGS | 99.64 | 0.40 | 4.10 | 99.13 | 93.98 |
| Task | Text | CB | WER | O@3 | SNR |
|---|---|---|---|---|---|
| CPH | 100.00 | 0.00 | 0.00 | 100.00 | 104.24 |
| CTT | 99.95 | 0.06 | 0.12 | 99.79 | 101.31 |
| IBC | 100.00 | 0.00 | 0.00 | 100.00 | 105.71 |
| SPH | 100.00 | 0.00 | 0.00 | 100.00 | 104.37 |
| SUI | 98.97 | 1.21 | 3.69 | 97.69 | 67.57 |
| BGS | 99.60 | 0.48 | 2.25 | 98.28 | 78.77 |
| Task | BayLing-Duplex | PersonaPlex |
|---|---|---|
| CPH | 216/216 | 216/216 |
| CTT | 119/119 | 119/119 |
| IBC | 55/55 | 55/55 |
| SPH | 137/137 | 137/137 |
| SUI | 200/200 | 200/200 |
| BGS | 100/100 | 100/100 |
| Model | MSS configuration |
|---|---|
| Moshiko | Dwell 16; hidden similarity 0.990; return hazard 0.050; ; batched CB0–CB2 correction. |
| Moshika | Dwell 16; hidden similarity 0.995; return hazard 0.229; ; batched CB0–CB2 correction. |
| Moshika-RL | Dwell 24; hidden similarity 0.995; r95 return-hazard calibration with threshold 0.465; ; batched CB0–CB2 correction. |
| BayLing-Duplex | User-silence threshold 0.005; 15-token repeat chunk representing five logical frames; exact-prefix verification. |
| PersonaPlex | Period-two silence block; dwell 4; eight-frame proposal window; text and all eight agent codebooks verified. |
| Model | LFF configuration |
|---|---|
| Moshiko | ; queue depth 3; persistent frame queue; prefix validation and suffix repair. |
| Moshika | ; queue depth 3; persistent frame queue; prefix validation and suffix repair. |
| Moshika-RL | ; queue depth 3; persistent frame queue; prefix validation and suffix repair. |
| BayLing-Duplex | Five-frame future-block horizon; exact-prefix commit in the FP32 flattened token stream. |
| PersonaPlex | ; queue depth 3; immediate playout with repaired model state. |
Full-Duplex Spoken Dialogue Systems
Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.
See all 129 papers