SCB: SpeechConversationBench for Evaluating Multi-Turn Reasoning in Speech-to-Speech Models

Kanpat Vesessook; Saksorn Ruangtanusak

cs.CLSep 30, 2026

SCB: SpeechConversationBench for Evaluating Multi-Turn Reasoning in Speech-to-Speech Models

Authors: Kanpat Vesessook, Saksorn Ruangtanusak

Organizations: SCBX R&D · SCB DataX, SCBX Group

Abstract

Speech-to-speech systems must solve tasks whose requirements emerge across conversational turns. We introduce SpeechConversationBench (SCB), a focused evaluation of spoken mathematical reasoning using 103 sharded GSM8K problems. The framework compares the original problem delivered in one turn (full), its concatenated information shards delivered together (concat), and incremental spoken disclosure across turns (sharded). We report final-answer accuracy for four commercial speech systems and LEGO, a proprietary speech pipeline developed internally by the SCBX Innovation Lab team with explicit conversational context management. Relative to concat, sharded accuracy decreases by 5.0-25.3 percentage points across the four commercial systems. LEGO achieves 77.5 percent accuracy in all three conditions, compared with 76.6 percent sharded accuracy for GPT-4o Realtime. The two single-turn baselines distinguish sensitivity to problem reformulation from the additional challenges introduced by incremental spoken interaction.

Figures & tables

Panel of Figure 1 — Figure 1: Spoken evaluation conditions. Full and Concat present information together; Sharded distributes it across an exchange. Final spoken answers are assessed for numerical correctness.

Figure 2: Sharded-conversation simulator reproduced from Laban et al. [ 7 ] . This reference architecture motivates the spoken adaptation; it does not specify our audio-processing or scoring components.

System	Full	Concat	Sharded	$\Delta_{S-C}$	$R_{S/C}$
GPT-4o Realtime	70.0	81.6	76.6	$-5.0$	93.9
GPT-4o Mini Realtime	75.5	77.7	52.4	$-25.3$	67.4
Gemini 2.5 Flash Live	54.4	62.1	46.6	$-15.5$	75.0
Gemini 2.5 Flash Preview †	85.0	80.0	55.0	$-25.0$	68.8
LEGO	77.5	77.5	77.5	$0.0$	100.0

Table 1: Final-answer accuracy (%) under each input condition.

\Delta_{S-C}

is the Sharded -minus- Concat change in percentage points;

R_{S/C}

is aggregate accuracy retention (%). Derived columns use the displayed accuracies. Bold marks the highest Sharded accuracy without implying significance.

Explore similar work

Voice of Reason: Reinforcement Learning for Spoken Math
Sep 16, 2026Timothée Weisselberger, Edouard Graves, Alexandre DéfossezNumerical Reasoning in Language Models RL for Language Model Reasoning
Kyutai Paris, France · Kyutai, Gradium Paris, France
Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models
Jul 22, 2026Pengchao Feng, Chao-Hong Tan, Qian Chen +3Efficient Language Model Reasoning Speech Language Models
Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai, China +2
RetroThinker: Enabling Retrospective Thinking in Speech LLMs
Sep 12, 2026Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed +1LLM Self-Correction Efficient Language Model Reasoning
The University of Texas at Austin · FAIR, Meta Superintelligence Labs

Sep 16, 2026, cs.CLJ/K move · Enter open · S save

Voice of Reason: Reinforcement Learning for Spoken Math

Timothée Weisselberger, Edouard Graves, Alexandre Défossez

Kyutai Paris, France · Kyutai, Gradium Paris, France

Numerical Reasoning in Language Models RL for Language Model Reasoning Mathematical Reasoning Speech Language Models

Open paper AskPDF arXiv

Speech language models enable richer spoken interactions between humans and machines than cascaded systems, allowing access to paralinguistic information and lower latency. However, their accuracy on mathematical reasoning benchmarks has lagged behind those of text models. Reinforcement learning (RL) with verifiable rewards has been instrumental in extending text models' capabilities for solving complex problems and limiting hallucinations. In this work, we explore applying RL to the GLM-4-Voice speech model (Zeng et al., 2024) to bridge the gap between textual and spoken mathematical problem solving. We first adapt the model to the domain using supervised fine-tuning on synthesized spoken question-answering data. We then show that, even without extra reasoning tokens, RL improves the accuracy on GSM8K beyond levels previously achieved for speech models only with supplementary reasoning traces. When combined with existing streaming reasoning techniques, we show further gains to 74.8% free-form accuracy. This establishes a new state-of-the-art for mathematical spoken abilities with speech-native models.

Abstract

Explore similar work

Voice of Reason: Reinforcement Learning for Spoken Math

Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models

RetroThinker: Enabling Retrospective Thinking in Speech LLMs