cs.CLAug 29, 2026
SaveHEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding
Organizations: IPAI, Seoul National University (SNU) · Department of ECE, SNU · Department of EE, Yonsei University · Department of CSE, SNU ·
Abstract
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Figures & tables
Figure 1: Illustration of HEAR benchmark. Inspired by Erber’s auditory hierarchy Erber (1982) , HEAR breaks speaker-attributed reasoning down into several foundational capabilities: discrimination, attribution, and reasoning.
Figure 2: Overview of our dataset and method on speaker-attributed reasoning. (A) Our dataset, CASH, randomly reassigns utterances to different speakers via voice cloning while preserving the whole utterance, yielding counterfactual variants whose ground-truth answer always flips. (B) We show an elementary speaker-attribution task from the What Do You Like? benchmark Wu et al. (2024) , where the query is unsolvable without identifying which speaker said what. (C) Our model first transcribes the multi-party audio scene, then reasons over the transcript for more reliable speaker-aware reasoning.
| Discrimination | Attribution | Reasoning | ||||||||
| Model | Size | VC | VCD | VL | CVA | VCA | IR | QR | TR | Avg. (%) |
| Proprietary Models | ||||||||||
| Gemini-3.1-Pro-preview | - | 68.4 | 89.5 | 93.2 | 94.1 | 86.4 | 96.2 (47.6) | 89.3 (53.6) | 91.3 (51.3) | 88.8 (75.8) |
| Gemini-3-Flash-preview | - | 52.6 | 66.0 | 81.4 | 73.9 | 60.1 | 91.9 (40.5) | 82.5 (39.3) | 80.9 (38.3) | 73.1 (58.5) |
| GPT-4o-audio-preview | - | 40.5 | 21.0 | 28.3 | 60.3 | 43.5 | 80.0 (15.7) | 53.6 (15.7) | 69.6 (17.4) | 48.5 (32.8) |
| Open-Source Models | ||||||||||
Table 1: Evaluation results of 20 leading Speech Language Models on the HEAR benchmark. Values in parentheses in the Reasoning columns report scores under the paired setting, where a prediction is counted as correct only when both the original audio clip and its corresponding semantic hallucination counterpart are correct. A2R results are averaged over three seeds.
Figure 3: Illustration of Find the Spy, a benchmark where speaker attribution is essential. Each player receives a word (e.g., Banana ), except for one spy who receives a different word (e.g., Apple ), and players describe their words in a random speaking order. To identify the spy, a model must do more than recognize which clue is inconsistent with the others: it must also determine who produced that clue .
Figure 4: Performance degradation under overlap samples and the effect of semantic hallucination.
| Model | WDYL | GAOKAO | FTS |
|---|---|---|---|
| Random Choice | 50.0 | 50.0 | 23.8 |
| Qwen3-Omni-30B-A3B-Instruct | 61.3 | 65.6 | 29.2 |
| Qwen3-Omni-30B-A3B-Thinking | 67.1 | 54.8 | 17.2 |
| A2R (CASH w/o HN) | 95.0 | 78.5 | 40.5 |
| A2R (CASH) | 97.7 | 88.2 | 48.2 |
Table 2: Generalization across three benchmarks where speaker attribution is essential. “HN” denotes voice-swapped hard negative variants in CASH.
| Model | Audio | WDYL | GAOKAO | FTS |
|---|---|---|---|---|
| Random Choice | - | 50.0 | 50.0 | 23.8 |
| Qwen3-Omni-30B-A3B-Instruct | Synthetic | 50.0 | 76.7 | 50.0 |
| A2R (CASH) | Synthetic | 96.0 | 93.3 | 65.0 |
| Qwen3-Omni-30B-A3B-Instruct | Human | 50.0 | 63.3 | 40.0 |
| A2R (CASH) | Human | 98.0 | 83.3 | 70.0 |
Table 3: Evaluation on human-recording attribution-critical tasks. We compare the 100 synthetic samples with corresponding recordings produced by human speakers.
Figure 5: Results on semantic hallucination pairs in the “Reasoning” category of HEAR. “HN” denotes voice-swapped hard negative variants in CASH. Lower same-answer rates indicate higher acoustic sensitivity, as each pair is designed to elicit different answers.
Figure 6: Ablation of training regimes on HEAR.
| Datasets | Model | Performance |
|---|---|---|
| AlpacaEval | BBH | AdvBench | CommonEval | Baseline | 4.38 94.00 99.42 4.07 |
| A2R | 4.31 93.90 96.35 3.75 | |
| Ablation (SFT) | 3.19 68.10 89.62 1.90 | |
| WildVoice | IFEval | MMSU | OpenBookQA | Baseline | 4.15 87.97 80.84 95.38 |
| A2R | 3.93 86.01 76.87 92.53 | |
| Ablation (SFT) | 2.14 78.05 75.73 90.33 |
Table 4: Evaluation on VoiceBench. We assess performance drop under dyadic conversational settings.
Figure 7: Prompt Analysis on benchmarks. “Reasoning” denotes a prompt that asks the model to provide brief reasoning before answering, while “Reasoning w/ transcript” asks for a transcript before answering.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Screenshot of the interface shown to human annotators during verification.
| Domain | #Clip | #Src. | #Spk. | Dur. | Spk/clip | Hours |
|---|---|---|---|---|---|---|
| VoxMM | 360 | 34 | 245 | 87.8 | 3.64 | 8.78 |
| AMI | 241 | 23 | 24 | 97.4 | 3.93 | 6.52 |
| ICSI | 286 | 6 | 21 | 94.9 | 4.65 | 7.54 |
| TOTAL | 887 | 63 | 290 | 92.7 | 4.05 | 22.84 |
Table 5: HEAR per-domain composition. #Src. : distinct source recordings. #Spk. : speakers, deduplicated within each corpus’s identity namespace. Dur. : average clip duration, in seconds. Hours : total speech-clip-hours.
Figure 9: HEAR composition. Left: clip distribution across the three source domains ( ). Right: question distribution across the three evaluation dimensions ( ).
Figure 10: Sub-dimension distribution of HEAR questions ( ). VC/VL/VCD make up Discrimination (top three slices), CVA/VCA make up Attribution, and QR/IR/TR make up Reasoning.
Figure 11: Distribution of VoxMM clips in HEAR by genre category ( ).
Figure 12: Per-domain summary statistics for HEAR.
Figure 13: Statistics of clip duration for HEAR.
Figure 14: Sample queries of Discrimination dimension in HEAR benchmark.
Figure 15: Sample queries of Attribution dimension in HEAR benchmark.
Figure 16: Sample queries of Reasoning dimension in HEAR benchmark.
Figure 17: Sample queries of Reasoning dimension in HEAR benchmark.
| Reward term | Symbol | Coefficient |
|---|---|---|
| Answer correctness | ||
| Response format | ||
| Speaker-tagged transcript (cpWER) | ||
| Speaker-count | ||
| Speaker-order (LCS) |
Table 6: Coefficients for the composite reward terms.
| Model | Checkpoint (HF / source) | Temp. | Top- | Max tokens |
|---|---|---|---|---|
| Audio-Flamingo 3 | nvidia/audio-flamingo-3-hf | 0.7 | 0.9 | 512 |
| Fun-Audio-Chat-8B | FunAudioLLM/Fun-Audio-Chat-8B | 0.6 | 0.95 | 512 |
| Gemma 4 E4B-it | google/gemma-4-E4B-it | 1.0 | 0.95 | 512 |
| Kimi-Audio-7B-Instruct | moonshotai/Kimi-Audio-7B-Instruct | 0.0 | – | 2,048 |
| MiDashengLM-7B | mispeech/midashenglm-7b | 0.0 | – | 512 |
| MiniCPM-o-2.6 | openbmb/MiniCPM-o-2_6 | 0.3 | – | 512 |
Table 7: Decoding hyperparameters used for the open-weight baselines in our HEAR evaluation. “–” denotes parameters left unset, using the vLLM engine’s default value (1.0) ( Kwon et al., 2023 ) . For A2R, we use the same parameters as the Qwen3-Omni-30B-A3B-Instruct model.
Figure 18: Instruction prompts used for GRPO experiments.
Figure 19: Instruction prompt template used for HEAR benchmark, consisting of a task-specific listening prefix and a common reasoning suffix.
Figure 20: Instruction prompt template used for What Do You Like and Gaokao benchmarks, consisting of a task-specific listening prefix and a common reasoning suffix.
Figure 21: Instruction prompt used for the Find the Spy benchmark.
More in this topic
Speech Language Models
Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.
See all 301 papers