cs.SDAug 20, 2026
SaveTowards Quantifying Benchmark Optimization in ASR Models
Organizations: Hume AI Research
Abstract
Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety of mechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.
Figures & tables
| Probe case | Reference rendering | Comparison rendering |
|---|---|---|
| Reference insertion | we should return to the Geneva format | we should return to Geneva format |
| Reference deletion | Mr. President | Thank you Mr. President |
| Reference substitution | in a position | in a role |
| Masked word | we also had 50 international US experts | we also had <masked> international US experts |
| Orthographic variant | Mr President | Mister President |
Figure 1: Example renderings used by the behavioral probes. Red marks the reference span being tested; green marks the competing audio-true or acoustically equivalent rendering. Reference insertion, deletion, and substitution are the three edit types used by the reference-disagreement probe. Masking induces a deletion-style comparison by silencing a word (where <masked> indicates a silent audio interval, not a literal model output), while orthographic switching compares two acoustically identical spellings.
Figure 2: Cross-model audit on VoxPopuli. WER (%) is the VoxPopuli-test score from the June 2026 Open ASR Leaderboard [ 38 ] . Kimi Audio is not on the leaderboard, and its score is computed using the leaderboard’s scoring. Consensus-panel members are scored against edits flagged unanimously by the remaining three members (§ 3.3 ).
Figure 3: ( 3(a) ) Masked-number accept-ref per corpus. ( 3(b) ) Orthographic switch rate on archaic spacing.
Figure 4: Difference in masked-number recovery between LibriSpeech test-set narrator clones and held-out libri-fresh narrator clones reading identical sentences (test-set minus held-out; positive values indicate greater recovery for test-set narrator clones). Sentence-clustered bootstrap 95% confidence intervals.
Figure 5: The trigger battery, VoxPopuli. Top row: voice conditions on identical transcripts—( 5(a) ) reference-disagreement and ( 5(b) ) masked-number accept-ref . Bottom row: the same probes with the trigger removed instead of the voice varied—truncation to the edit, an appended s conversational donor, or the learned register direction projected out of one encoder layer ( 5(c) , 5(d) ). Wilson CIs.
Figure 6: Switching the benchmark-optimized policy on and off. ( 6(a) ) Input level: On real clips (top left) a conversational donor collapses accept-ref while a VoxPopuli donor leaves it intact; on ep-fresh clones of the same sentences (top right) a VoxPopuli donor re-ignites it while the conversational donor does not. ( 6(b) ) Activation level: projecting out the learned direction on real benchmark edits (bottom left) and adding it on generic-voice clones (bottom right). The remaining consensus-panel members show no effect, like Voxtral-Mini-3B.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| consensus | human-annotated edits | |||
|---|---|---|---|---|
| model | accept-ref | accept-ref | CI | |
| Cohere-Transcribe | 0.30 | 0.52 | [0.47, 0.58] | 253/483 |
| Granite-Speech-4.1-2B | 0.21 | 0.42 | [0.36, 0.47] | 211/508 |
| Canary-Qwen-2.5B | 0.23 | 0.41 | [0.36, 0.47] | 210/507 |
| Higgs-Audio-v3-8B | 0.21 | 0.39 | [0.33, 0.44] | 200/517 |
| Phi-4-Multimodal | 0.19 | 0.38 | [0.33, 0.44] | 195/510 |
Table 1: accept-ref on VoxPopuli-AA human-annotated edits, beside the consensus rates of Figure 2 . Ordering is nearly identical aside from Granite and Canary swapping places.
Figure 7: Reference-disagreement accept-ref on the real VoxPopuli recordings under content-preserving perturbations (additive noise dB; measured room reverberation, RT60 ). Wilson CIs.
Figure 8: Honorific switch rate ( Mr / Mister ), all 11 models. A rate above (dashed) means the model tracks each corpus’s convention at rates above chance.
Figure 9: Base-anchored Mister rate for the same sentence rendered in a cloned VoxPopuli-speaker voice vs. a cloned LibriSpeech-narrator voice; ticks mark the real-recording rate. Left: VoxPopuli Mr sentences ( ; voice transfer). Right: LibriSpeech Mister sentences ( ; voice erosion). Granite retains conventions on real audio but loses them under any TTS clone, so its clone cells are relatively uninformative; Whisper, Phi-4, and Voxtral never emit Mister .
Figure 10: Audio lift (Eq. 1 ), in nats per character; bootstrap CIs. Parakeet-TDT is excluded (no teacher-forced readout). ( 10(a) ) The reference disagreement audio lift. ( 10(b) ) The masked entity audio lift for different voice conditions for paired sentences.
| model | full | truncated | donor ablated | activation ablated |
|---|---|---|---|---|
| Cohere-Transcribe | 0.30 | 0.13 | 0.06 | 0.04 |
| Canary-Qwen-2.5B | 0.23 | 0.12 | 0.05 | 0.02 |
| Granite-Speech-4.1-2B | 0.21 | 0.12 | 0.20 | 0.14 |
| Higgs-Audio-v3-8B | 0.21 | 0.12 | 0.03 | – |
| Phi-4-Multimodal | 0.19 | 0.09 | 0.05 | 0.20 |
| Parakeet-TDT-0.6B-v2 | 0.18 | 0.08 | 0.06 | 0.01 |
Table 2: Reference-disagreement accept-ref on consensus edits as audio context is removed or the trigger is ablated. truncated cuts the audio to a tight window around the edit span ( aligned word s); donor ablated appends an s conversational donor to the full clip; activation ablated projects the learned register direction out of a single encoder layer.
| consensus accept-ref | masked accept-ref | |||
|---|---|---|---|---|
| model | VoxPopuli | ep-fresh | VoxPopuli | ep-fresh |
| Cohere-Transcribe | 0.304 | 0.122 | 0.185 | 0.074 |
| Canary-Qwen-2.5B | 0.233 | 0.150 | 0.051 | 0.062 |
| Granite-Speech-4.1-2B | 0.208 | 0.117 | 0.051 | 0.062 |
| Higgs-Audio-v3-8B | 0.211 | 0.168 | 0.070 | 0.040 |
| Phi-4-Multimodal | 0.188 | 0.193 | 0.076 | 0.044 |
Table 3: Full-probe accept-ref on real audio with each corpus’s own content: VoxPopuli-test vs ep-fresh . Masked columns score the corpus-paired subsets, so the VoxPopuli masked rates differ from the full masked set of § 4 .
Figure 11: Context gating on all consensus edits, conditioned on reproduction: for each model with a high accept-ref , the edits it reproduces on the full clip, re-decoded on the isolated window of Table 2 . Reference-extra words mostly disappear and suppressed audible words resurface once the surrounding benchmark context is cut away. Wilson CIs.
Figure 12: Edit-locus dissociation (Cohere, Granite, Canary-Qwen). Isolating the span restores faithful transcription for both edit types (context gating; the all-model version is Figure 11 ). Patching in a context-free encoding of the span removes reference insertions (encoder-side) but never restores reference omissions (decoder-side).
Figure 13: Masked accept-ref , real VoxPopuli minus a register-matched held-out clone (fresh post-cutoff parliament), same transcript, paired CI. Positive means the model reads a silenced number from the benchmark voice but not a held-out voice; significant for Cohere and Canary (red).
| real | vox-clone | ep-fresh | generic | real ep-fresh | real generic | |
|---|---|---|---|---|---|---|
| Cohere-Transcribe | ||||||
| Canary-Qwen-2.5B | ||||||
| Granite-Speech-4.1-2B | ||||||
| Phi-4-Multimodal | ||||||
| Higgs-Audio-v3-8B | ||||||
| Whisper-Large-v3 |
Table 4: Audio lift (Eq. 1 , nats/char) of the silenced number span by voice condition ( paired sentences passing the intelligibility gate in every clone condition). Diff columns: paired differences, bootstrap CIs; bold marks CIs excluding zero. Whisper’s lift rises on clean TTS, so a drop in lift is likely not a synthesis artifact; Qwen3’s lift is negative in every condition, so its diffs do not indicate recovery. Parakeet-TDT has no teacher-forced readout.
| model | truncated | attn-isolated | translate | full |
|---|---|---|---|---|
| Voxtral-Mini-3B | 1.00 | 1.00 | 1.00 | 1.00 |
| Whisper-Large-v3 | 1.00 | 1.00 | 1.00 | 1.00 |
| Moonshine-Streaming | 1.00 | 1.00 | – | 1.00 |
| Qwen3-ASR-0.6B | 0.94 | 0.95 | 1.00 | 1.00 |
| Kimi-Audio-7B | 1.00 | 0.89 | – | 0.89 |
| Cohere-Transcribe | 0.94 | 0.26 | – | 0.00 |
Table 5: The opening-courtesy case study: rate at which the audible courtesy is present in the output. truncated : the audio is cut to the opener; attn-isolated keeps the full-clip audio encoding but restricts the decoder’s attention over it to the opener’s frames. translate : the same audio decoded under an English Spanish translation instruction. full : the entire clip. – marks models without translation or attention-isolation capabilities.
More in this topic
ASR Evaluation
Automatic speech recognition (ASR) evaluation measures transcript accuracy, robustness, fairness, and usefulness across languages, speakers, and real-world conditions.
See all 144 papers