Speech tokenizers serve as speech front ends, encoding input speech signals into representations for downstream models, such as automatic speech recognition (ASR) systems and large audio-language models (LALMs). In this paper, we find that speech tokenizers are fragile at the semantic level: small perturbations can disrupt their semantic encoding process and consequently cause severe degradation in downstream ASR and LALM performance. To study this vulnerability, we propose T-SemAttack, a transferable semantic attack based on a semantic encoder ensemble. By jointly drifting the representation spaces of multiple semantic encoders, T-SemAttack effectively disrupts the semantic content of speech while preserving perceptual quality, thereby inducing error features. We further analyze layer-wise representation drift and cross-model gradient alignment. Our findings show that perturbations can induce substantial shifts in deep semantic representations and that the input gradients of different surrogate encoders exhibit directional alignment. These observations suggest that shared adversarially sensitive directions in the input space may be a key factor underlying cross-model attack transferability. Building on these, we introduce ROSETok, a robust speech tokenizer that combines noisy training with robust semantic distillation to improve reconstruction fidelity and downstream SpeechLLM robustness. Extensive experiments on multiple datasets, 9 open-source and black-box ASR systems, and 4 LALMs demonstrate that T-SemAttack achieves strong transferable attack performance, while the proposed Robust Speech Tokenizer exhibits robustness under high-fidelity reconstruction and downstream tasks. Our code and demo are available at https://wxzyd123.github.io/t-semattack.
Figures & tables
Figure 1: Gaussian Noise vs. T-SemAttack across Tokenizers. We feed the pre-quantized representations from each tokenizer into SmolLM2-135M Allal et al. (2025) and evaluate downstream semantic understanding on the FSC spoken command intent recognition task (↑) .
Figure 2: The architecture and pipeline of T-SemAttack and ROSETok.
Figure 3: Layer-wise analysis of representation similarity and effectiveness of perturbation magnitude.
Methods
RTF ( ↓ )
SNR ( ↑ )
Surrogate Model
Open-source Model
Black-box Model
Avg.
HuBERT
Whisper
Qwen3-ASR
VibeVoice
FireRed-ASR
Fun-ASR
iFlytek
Doubao
GPT-4o
LibriTTS: A single-speaker English dataset.
Clean
–
–
2.694
2.079
2.369
2.156
2.017
1.866
3.648
3.463
2.234
2.503
Gaussian Noise
0.003
15.197
3.228
2.213
2.583
2.736
2.933
2.488
3.837
3.691
6.677
3.376
E2E-VGuard
26.797
14.341
10.781
5.834
6.781
8.325
11.376
8.842
8.672
7.618
8.030
8.473
AudioShield
–
-2.440
16.007
10.363
15.450
15.241
13.800
12.171
16.815
14.273
11.049
13.908
Table 1: Comparison of attack efficiency, perceptual quality, and attack effectiveness across ASR systems and datasets. WER (%) is reported, where higher values indicate stronger attack effectiveness. The best results are highlighted and second-best results are underlined .
Figure 4: ASR performance across different datasets and LALMs, measured by WER (%).
Method
Denoiser
DeepFilterNet3
Adversarial Defense
Diffusion
Audio Transformation
Whisper
VibeVoice
Whisper
VibeVoice
Resample
Mel
Q-D
AudioPure
Speed
PS
Band-filter
Gaussian Noise
2.964
1.720
2.468
1.133
5.256
6.569
2.984
8.737
57.517
3.735
20.334
E2E-VGuard
10.205
7.276
10.168
7.148
13.841
17.650
9.447
13.898
67.112
10.282
28.863
AudioShield
9.930
9.738
10.741
9.187
16.030
16.751
14.673
37.497
61.546
13.379
23.901
T-SemAttack
96.763
42.901
113.997
37.805
55.805
58.997
73.364
21.571
89.898
70.635
71.085
Table 2: Robustness evaluation against advanced denoising models across two models, adversarial audio defense, and audio transformations on the VibeVoice-ASR with averaged WER(%) values.
Input
Qwen2-Audio
Qwen3-Omni
Audio-Flamingo 3
Gemini 3 Flash
Avg. ( ↓ )
Clean
6.63
7.73
8.14
7.81
7.58
T-SemAttack
4.58
5.23
6.20
5.89
5.48
Table 3: Response Quality Scores of LALMs on AIR-Bench with and without T-SemAttack. Avg. denotes the mean across the four models.
Model
Nq
RTF ( ↓ )
Reconstruction ( ↑ )
Downstream SpeechLLM ( ↑ )
Clean
Gaussian
T-SemAttack
Clean
Gaussian
T-SemAttack
Acoustic and Semantic Speech Tokenizer
SNAC
4
0.003
0.874
0.817
0.782
28.3
5.6
14.6
XY-Tokenizer
8
0.025
0.905
0.849
0.762
92.3
87.8
23.1
X-Codec 2
1
0.021
0.864
0.808
0.771
20.1
8.1
7.4
DAC
9
0.008
0.943
0.891
0.830
25.1
9.1
13.9
Table 4: Reconstruction and downstream evaluation of speech tokenizers under different inputs.
Figure 9
Figure 6: Ablation study for the T-SemAttack. (a)(b) Component Effect. (c)(d) Sensitivity to λpsy .
Teacher
Clean inputs
Noisy inputs
FSC (MLP)
STOI( ↑ )
PESQ( ↑ )
STOI( ↑ )
PESQ( ↑ )
FSC( ↑ )
Whisper
0.965
3.063
0.950
2.431
73.90
HuBERT
0.965
3.088
0.950
2.453
31.48
Table 6: Effect of the semantic teacher on ROSETok-Base reconstruction and downstream semantic performance.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Method
CosyVoice [ 12 ]
CosyVoice2 [ 13 ]
Step-Audio-TTS [ 38 ]
WER( ↑ )
SIM( ↓ )
WER( ↑ )
SIM( ↓ )
WER( ↑ )
SIM( ↓ )
Clean
4.583
0.599
4.484
0.836
3.734
0.772
T-SemAttack
52.476
0.212
33.427
0.330
122.746
0.299
Appendix
Table 7: Case study of T-SemAttack in the TTS protection scenario. The higher WER(%) represents more severe semantic errors in the generated speech. The lower the SIM, the lower the similarity between the generated speech and the original speaker.
Method
MOS-Q ( ↑ )
MOS-C ( ↑ )
MOS-S ( ↑ )
E2E-VGuard
2.67 ± 0.21
4.29 ± 0.17
3.98 ± 0.20
AudioShield
2.63 ± 0.30
3.32 ± 0.31
1.66 ± 0.33
T-SemAttack
3.01 ± 0.28
4.32 ± 0.19
3.87 ± 0.25
Appendix
Table 8: Subjective evaluation from audio quality, content intelligibility, and speaker similarity aspects with 95% confidence intervals.
Figure 7: The screenshot of the questionnaire for subjective experiments.
Hyperparameter
Value
lr
1e−3
lr_min
1e−6
max_steps
500
ϵ
8/255
Appendix
Table 9: Hyperparameters of T-SemAttack.
Dataset
Subset
Duration (# h)
Task
Language
Usage
Librispeech
train & val
960
ASR
English
Ablation & ROSETok-AAF
Emilia
EN
46,800
ASR
English
ROSETok-Base
Emilia
ZH
49,900
ASR
Chinese
ROSETok-Base
Appendix
Table 10: Detailed information of training datasets.
Hyperparameter
Value
ROSETok-Base
optimizer
AdamW
lr
1.0×10−4
lr_scheduler
CosineLR
min_lr
1.0×10−5
batch_size
128
Appendix
Table 11: Training Hyperparameters of the ROSETok.
Stage
RD Mean
RD Std
RD Median
CD Mean
CD Median
Amp(RD)
Waveform
0.154
0.023
0.153
0.322
0.326
1.00×
Mel Spectrogram
0.771
0.070
0.772
0.357
0.347
5.01×
Encoder Output
1.520
0.055
1.530
1.248
1.250
9.87×
Appendix
Table 12: Perturbation amplification results in the S3 Tokenizer.
Figure 8: Single-sample attention maps of Qwen2-Audio under clean and adversarial inputs. Each row corresponds to one Transformer layer, while the left and right columns show clean and adversarial maps, respectively. Green dashed lines mark the audio-token boundaries, and red dashed lines mark the beginning of the target-text region.
Figure 9: Single-sample attention difference maps between adversarial and clean inputs. The visualized value is ΔAl=Aladv−Alclean . Red regions indicate increased attention under adversarial input, while blue regions indicate decreased attention.
Method
WER (%)
Transcriptions
Whisper Large
Original
–
roaming endlessly around the park she wants to go home
Clean
0.000
roaming endlessly around the park she wants to go home
Gaussian
10.000
blooming endlessly around the park she wants to go home
E2E-VGuard
10.000
roaming endlessly around the park he wants to go home
AudioShield
90.000
roman did not listen to the wrong that hour she wanted to go home
Semantic speech tokenizers have become a widely used interface for Audio-LLMs, owing to their compact single-codebook design and strong linguistic alignment. However, their focus on linguistic abstraction induces acoustic blindness, limiting their applicability beyond speech-centric tasks. We propose UniAudio-Token, a framework that empowers semantic tokenizers with general audio perception without compromising speech ability. Instead of altering the semantic paradigm, UniAudio-Token mitigates its information loss through two key innovations: (1) Semantic-Acoustic Primitives (SAP) provide structured supervision by decomposing audio into linguistic content, vocal attributes, and auditory-scene primitives; and (2) Semantic-Acoustic Equilibrium (SAE) introduces a content-aware gating mechanism that adaptively restores fine-grained acoustic details from shallow layers. Extensive evaluations show that UniAudio-Token learns comprehensive universal representations while preserving high-fidelity speech generation. When integrated with downstream LLMs, it outperforms all single-codebook baseline tokenizers on both understanding and generation tasks, effectively serving as a unified audio interface. We publicly release all our code, including training and inference scripts, together with the model checkpoints at https://github.com/Tencent/Universal_Audio_Tokenizer.
Yuhan Song, Linhao Zhang, Aiwei Liu +6
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Basic Model Technology Center, WeChat AI, Tencent Inc.
Speech tokenizers are essential for connecting speech to large language models (LLMs) in multimodal systems. Speech tokenizers are expected to preserve both semantic and acoustic information for downstream understanding and generation tasks. However, emerging evidence suggests that the term "semantic" in speech processing does not align with linguistic lexical-semantic, leading to a mismatch between speech and text modality. In this paper, we systematically analyze the information encoded by several widely used speech tokenizers, evaluating their lexical-semantic and phonetic content through three tasks. Our results show that current tokenizers primarily capture phonetic rather than lexical-semantic structure, deriving practical implications for the design of next-generation speech tokenization methods. Code is released to public at https://github.com/Alexuan/codec_probing_release.
Xuan Shi, Chang Zeng, Tiantian Feng +3
University of Southern California, USA · Dolby Laboratories, USA
Many organizations fine-tune publicly available pretrained Automatic Speech Recognition (ASR) models and deploy them in black-box settings, assuming limited access provides protection. We show this assumption is fragile: adversarial perturbations crafted on the public base model transfer effectively to fine-tuned target models, severely degrading performance and posing concerns for safety-critical applications. We propose TransferBreaker, a unified fine-tuning framework that suppresses adversarial transfer by integrating Base Adversarial Fine-Tuning, which restricts adversarial training to base-effective perturbations; Latent Jacobian Regularization, which enforces latent-space invariance by suppressing adversarially sensitive directions; and HybridGrad-AFT, which improves robustness against adaptive attacks by interpolating transferable perturbations from base and target gradients. We theoretically justify all components and evaluate TransferBreaker across three languages and four large ASR models, reducing adversarial WER from 92.6 to 27.8. Our code is publicly available at https://github.com/rohban-lab/TransferBreaker.
Mojtaba Nafez, Aref Mousavi, Mohammad Ebrahim Mahdavi +3
Idiap Research Institute · École Polytechnique Fédérale de Lausanne (EPFL) · Department of Computer Engineering Sharif University of Technology +1