eess.ASNov 11, 2026

Speech Tokenizers are Vulnerable: Transferable Semantic Attack and Robust Tokenizer

Authors: Zhisheng Zhang, Yifan Mi, Yixuan Zhou, Haiyun Li, Shengbo Cai, Dongfu Song, Ziang Xu, Jie Hao, 

Organizations: Shenzhen International Graduate School, Tsinghua University · Beijing University of Posts and Telecommunications

Abstract

Speech tokenizers serve as speech front ends, encoding input speech signals into representations for downstream models, such as automatic speech recognition (ASR) systems and large audio-language models (LALMs). In this paper, we find that speech tokenizers are fragile at the semantic level: small perturbations can disrupt their semantic encoding process and consequently cause severe degradation in downstream ASR and LALM performance. To study this vulnerability, we propose T-SemAttack, a transferable semantic attack based on a semantic encoder ensemble. By jointly drifting the representation spaces of multiple semantic encoders, T-SemAttack effectively disrupts the semantic content of speech while preserving perceptual quality, thereby inducing error features. We further analyze layer-wise representation drift and cross-model gradient alignment. Our findings show that perturbations can induce substantial shifts in deep semantic representations and that the input gradients of different surrogate encoders exhibit directional alignment. These observations suggest that shared adversarially sensitive directions in the input space may be a key factor underlying cross-model attack transferability. Building on these, we introduce ROSETok, a robust speech tokenizer that combines noisy training with robust semantic distillation to improve reconstruction fidelity and downstream SpeechLLM robustness. Extensive experiments on multiple datasets, 9 open-source and black-box ASR systems, and 4 LALMs demonstrate that T-SemAttack achieves strong transferable attack performance, while the proposed Robust Speech Tokenizer exhibits robustness under high-fidelity reconstruction and downstream tasks. Our code and demo are available at https://wxzyd123.github.io/t-semattack.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception

    May 29, 2026Yuhan Song, Linhao Zhang, Aiwei Liu +6Audio Representation LearningAudio Understanding

  2. Speech Codec Probing from Semantic and Phonetic Perspectives

    Mar 11, 2026Xuan Shi, Chang Zeng, Tiantian Feng +3Audio Representation LearningNeural Audio Codecs

  3. TransferBreaker: Breaking Adversarial Transferability in Fine-Tuned Speech Recognition

    Oct 6, 2026Mojtaba Nafez, Aref Mousavi, Mohammad Ebrahim Mahdavi +3Adversarial Robustness of ASRAdversarial Transferability