eess.ASApr 13, 2026

Why Your Tokenizer Fails in Information Fusion: A Timing-Aware Pre-Quantization Fusion for Video-Enhanced Audio Tokenization

Authors: Xiangyu Zhang, Benjamin John Southwell, Siqi Pan, Xinlei Niu, Beena Ahmed, Julien Epps

Organizations: School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, Australia.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Audio tokenization has emerged as a critical component in end-to-end audio language models, enabling efficient discrete representation learning for both audio understanding and generation tasks. However, existing audio tokenizers face fundamental limitations in understanding tasks due to single-modality constraints, particularly when audio signals contain ambiguous or incomplete information. While incorporating additional modality information can significantly enhance audio understanding, current multimodal fusion approaches invariably degrade reconstruction quality. This degradation is unacceptable for end-to-end audio systems that require high-fidelity audio generation capabilities. In this work, we investigate the root causes of reconstruction quality degradation in video-enhanced audio tokenization and present three key findings. First, the location of fusion within the tokenizer architecture is crucial for preserving reconstruction quality. Second, we show that contrastive learning, though effective in continuous representation fusion, is unsuitable for discrete tokenizers as it fails to enhance downstream task performance. Third, while feature-dimension fusion approaches achieve moderate success, we discover that fusing along the temporal axis -- guided by the concept of distinctive features -- yields significantly better results. Building on these insights, we introduce the Timing-Aware Pre-Quantization Fusion for Video-Enhanced Audio Tokenization, the first approach to successfully integrate visual information into audio tokenizer architectures while preserving reconstruction fidelity. Our approach not only maintains high-fidelity reconstruction but also achieves superior performance on downstream understanding tasks compared with audio-only tokenizers and established multimodal fusion baselines.

Figures & tables

Fig. 1 : Comparison of fusion strategies in video-enhanced audio tokenization. (a) Pre-quantization fusion integrates visual features with audio representations before the quantization layer through contrastive learning or knowledge distillation. (b) Post-quantization fusion incorporates visual information within the residual vector quantizer (RVQ) during the quantization process using contrastive learning or knowledge distillation approaches.
Fig. 2 : Downstream evaluation framework. Discrete VQ codes from the audio tokenizer are mapped to language model embeddings via a trainable Audio Projector. The Llama 3.1 8B model (frozen) processes audio tokens with textual questions for classification. Only the Audio Projector and classification head are trainable, isolating tokenization quality effects.
Fusion StrategyFusion Methodλfusion\lambda_{fusion}Mel Error ↓\downarrowSTFT Distance ↓\downarrowViSQOL ↑\uparrowSI-SDR ↑\uparrowAVQA Accuracy ↑\uparrow
Audio-Only Baseline-00.4660.7864.3303.8640.6474
Quantization-LevelContrastive10.4800.8184.2993.6100.5399
Quantization-LevelContrastive1200.6441.1733.9411.2150.4101
Quantization-LevelDistillation10.4810.8374.2483.8250.6838
Quantization-LevelDistillation1200.5010.8694.2522.7750.5004
Pre-QuantizationContrastive10.4680.8174.3354.0580.5507
TABLE I : Comparison of fusion strategies for video-enhanced audio tokenization. Results show reconstruction quality metrics and downstream task performance on AVQA. Lower values indicate better performance for Mel Error and STFT Distance, while higher values are better for ViSQOL, SI-SDR, and AVQA Accuracy.
Fig. 3 : Gradient flow analysis comparing quantization-level fusion (left) and pre-quantization fusion (right) during training. Top row shows gradient norm evolution across model components; bottom row displays gradient variance as a stability metric. Pre-quantization fusion exhibits smoother gradient flow and higher stability, particularly in encoder and transformation layers.
Fig. 4 : Overview of the proposed Timing-Aware Pre-Quantization Fusion (TAPF) architecture. Left: Visual features are fused with continuous audio features before quantization, guided by a novel distillation loss. Right: The loss mechanism aligns each video feature with audio features inside a dynamically-sized temporal window, enabling flexible cross-modal alignment.
ModelDatasetQuantizerFrame RateToken RateViSQOL ↑\uparrowAVQA Accuracy ↑\uparrow
Audio-Visual Baseline
A-V Static FusionAudioSetRVQ8504004.2800.6952
A-V Static FusionAudioSetFSQ50503.9420.5832
Audio-Only Tokenizer Baselines
WavTokenizerSeveralVQ75754.3320.6734
DAC TokenizerSeveralRVQ931.252814.3040.6561
TABLE II : Performance comparison of the proposed TAPF model against audio-visual and audio-only baselines on reconstruction quality (ViSQOL) and downstream understanding (AVQA Accuracy).
Model ConfigurationViSQOL ↑\uparrowAVQA Acc. ↑\uparrow
TAPF (Full Model)4.0970.6941
Ablation of Core Components
- w/o Dynamic Window3.9970.5160
Window Size Variants
Wmin=1,Wmax=5W_{\text{min}}=1,W_{\text{max}}=53.980.4900
Wmin=1,Wmax=7W_{\text{min}}=1,W_{\text{max}}=7 (Ours)4.0970.6941
TABLE III : Ablation Study of TAPF Components (FSQ Quantizer)
More in this topic

Audio Tokenization

Audio tokenization converts speech, music, and other sounds into compact units for machine learning, generation, and understanding.

See all 43 papers

Explore similar work

CardsList
Jun 29, 2026

AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation

Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present AVTok, a novel unified tokenizer designated for holistic audio-video generation. AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries to efficiently and effectively encode an audio-video pair into a compact one-dimensional latent representation with a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise a hierarchical training strategy to progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified large multimodal models for audio-video generation.

Kien T. Pham, I Chieh Chen, Qifeng Chen +1

The Hong Kong University of Science and Technology

Audio-Video GenerationAudio-Visual AlignmentMultimodal Token Compression
AskPDF
Feb 27, 2026

DashengTokenizer: One layer is enough for unified audio understanding and generation

This paper introduces DashengTokenizer, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks. Unlike conventional approaches, which train acoustic tokenizers and subsequently integrate frozen semantic knowledge, our method inverts this paradigm: we leverage frozen semantic features and inject acoustic information. In linear evaluation across 22 diverse tasks, our method outperforms previous audio codec and audio encoder baselines by a significant margin while maintaining competitive audio reconstruction quality. Notably, we demonstrate that this acoustic injection improves performance for tasks such as speech emotion recognition, music understanding, and acoustic scene classification. We further evaluate the tokenizer's generative performance on text-to-audio (TTA), text-to-music (TTM), and speech enhancement (SE). Our approach surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder. Finally, our results challenge the prevailing assumption that VAE-based architectures are a prerequisite for audio synthesis. Checkpoints are available at https://huggingface.co/mispeech/dashengtokenizer.

Heinrich Dinkel, Xingwei Sun, Gang Li +8

MiLM Plus, Xiaomi Inc., Beijing, China

Audio TokenizationAudio Representation LearningAudio UnderstandingText-to-Audio GenerationText-to-Music Generation
AskPDF
Jun 1, 2026

EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement

Audio tokenizers serve as the discrete interface between continuous audio and Audio Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation. Reconstruction-oriented codecs preserve acoustic fidelity but lack rich semantics, while semantic-aware tokenizers typically rely on separate semantic and acoustic streams, introducing redundancy or misalignment. We propose EntangleCodec, a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations before quantization. By aligning audio with rich captions rather than ASR transcripts, EntangleCodec captures linguistic content, speaker identity, emotion, prosody, and acoustic scenes within a compact token stream. A flow-matching diffusion decoder further enables high-quality reconstruction across speech, music, and general audio. EntangleCodec achieves reconstruction quality competitive with specialized codecs, outperforms all codec-based baselines on audio understanding by up to +7.4% on MMAR, and supports both TTS and TTA generation in a unified framework. Furthermore, EntangleCodec-based audio language models demonstrate strong scaling behavior: even at 0.6B parameters, the model surpasses specialized continuous-representation LLMs with over 13B parameters across three benchmarks using \textbf{22×\times} fewer parameters; scaling to 8B further establishes new state-of-the-art results on MMAR, highlighting that representation quality is as critical as model scale in audio language modeling. Code and model weights are available at https://github.com/luckyerr/EntangleCodec.

Hui Li, Yangfan Gao, Junlin Shang +4

Fudan University

Audio Representation LearningNeural Audio CodecsAudio UnderstandingAudio-Language ModelsAudio TokenizationDiscrete Speech Representation Learning
AskPDF