cs.CLMar 27, 2026

Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR

Authors: Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Bañeras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, 

Organizations: Idiap Research Institute, Switzerland · EPFL, Switzerland · Uniphore, U.S.A. · University of Zurich, Switzerland · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.

Figures & tables

Figure 1: Overview of Abstract Compression for context-aware ASR. Prior conversational turns are represented by both transcript and audio. In our implementation, the audio from each prior turn is distilled into a fixed number of latent tokens, while transcript is retained explicitly. These compressed context representations are provided to the LLM alongside the current turn’s full-resolution audio tokens. This preserves part of the conversational context while reducing the cost of raw-context conditioning.
ConfigurationN train{}_{\text{train}}N decode{}_{\text{decode}}WERBias-WER
DefinedAI Test Set (In-domain)
Single-turn †0013.421.0
Single-turn007.613.5
59.115.1
Multi-turn, RawR[1,10]\mathcal{R}_{[1,10]}57.513.3
107.513.1
Table 1: ASR performance on the DefinedAI (in-domain) and WoW (out-of-domain) test sets. In the Configuration column, the first term denotes whether the model is trained in a single-turn or multi-turn setting, while the second denotes the type of context representation, i.e., raw or compressed. Single-turn † denotes the open-source Phi-4-Multimodal model without any fine-tuning. All other models are fine-tuned on the DefinedAI train split. N train{}_{\text{train}} and N decode{}_{\text{decode}} denote the numbers of context turns used during training and testing, respectively. R[1,10]\mathcal{R}_{[1,10]} indicates random sampling of 1 to 10 raw-context turns during training, while 0 ⁣→ ⁣100\!\to\!10 denotes the curriculum used for compressed-context training. WER is computed on the full transcript and Bias-WER on entity words only; lower is better for both.
CompressionTrainingDefinedAIWoW
SettingDataWERWER
Full audioDefinedAI (40h)7.613.4
K=4K=4DefinedAI (40h)51.272.3
K=8K=8DefinedAI (40h)46.366.4
K=16K=16DefinedAI (40h)37.154.2
K=16K=16LibriSpeech (960h)21.229.1
Table 2: Stage 1 feasibility. Single-turn ASR with either full-resolution audio or compressed audio tokens. The row Full audio (no compression) uses the original audio-token sequence and serves as a reference. For the compressed rows, KK denotes the number of latent audio tokens. The Phi-4-Multimodal model is frozen, and only the compression module is trained.
Figure 2: Compression rates as a function of context size on DefinedAI test set. Top: audio compression rate ρnaudio\rho_{n}^{\text{audio}} (see Eq. 8 ). Bottom: overall context compression rate ρncontext\rho_{n}^{\text{context}} (see Eq. 9 ). Lines show the median across examples and shaded regions denote interquartile range.
LatentStage 1N decode{}_{\text{decode}}WERBias-
Tokens ( KK )Train DataWER
4DefinedAI (40h)013.526.6
113.226.0
513.325.2
1013.425.3
8DefinedAI (40h)013.727.0
Table 3: Stage 2 ablations on the WoW test set. We vary the number of latent audio tokens KK , the Stage 1 data used to train the compression module, and the number of context turns provided at inference time (N decode{}_{\text{decode}} ). Results show that larger bottlenecks make compressed context more useful, while stronger Stage 1 single-turn alignment does not necessarily translate into better downstream contextual ASR.
More in this topic

Contextual ASR

See all 19 papers

Explore similar work

CardsList
Apr 1, 2026

Speech LLMs are Contextual Reasoning Transcribers

Despite extensions to speech inputs, effectively leveraging the rich knowledge and contextual understanding of large language models (LLMs) in automatic speech recognition (ASR) remains non-trivial, as the task primarily involves direct speech-to-text mapping. To address this, this paper proposes chain-of-thought ASR (CoT-ASR), which constructs a reasoning chain that enables LLMs to first analyze the input speech and generate contextual analysis, thereby fully exploiting their generative capabilities. With this contextual reasoning, CoT-ASR then performs more informed speech recognition and completes both reasoning and transcription in a single pass. Moreover, CoT-ASR naturally supports user-guided transcription: while designed to self-generate reasoning, it can also seamlessly incorporate user-provided context to guide transcription, further extending ASR functionality. To reduce the modality gap, this paper introduces a CTC-guided Modality Adapter, which uses CTC non-blank token probabilities to weight LLM embeddings, efficiently aligning speech encoder outputs with the LLM's textual latent space. Experiments show that, compared to standard LLM-based ASR, CoT-ASR achieves a relative reduction of 8.7% in word error rate (WER) and 16.9% in entity error rate (EER).

Keqi Deng, Ruchao Fan, Bo Ren +2

Microsoft Core AI, USA

Contextual ASRAutomatic Speech RecognitionSpeech Language Models
AskPDF
May 17, 2026

Compress the Context, Keep the Commitments: A Formal Framework for Verifiable LLM Context Compression

LLM context is not just tokens; it is a set of commitments. Long-running conversations accumulate goals, constraints, decisions, preferences, tool results, retrieved evidence, artifacts, and safety boundaries that future responses must preserve. Existing context-management methods reduce length through truncation, retrieval, summarization, memory systems, or token-level prompt compression, but they rarely specify which semantic commitments must survive compression or how their preservation should be measured. We propose Context Codec, a commitment-level framework for compressing prompts and chat histories. Context Codec represents dialogue state as typed, source-grounded semantic atoms with canonical identity, equivalence, conflict, confidence, risk, and evidence spans. It separates five concerns - extraction, normalization, representation, rendering, and verification - and introduces metrics for Critical Atom Recall, Weighted Atom Recall, Commitment Density, and round-trip recoverability. It also defines a taxonomy of semantic compression errors, a concrete normalization procedure, conservative fallback rules for low-confidence and safety-critical atoms, and Context Compression Language (CCL), an ASCII-first compact rendering of canonical JSON atoms. In a small diagnostic study, CCL-Core occupies a useful middle ground between structured prose and JSON: more explicit and auditable than prose, usually more compact than JSON, and less risky than heavily minified notation. The result is not a claim that shorthand solves compression, but a framework for making context compression verifiable: compress the conversation, keep the commitments.

Natalia Trukhina, Vadim Vashkelis

Embedded Intelligence Lab (EMILAB)

LLM CompressionLLM Context CompressionLLM Context Management
AskPDF
Sep 30, 2026

Audio Token Attention Is Predictable Before the Language Model Runs

A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at ρ≥.69ρ\geq .69 on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: https://audio-triage.github.io

Kyoungjun Park, Yunzhe Li, Lili Qiu

The University of Texas at Austin

Audio Token CompressionAudio UnderstandingAudio-Language ModelsToken PruningLLM Inference AccelerationToken Compression
AskPDF