Organizations: Idiap Research Institute, Switzerland · EPFL, Switzerland · Uniphore, U.S.A. · University of Zurich, Switzerland · Brno University of Technology, Czech Republic
Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.
Figures & tables
Figure 1: Overview of Abstract Compression for context-aware ASR. Prior conversational turns are represented by both transcript and audio. In our implementation, the audio from each prior turn is distilled into a fixed number of latent tokens, while transcript is retained explicitly. These compressed context representations are provided to the LLM alongside the current turn’s full-resolution audio tokens. This preserves part of the conversational context while reducing the cost of raw-context conditioning.
Configuration
N train
N decode
WER
Bias-WER
DefinedAI Test Set (In-domain)
Single-turn †
0
0
13.4
21.0
Single-turn
0
0
7.6
13.5
5
9.1
15.1
Multi-turn, Raw
R[1,10]
5
7.5
13.3
10
7.5
13.1
Table 1: ASR performance on the DefinedAI (in-domain) and WoW (out-of-domain) test sets. In the Configuration column, the first term denotes whether the model is trained in a single-turn or multi-turn setting, while the second denotes the type of context representation, i.e., raw or compressed. Single-turn † denotes the open-source Phi-4-Multimodal model without any fine-tuning. All other models are fine-tuned on the DefinedAI train split. N train and N decode denote the numbers of context turns used during training and testing, respectively. R[1,10] indicates random sampling of 1 to 10 raw-context turns during training, while 0→10 denotes the curriculum used for compressed-context training. WER is computed on the full transcript and Bias-WER on entity words only; lower is better for both.
Compression
Training
DefinedAI
WoW
Setting
Data
WER
WER
Full audio
DefinedAI (40h)
7.6
13.4
K=4
DefinedAI (40h)
51.2
72.3
K=8
DefinedAI (40h)
46.3
66.4
K=16
DefinedAI (40h)
37.1
54.2
K=16
LibriSpeech (960h)
21.2
29.1
Table 2: Stage 1 feasibility. Single-turn ASR with either full-resolution audio or compressed audio tokens. The row Full audio (no compression) uses the original audio-token sequence and serves as a reference. For the compressed rows, K denotes the number of latent audio tokens. The Phi-4-Multimodal model is frozen, and only the compression module is trained.
Figure 2: Compression rates as a function of context size on DefinedAI test set. Top: audio compression rate ρnaudio (see Eq. 8 ). Bottom: overall context compression rate ρncontext (see Eq. 9 ). Lines show the median across examples and shaded regions denote interquartile range.
Latent
Stage 1
N decode
WER
Bias-
Tokens ( K )
Train Data
WER
4
DefinedAI (40h)
0
13.5
26.6
1
13.2
26.0
5
13.3
25.2
10
13.4
25.3
8
DefinedAI (40h)
0
13.7
27.0
Table 3: Stage 2 ablations on the WoW test set. We vary the number of latent audio tokens K , the Stage 1 data used to train the compression module, and the number of context turns provided at inference time (N decode ). Results show that larger bottlenecks make compressed context more useful, while stronger Stage 1 single-turn alignment does not necessarily translate into better downstream contextual ASR.