Large audio-language models (LALMs) have become highly capable on a wide range of tasks spanning speech, music, and environmental sound. However, many modern LALMs resample input audio to 16 kHz mono, making all frequencies above 8 kHz unobservable. We present SpectralBridge, a lightweight extension that gives such a model access to the 8-16 kHz band by joining low- and upper-band representations with time-aligned paired cross-frequency attention. Applied to Audio Flamingo Next, the method trains only a small adapter and rank-8 LoRA modules, while leaving the original audio encoder, audio projector, and base language model frozen. SpectralBridge improves both zero-shot transfer and downstream fine-tuning: it raises zero-shot balanced accuracy on the ESC-50 audio classification dataset from 75.5 to 90.5, and when fine-tuned on Insect-10, a 10-way insect species task we construct from InsectSet459 recordings, under the same budget as a LoRA fine-tuned baseline, it reaches 55.4 balanced accuracy against the baseline's 26.2. On a benchmark where the answer depends only on the 8-16 kHz band, it gains 30 mAP over the frozen model. Erasing the upper-band stream from the trained model removes these gains, showing that they come from the high-frequency signal itself rather than from extra tokens or the adapted prompt format.
Figures & tables
Figure 1: Overview of SpectralBridge . The 32 kHz input is split into the stock low-band view and an 8–16 kHz view heterodyned into the encoder passband; both, plus a silence null stream, pass through the shared frozen AF-Whisper encoder. The trained adapter attends across the two per-timestamp frequency tokens and adds the gated result back to both streams; the frozen projector yields the 2N -token audio prefix for the frozen LLM with trained LoRA.
Model
BAcc
mAP
Paired rank
Frozen AF-Next
51.3±1.2
51.3±1.2
45.0±2.1
Low-band LoRA
51.2±1.1
51.4±0.9
45.2±1.8
Simple two-view LoRA
60.7±1.1
68.0±1.3
72.5±1.4
SpectralBridge paired 2N
69.7±1.0
81.4±0.8
80.0±1.1
SpectralBridge , high erased
49.7±1.3
49.4±1.4
45.6±2.2
Table 1: Constructed-FSD sealed test. Values are percentages with ± reporting SEM across seeds. The dataset was constructed to ensure that the task cannot be solved without using upper-band evidence, which is why the Low-band LoRA baseline performs at chance, as does the Frozen AF-Next.
Method
ESC-50 10-way
TinySOL 14-way
Frozen AF-Next
75.5±0.8
13.4±0.5
Low-band LoRA
83.8±0.6
27.9±0.4
Simple two-view LoRA
89.5±0.5
24.4±0.3
SpectralBridge paired 2N
90.5±0.4
28.6±0.3
SpectralBridge , high erased
86.8±0.7
24.3±0.4
Table 2: Real multiclass zero-shot benchmarks. Values are balanced accuracy; ± reports SEM across seeds.
Model
Dev BAcc
Low-band LoRA
26.2
SpectralBridge paired 2N
55.4
SpectralBridge paired 2N, high-erased
22.7
Table 3: Insect-10 downstream fine-tuning results on the 113-item test split (10-way species classification over 10-second excerpts of InsectSet459 validation audio). Values are balanced accuracy in percent, averaged across seeds, after 300 fine-tuning steps for every model.
Model
Test BAcc ( ± SEM)
Test mAP ( ± SEM)
Frozen AF-Next
55.91±0.34
64.36±0.72
Low-band LoRA
56.76±2.62
72.28±0.78
Simple two-view LoRA
58.40±2.24
74.93±0.88
Simple two-view LoRA, high-erased
56.82±3.19
74.11±1.00
SpectralBridge paired 2N
62.14±2.35
75.53±1.47
SpectralBridge paired 2N, high-erased
57.09±1.10
73.36±1.81
Table 4: CEB bird-pair downstream fine-tuning test results. Values are percentages; ± reports SEM across seeds.
Tokens
Method
BAcc
mAP
Rank
N
Frozen AF-Next
48.3±1.1
52.2±0.9
51.7±1.8
2N
Simple two-view LoRA
62.2±2.3
76.0±0.8
76.1±1.2
2N
SpectralBridge paired
71.9±1.5
83.3±0.6
80.6±1.4
N
Half-rate student
72.2±1.1
83.6±0.8
82.2±1.5
Table 5: Token-budget comparison on the constructed-FSD development split. Note that this is a different subset from the sealed test split in Table 1 , so absolute values are not directly comparable. Values are percentages; ± reports SEM across seeds where available.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Qualitative ESC-50 case study. Only the paired same-time low/high fusion model consistently selects the correct target class across three seeds. Nulling the high-frequency view changes the paired model’s prediction while keeping the checkpoint and token layout fixed.
Ablation
What we change
What it shows
Low-band LoRA
Train the same LoRA modules, but provide only the stock low-band audio tokens.
Separates answer-format and prompt adaptation from true high-frequency use.
Simple 2N
Concatenate low and translated-high streams without paired cross-band attention.
Tests whether merely exposing the upper band is enough.
High-erased control
Keep the same trained checkpoint and token layout, but replace the high stream by digital silence.
Measures the causal contribution of the 8–16 kHz waveform.
Paired residual disabled
Keep both streams but turn off the learned paired residual.
Isolates the value of timestamp-local cross-frequency interaction.
Exact- N compression
Reduce the wideband prefix from 2N to N tokens by half-rate pooling.
Tests whether the high-band gain requires doubling the audio-token budget.
No high-token block
Keep only the adapter-enriched low tokens and remove dedicated high-band slots.
Tests whether high evidence can be fully folded into the low stream.
Appendix
Table 6: Summary of the main ablations and what each one tests.
Large audio language models (LALMs) extend large language models with an audio encoder and large-scale audio data. However, the scarcity of high-quality annotated audio data remains a fundamental bottleneck for scaling. Through probing signal detectability analysis, we identify fine-grained spectrotemporal perceptual weaknesses in a foundation LALM. To address these challenges, we propose Spectrotemporal Counting (SpectCount), a data-efficient fine-tuning approach based on fully synthetic audio signals generated on-the-fly, without relying on real-world audio, annotations, or pretrained generative models. SpectCount not only resolves the observed weaknesses but also improves performance on diverse auditory benchmarks spanning sound, music, and speech, unseen during fine-tuning. These results suggest that weakness-targeted synthetic signals provide a data-efficient path toward enhanced auditory understanding capabilities in LALMs.
Seonuk Kim, Yonghyeon Jun, Ju Yeon Kang +3
Department of Electrical and Computer Engineering and INMC, Seoul National University, Seoul, South Korea
Large audio language models (LALMs) are a class of foundation models for audio understanding. Existing LALMs tend to degrade significantly in real-world noisy acoustic conditions where speech and non-speech sounds interfere. While noise-aware fine-tuning can improve robustness, it requires task-specific noisy data and expensive retraining, limiting scalability. To address this issue, we propose Focus-Then-Listen (FTL), a plug-and-play audio enhancer that improves LALMs' noise robustness. Specifically, FTL first separates the input waveform into speech and non-speech, and a modality router is applied to predict the target audio modality (e.g., speech) based on the user's instruction. Finally, a modality-aware fusion block generates a task-adaptive enhanced signal for improved downstream perception and reasoning. Experiments across multiple LALMs and tasks show that FTL improves performance across different noise levels without fine-tuning on LALMs.
Han Yin, Yang Xiao, Younghoo Kwon +2
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea · University of Melbourne, Australia
Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.
Neta Glazer, Lenny Aharon, Ethan Fetaya
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA