SpectralBridge: Adding High-Frequency Capability to Large Audio-Language Models

Sergei Polezhaev; Saba Anjum; Michael Gormish

cs.SDOct 12, 2026

SpectralBridge: Adding High-Frequency Capability to Large Audio-Language Models

Authors: Sergei Polezhaev, Saba Anjum, Michael Gormish

Organizations: Nebius AI

Abstract

Large audio-language models (LALMs) have become highly capable on a wide range of tasks spanning speech, music, and environmental sound. However, many modern LALMs resample input audio to 16 kHz mono, making all frequencies above 8 kHz unobservable. We present SpectralBridge, a lightweight extension that gives such a model access to the 8-16 kHz band by joining low- and upper-band representations with time-aligned paired cross-frequency attention. Applied to Audio Flamingo Next, the method trains only a small adapter and rank-8 LoRA modules, while leaving the original audio encoder, audio projector, and base language model frozen. SpectralBridge improves both zero-shot transfer and downstream fine-tuning: it raises zero-shot balanced accuracy on the ESC-50 audio classification dataset from 75.5 to 90.5, and when fine-tuned on Insect-10, a 10-way insect species task we construct from InsectSet459 recordings, under the same budget as a LoRA fine-tuned baseline, it reaches 55.4 balanced accuracy against the baseline's 26.2. On a benchmark where the answer depends only on the 8-16 kHz band, it gains 30 mAP over the frozen model. Erasing the upper-band stream from the trained model removes these gains, showing that they come from the high-frequency signal itself rather than from extra tokens or the adapted prompt format.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of SpectralBridge . The 32 kHz input is split into the stock low-band view and an 8–16 kHz view heterodyned into the encoder passband; both, plus a silence null stream, pass through the shared frozen AF-Whisper encoder. The trained adapter attends across the two per-timestamp frequency tokens and adds the gated result back to both streams; the frozen projector yields the $2N$ -token audio prefix for the frozen LLM with trained LoRA.

Model	BAcc	mAP	Paired rank
Frozen AF-Next	$51.3\pm 1.2$	$51.3\pm 1.2$	$45.0\pm 2.1$
Low-band LoRA	$51.2\pm 1.1$	$51.4\pm 0.9$	$45.2\pm 1.8$
Simple two-view LoRA	$60.7\pm 1.1$	$68.0\pm 1.3$	$72.5\pm 1.4$
SpectralBridge paired 2N	$\mathbf{69.7\pm 1.0}$	$\mathbf{81.4\pm 0.8}$	$\mathbf{80.0\pm 1.1}$
SpectralBridge , high erased	$49.7\pm 1.3$	$49.4\pm 1.4$	$45.6\pm 2.2$

Table 1: Constructed-FSD sealed test. Values are percentages with

\pm

reporting SEM across seeds. The dataset was constructed to ensure that the task cannot be solved without using upper-band evidence, which is why the Low-band LoRA baseline performs at chance, as does the Frozen AF-Next.

Method	ESC-50 10-way	TinySOL 14-way
Frozen AF-Next	$75.5\pm 0.8$	$13.4\pm 0.5$
Low-band LoRA	$83.8\pm 0.6$	$27.9\pm 0.4$
Simple two-view LoRA	$89.5\pm 0.5$	$24.4\pm 0.3$
SpectralBridge paired 2N	$\mathbf{90.5\pm 0.4}$	$\mathbf{28.6\pm 0.3}$
SpectralBridge , high erased	$86.8\pm 0.7$	$24.3\pm 0.4$

Table 2: Real multiclass zero-shot benchmarks. Values are balanced accuracy;

\pm

reports SEM across seeds.

Model	Dev BAcc
Low-band LoRA	$26.2$
SpectralBridge paired 2N	$\mathbf{55.4}$
SpectralBridge paired 2N, high-erased	$22.7$

Table 3: Insect-10 downstream fine-tuning results on the 113-item test split (10-way species classification over 10-second excerpts of InsectSet459 validation audio). Values are balanced accuracy in percent, averaged across seeds, after 300 fine-tuning steps for every model.

Model	Test BAcc ( $\pm$ SEM)	Test mAP ( $\pm$ SEM)
Frozen AF-Next	$55.91\pm 0.34$	$64.36\pm 0.72$
Low-band LoRA	$56.76\pm 2.62$	$72.28\pm 0.78$
Simple two-view LoRA	$58.40\pm 2.24$	$74.93\pm 0.88$
Simple two-view LoRA, high-erased	$56.82\pm 3.19$	$74.11\pm 1.00$
SpectralBridge paired 2N	$\mathbf{62.14\pm 2.35}$	$\mathbf{75.53\pm 1.47}$
SpectralBridge paired 2N, high-erased	$57.09\pm 1.10$	$73.36\pm 1.81$

Table 4: CEB bird-pair downstream fine-tuning test results. Values are percentages;

\pm

reports SEM across seeds.

Tokens	Method	BAcc	mAP	Rank
$N$	Frozen AF-Next	$48.3\pm 1.1$	$52.2\pm 0.9$	$51.7\pm 1.8$
$2N$	Simple two-view LoRA	$62.2\pm 2.3$	$76.0\pm 0.8$	$76.1\pm 1.2$
$2N$	SpectralBridge paired	$71.9\pm 1.5$	$83.3\pm 0.6$	$80.6\pm 1.4$
$N$	Half-rate student	$\mathbf{72.2\pm 1.1}$	$\mathbf{83.6\pm 0.8}$	$\mathbf{82.2\pm 1.5}$

Table 5: Token-budget comparison on the constructed-FSD development split. Note that this is a different subset from the sealed test split in Table 1 , so absolute values are not directly comparable. Values are percentages;

\pm

reports SEM across seeds where available.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Qualitative ESC-50 case study. Only the paired same-time low/high fusion model consistently selects the correct target class across three seeds. Nulling the high-frequency view changes the paired model’s prediction while keeping the checkpoint and token layout fixed.

Ablation	What we change	What it shows
Low-band LoRA	Train the same LoRA modules, but provide only the stock low-band audio tokens.	Separates answer-format and prompt adaptation from true high-frequency use.
Simple 2N	Concatenate low and translated-high streams without paired cross-band attention.	Tests whether merely exposing the upper band is enough.
High-erased control	Keep the same trained checkpoint and token layout, but replace the high stream by digital silence.	Measures the causal contribution of the 8–16 kHz waveform.
Paired residual disabled	Keep both streams but turn off the learned paired residual.	Isolates the value of timestamp-local cross-frequency interaction.
Exact- $N$ compression	Reduce the wideband prefix from $2N$ to $N$ tokens by half-rate pooling.	Tests whether the high-band gain requires doubling the audio-token budget.
No high-token block	Keep only the adapter-enriched low tokens and remove dedicated high-band slots.	Tests whether high evidence can be fully folded into the low stream.

Appendix

Table 6: Summary of the main ablations and what each one tests.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Models
Jun 5, 2026Seonuk Kim, Yonghyeon Jun, Ju Yeon Kang +3LLM Fine-Tuning Audio Understanding
Department of Electrical and Computer Engineering and INMC, Seoul National University, Seoul, South Korea
Focus Then Listen: An Empirical Study of Plug-and-Play Audio Enhancer for Noise-Robust Large Audio Language Models
Mar 5, 2026Han Yin, Yang Xiao, Younghoo Kwon +2Audio Understanding Audio-Language Models
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea · University of Melbourne, Australia
Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering
Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language Models Audio-Language Model Evaluation
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA

Jun 5, 2026J/K move · Enter open · S save

SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Models

Seonuk Kim, Yonghyeon Jun, Ju Yeon Kang, Jimin Hong+2

Department of Electrical and Computer Engineering and INMC, Seoul National University, Seoul, South Korea

LLM Fine-Tuning Audio Understanding Audio-Language Models

Open paper AskPDF arXiv

Large audio language models (LALMs) extend large language models with an audio encoder and large-scale audio data. However, the scarcity of high-quality annotated audio data remains a fundamental bottleneck for scaling. Through probing signal detectability analysis, we identify fine-grained spectrotemporal perceptual weaknesses in a foundation LALM. To address these challenges, we propose Spectrotemporal Counting (SpectCount), a data-efficient fine-tuning approach based on fully synthetic audio signals generated on-the-fly, without relying on real-world audio, annotations, or pretrained generative models. SpectCount not only resolves the observed weaknesses but also improves performance on diverse auditory benchmarks spanning sound, music, and speech, unseen during fine-tuning. These results suggest that weakness-targeted synthetic signals provide a data-efficient path toward enhanced auditory understanding capabilities in LALMs.

Abstract

Audio-Language Models

Explore similar work

SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Models

Focus Then Listen: An Empirical Study of Plug-and-Play Audio Enhancer for Noise-Robust Large Audio Language Models

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering