Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment

Mustafa Talha İlerisoy; Hung Manh Pham; Mathias Funk; Mykola Pechenizkiy; Aaqib Saeed

cs.CLAug 30, 2026

Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment

Authors: Mustafa Talha İlerisoy, Hung Manh Pham, Mathias Funk, Mykola Pechenizkiy, Aaqib Saeed

Organizations: Eindhoven University of Technology, Eindhoven, Netherlands · Singapore Management University, Singapore

Abstract

Self-supervised respiratory encoders lack semantic grounding in clinical domain needed for zero-shot inference, limiting their utility without task-specific labeled data. We propose a framework that aligns these encoders with medical terminology in a shared latent space turning them into a zero-shot-capable foundation model. To address paired data scarcity, we use a medical LLM to synthesize structured reports from metadata, creating dense semantic anchors for contrastive learning. Our training combines a sigmoid-based contrastive loss with encoder's native SSL objective and similarity-aware negative sampling to sharpen pathological boundaries. Across 9 tasks on 6 datasets, our method achieves a 61.3% mean zero-shot AUC, surpassing CLAP (51.4%) and Qwen2-Audio (54.9%) while reaching the highest linear probing AUC (71.6%) with only 43% of data used by full-scale baselines, showing that structured semantic alignment outperforms large-scale, general-purpose models in clinical diagnostics.

Figures & tables

Figure 1: Overview of REACH: a semantic alignment framework that repurposes pre-trained unimodal encoders for zero-shot respiratory sound classification via LLM-augmented report synthesis, similarity-aware negative sampling, and structure-preserving contrastive alignment.

Dataset	ID	Task	Modality	#Samples
CovidUK [ 25 ]	T1	Covid/Non-covid	Exhalation	840 / 1660
	T2	Covid/Non-covid	Cough	840 / 1660
COUGHVID [ 26 ]	T3	Covid/Non-covid	Cough	547 / 5628
	T4	Female/Male	Cough	2468 / 4795
ICBHI [ 27 ]	T5	COPD/Healthy	Lung sounds	793 / 35
Coswara [ 28 ]	T6	Smoker/Non-smoker	Cough	201 / 747

Table 1: Summary of curated datasets; shaded rows denote OOD datasets withheld from pre-training and alignment.

Linear Probing $\uparrow$
	In Domain					OOD
	T1	T2	T3	T4	T5	T6	T7	T8	T9	Avg
CLAP	56.5	64.8	59.9	66.5	93.3	68.0	74.2	69.7	63.6	68.5
OpenSMILE	55.0	64.9	53.7	67.7	57.9	53.4	75.3	63.6	49.4	60.1
VGGish	58.0	55.7	53.8	60.0	60.5	50.7	60.6	60.5	59.0	57.6
AudioMAE	54.9	61.6	55.4	62.8	88.6	54.9	72.4	61.6	51.0	62.6

Table 2: Benchmark results (% AUC). † : trained on open access data only (

\approx

43%). Shaded : full corpus, excluded from ranking.

\Delta

: gain over OGT † . Bold and underline indicates the best and the second best respectively.

Configuration	Lin. Probe $\uparrow$	$k$ - NN $\uparrow$	Zero Shot $\uparrow$
Audio Encoder from Scratch	66.0	58.8	55.1
Random Negatives	70.6	63.9	53.5
No $\mathcal{L}_{\text{MSE}}$	71.1	62.8	54.1
Frozen Audio Backbone	66.9	58.5	48.8
BERT Text Encoder	65.1	59.6	49.9
Negative Sampling K=100	69.9	61.1	54.2

Table 3: Ablation results (mean % AUC across 9 tasks).

Explore similar work

Adaptive Test-Time Scaling for Zero-Shot Respiratory Audio Classification
Apr 14, 2026Tsai-Ning Wang, Herman Teun den Dekker, Lin-Lin Chen +2Respiratory Sound Classification Test-Time Scaling
Eindhoven Artificial Intelligence Systems Institute, The Netherlands
RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification
Jun 8, 2026Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang +2Healthcare Audio-Language Models
The Ohio State University · University of Southern California · University of Chicago
ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification
Apr 20, 2026Florian Kittler, Sheethal Bhat, Andreas MaierVLM Robustness Chest X-Ray Classification
Friedrich-Alexander University Erlangen-Nuremberg, Erlangen 91054, Germany

Apr 14, 2026J/K move · Enter open · S save

Adaptive Test-Time Scaling for Zero-Shot Respiratory Audio Classification

Tsai-Ning Wang, Herman Teun den Dekker, Lin-Lin Chen, Neil Zeghidour+1

Eindhoven Artificial Intelligence Systems Institute, The Netherlands

Respiratory Sound Classification Test-Time Scaling

Open paper AskPDF arXiv

Automated respiratory audio analysis promises scalable, non-invasive disease screening, yet progress is limited by scarce labeled data and costly expert annotation. Zero-shot inference eliminates task-specific supervision, but existing methods apply uniform computation to every input regardless of difficulty. We introduce TRIAGE, a tiered zero-shot framework that adaptively scales test-time compute by routing each audio sample through progressively richer reasoning stages: fast label-cosine scoring in a joint audio-text embedding space (Tier-L), structured matching with clinician-style descriptors (Tier-M), and retrieval-augmented large language model reasoning (Tier-H). A confidence-based router finalizes easy predictions early while allocating additional computation to ambiguous inputs, enabling nearly half of all samples to exit at the cheapest tier. Across nine respiratory classification tasks without task-specific training, TRIAGE achieves a mean AUROC of 0.744, outperforming prior zero-shot methods and matching or exceeding supervised baselines on multiple tasks. Our analysis show that test-time scaling concentrates gains where they matter: uncertain cases see up to 19% relative improvement while confident predictions remain unchanged at minimal cost.

Abstract

Explore similar work

Adaptive Test-Time Scaling for Zero-Shot Respiratory Audio Classification

RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification

ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification