Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis

Pengfei Zhang; Tianxin Xie; Minghao Yang; Li Liu

eess.ASFeb 16, 2026

Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis

Authors: Pengfei Zhang, Tianxin Xie, Minghao Yang, Li Liu

Organizations: The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Deep learning-based respiratory auscultation is currently hindered by two fundamental challenges: (i) inherent information loss, as converting signals into spectrograms discards transient acoustic events and clinical context; (ii) limited data availability, exacerbated by severe class imbalance. To bridge these gaps, we present Resp-Agent, an autonomous multimodal system orchestrated by a novel Active Adversarial Curriculum Agent (Thinker-A $^2$ CA). Unlike static pipelines, Thinker-A $^2$ CA serves as a central controller that actively identifies diagnostic weaknesses and schedules targeted synthesis in a closed loop. To address the representation gap, we introduce a modality-weaving Diagnoser that weaves clinical text with audio tokens via strategic global attention and sparse audio anchors, capturing both long-range clinical context and millisecond-level transients. To address the data gap, we design a flow matching Generator that adapts a text-only Large Language Model (LLM) via modality injection, decoupling pathological content from acoustic style to synthesize hard-to-diagnose samples. As a foundation for this work, we introduce Resp-229k, a benchmark corpus of 229k recordings paired with LLM-distilled clinical narratives. Extensive experiments demonstrate that Resp-Agent consistently outperforms prior approaches across diverse evaluation settings, improving diagnostic robustness under data scarcity and long-tailed class imbalance. Our code and data are available at https://github.com/zpforlove/Resp-Agent.

Figures & tables

Figure 1: Overview of Resp-Agent. The framework functions as a closed-loop system composed of three interacting modules: (a) Thinker: A compute-aware planner (Thinker-A 2 CA) that parses semantic intents and routes tasks to other agents based on recycled error profiles and calibrated confidence. (b) Generator: A synthesis module utilizing modality injection to condition the Resp-MLLM on both textual diagnosis and reference acoustic style, decoding discrete units via conditional flow matching. (c) Diagnoser: A clinical inference module employing modality weaving to fuse EHR summaries with audio features early in the network, leveraging sparse global attention for robust cross-modal reasoning.

(a) Resp-229k split statistics (effective samples)
Split	#Files	Hours	Mean (s)	Max (s)
Train	196,654	341	6.2	86
Valid	16,931	31	6.6	71
Test	15,516	36	8.4	30
Total	229,101	408	6.4	86
(b) Source datasets for the curation of Resp-229k

Table 1: Resp-229k overview: split statistics and source datasets. The dataset identifiers correspond to UK COVID-19 ( Coppock et al., 2024 ; Budd et al., 2024 ; Pigoli et al., 2022 ) , ICBHI ( Rocha et al., 2017 ) , SPRSound ( Zhang et al., 2022 ) , COUGHVID ( Orlandic et al., 2021 ) , and KAUH ( Fraiwan et al., 2021 ) .

Figure 2: Detailed architecture of Resp-MLLM (Stage 1 of the Generator). The model functions as a style-conditioned multimodal unit generator. Top: A modality injection mechanism fuses textual diagnosis semantics with acoustic style embeddings (projected from temporally pooled BEATs features) to prompt the Qwen3-0.6B-Base backbone. Bottom: A leak-free conditioning strategy is employed during training: random mask sampling ( $\mathcal{M}\approx 10\%$ ) prevents the model from peeking at oracle tokens, ensuring robust autoregressive prediction of discrete acoustic units.

Figure 3: Diagnoser Architecture: Modality Weaving with Strategic Global Attention. The framework comprises three key mechanisms: (1) Input-Level Modality Weaving: EHR text tokens and projected audio features (extracted via BEATs ) are fused into a single token-aligned sequence at the input layer, enabling native cross-modal interaction. (2) Strategic Global Attention: A Longformer backbone combines efficient sliding-window attention with a sparse set of global tokens, which includes textual sentinels and distributed Audio Anchors to model long-range dependencies with linear complexity. (3) Audio Anchor Mechanism: Anchors act as cross-modal hubs spaced at $\approx$ 80ms intervals, allowing distinct text symptoms (e.g., “wheeze”) to directly query transient acoustic events, thereby capturing fine-grained temporal structures without quadratic computational costs.

Method	Backbone	Pretraining Data	$S_{p}$ (%)	$S_{e}$ (%)	Score (%)
SE+SA ( Yang et al., 2020 )	ResNet18	-	81.25	17.84	49.55
LungRN+NL ( Ma et al., 2020 )	ResNet - NL	-	63.20	41.32	52.26
RespireNet ( Gairola et al., 2021 )	ResNet34	IN	72.30	40.10	56.20
Chang et al. ( Chang et al., 2022 )	CNN8 - dilated	-	69.92	35.85	52.89
Ren et al. ( Ren et al., 2022 )	CNN8 - Pt	-	72.96	27.78	50.37
Wang et al. ( Wang and Wang, 2022 )	ResNeSt	IN	70.40	40.20	55.30

Table 2: RSC performance on the ICBHI dataset using the official 60–40% train–test split. In the Pretraining Data column, IN, AS, LA, HF, and SPR refer to ImageNet ( Deng et al., 2009 ) , AudioSet ( Gemmeke et al., 2017 ) , LAION - Audio - 630K ( Wu et al., 2023 ) , HF_Lung_V1 ( Hsu et al., 2021 ) , and SPRSound, respectively. * denotes the previous state-of-the-art ICBHI Score. The SOTA and second-best results are highlighted in bold and by underlining, respectively.

Planner policies under matched budget (Exp. 1)
Setting	Method	$B$ (k)	Acc	Macro-F1	Macro-F1 ${}_{\text{tail}}$	LoSO Macro-F1 / tail
Test-CD	No-Synth (CE)	0	0.849	0.212	0.074	0.237 / 0.086
Test-CD	Random	50	0.869	0.442	0.291	–
Test-CD	Class-Prior	50	0.876	0.512	0.349	0.473 / 0.334
Test-CD	Uncertainty-Static	50	0.881	0.546	0.376	–
Test-CD	Thinker-A 2 CA	50	0.887	0.598	0.421	0.532 / 0.383

Table 3: Summary of downstream Diagnoser performance on Test-CD under different planner policies and imbalance remedies. Macro-F1

{}_{\text{tail}}

is computed over the 8 rarest classes. “–” indicates that LoSO results are not reported for that setting due to the computational cost of retraining across all five folds.

Generator disentanglement (Exp. 6)
Test	Style-Sim $\uparrow$	P-Acc $\uparrow$	FAD $\downarrow$
Style-swap (avg over 4 styles)	0.91	97.9%	1.18
Content-swap (avg over 4 labels)	0.93	96.1%	1.19
Diagnoser ablations on Test-CD (Exp. 7)
Config	Acc	Macro-F1
Late Fusion, Raw Metadata, no anchors	0.780	0.145

Table 4: Compact summary of Generator content–style disentanglement

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Class name (unified)	Total count
Control Group	156,527
COVID-19	77,994
Pneumonia	1,909
COPD	820
Asthma	324
Bronchitis	188

Appendix

Table 5: Unified 16-class taxonomy and sample distribution for Resp-229k (raw

N=238{,}074

). The counts highlight the severe long-tail imbalance.

Model	Modality	Accuracy	Macro-F1
LSTM (main paper baseline)	Text	0.0912	0.0401
BERT-base	Text	0.1420	0.0710
RoBERTa-base	Text	0.1513	0.0742
Longformer-base (text-only)	Text	0.1585	0.0813
Conformer (audio-only)	Audio	0.7200	0.1935
Resp-Agent Diagnoser (Ours)	Audio + Text	0.8494	0.2118

Appendix

Table 6: Performance comparison of text-only, audio-only, and multimodal models on the cross-domain test set (original, imbalanced data). Text-only Transformers improve over LSTM but fail to match audio-only baselines, justifying the need for multimodal fusion.

Model / Fusion Strategy	Modality	Accuracy	Macro-F1
Conformer (audio-only, main paper)	Audio	0.7200	0.1935
Whisper-Small (audio-only)	Audio	0.7310	0.2010
Conformer + LSTM (Concat-MLP)	Audio + Text (Late)	0.8012	0.2003
Conformer + BERT (Concat-MLP)	Audio + Text (Late)	0.8124	0.2040
Conformer + BERT (Logit-Voting)	Audio + Text (Late)	0.8043	0.1992
Resp-Agent Diagnoser (Ours)	Audio + Text (Weaving)	0.8494	0.2118

Appendix

Table 7: Ablation on fusion strategies and audio encoder backbones (original, imbalanced data). Deep Modality Weaving outperforms shallow fusion methods, and the choice of fusion architecture outweighs marginal gains from changing the audio backbone.

	Accuracy		Macro-F1		$\Delta$ vs. Conformer (audio-only)
Model	Original	Balanced	Original	Balanced	$\Delta$ Acc (Orig.)	$\Delta$ Acc (Bal.)	$\Delta$ F1 (Orig.)	$\Delta$ F1 (Bal.)
LSTM (text-only)	0.0912	0.3020	0.0401	0.2140	-0.6288	-0.4800	-0.1534	-0.3220
Conformer (audio-only)	0.7200	0.7820	0.1935	0.5360	0.0000	0.0000	0.0000	0.0000
Longformer (no anchors)	0.6495	0.7630	0.1890	0.5200	-0.0705	-0.0190	-0.0045	-0.0160
Ours: Resp-Agent (multimodal)	0.8494	0.8870	0.2118	0.5980	+0.1294	+0.1050	+0.0183	+0.0620

Appendix

Table 8: Summary of performance on Resp-229k under the original (imbalanced) and class-balanced regimes. The substantial gains in the balanced regime highlight the efficacy of the Generator, while the multimodal improvements confirm the value of the Diagnoser’s architecture.

Training Data Strategy	Test Acc.	$\Delta$ Acc	F1-Macro	$\Delta$ F1
Original Imbalanced	0.7200	–	0.1935	–
naive Augmentation Balanced	0.6914	-0.0286	0.1688	-0.0247

Appendix

Table 9: Degradation of Cross-Domain Performance with naive Augmentation. The model trained on the balanced set created by traditional augmentation techniques performs worse on unseen test data than the model trained on the original imbalanced set, highlighting the failure of these methods to produce generalizable synthetic data.

Generative Model	Cosine Similarity $\uparrow$	FAD $\downarrow$
c-WaveGAN	$0.61\pm 0.15$	2.85
AudioLDM 2 (fine-tuned)	$0.76\pm 0.11$	1.92
StableAudio Open (fine-tuned)	$0.83\pm 0.08$	1.54
Resp-Agent (Ours)	$0.92\pm 0.04$	1.13

Appendix

Table 10: Objective evaluation of individualized audio reconstruction. Our Generator (Resp-Agent) achieves the highest style adherence (Cosine Similarity) and best perceptual fidelity (FAD), outperforming strong generative baselines including a fine-tuned StableAudio Open model.

Training Set Strategy	Accuracy	F1-Macro	Relative $\Delta$ F1 (vs. Imbalanced)
Original Imbalanced	0.8494	0.2118	–
naive Augmentation Balanced	0.7520	0.1720	$-0.0398$
c-WaveGAN Balanced	0.8650	0.4520	$+0.2402$
AudioLDM 2 Balanced	0.8781	0.5265	$+0.3147$
StableAudio Open Balanced	0.8830	0.5620	$+0.3502$
Resp-Agent Balanced	0.8870	0.5980	$+0.3862$

Appendix

Table 11: Performance on the cross-domain Test-CD set using different balanced training sets (multimodal Longformer Diagnoser). Balancing with Resp-Agent yields the largest improvement in Macro-F1 over the imbalanced baseline, even compared to a strong StableAudio Open baseline.

Training Set Strategy	Accuracy	F1-Macro
Original Imbalanced	0.7200	0.1935
naive Augmentation Balanced	0.6914	0.1688
c-WaveGAN Balanced	0.7420	0.4010
AudioLDM 2 Balanced	0.7560	0.4760
StableAudio Open Balanced	0.7700	0.5050
Resp-Agent Balanced	0.7820	0.5360

Appendix

Table 12: Performance on the cross-domain Test-CD set using different balanced training sets (unimodal Conformer). StableAudio Open improves substantially over older baselines, but the Resp-Agent Balanced regime remains strongest.

K (Style Tokens)	Similarity (Cosine) $\uparrow$	FAD $\downarrow$	Longformer F1-Macro $\uparrow$
0 (Style Disabled)	0.80 $\pm$ 0.09	1.52	0.542
2	0.85 $\pm$ 0.08	1.38	0.563
4	0.87 $\pm$ 0.06	1.29	0.577
8 (Default)	0.92 $\pm$ 0.04	1.13	0.591

Appendix

Table 13: Ablation Study on the Number of Style Tokens (K). Performance improves consistently with a larger style prefix, validating the effectiveness of our style conditioning mechanism.

Decoder Type	Steps	FAD $\downarrow$	Similarity $\uparrow$	Inference Latency
DDPM	32	1.31	0.90	1.0 $\times$
CFM (Ours)	32	1.13	0.92	$\approx$ 0.6 $\times$

Appendix

Table 14: Comparison of Decoder Paradigms. Conditional Flow-Matching (CFM) surpasses the traditional DDPM in both audio quality (FAD, Similarity) and inference speed.

Metric	Value
Total records	238,074
Heuristic suspicious ( EMPTY / OVERLONG / PROMPT_LEAK )	3,356
Randomly audited (heuristic-clean samples)	2,300
Total sent to QA LLM	5,656
LLM kept as OK	3,766
LLM rewrote (suspicious only)	1,774

Appendix

Table 15: Quantitative results of the two-stage text summarization QA pipeline (

N=238{,}074

). The low rewrite rate confirms the high fidelity of the initial synthesis.

Heuristic Category (Suspicious)		LLM-Identified Error Type (All Audited)
Type	Count	Error Type	Count
OVERLONG	3,031	OK	3,766
PROMPT_LEAK	300	OVERLONG_OR_PROMPT_LEAK	1,747
EMPTY_OR_TRUNCATED	25	OTHER_QUALITY_ISSUE	113
		EMPTY_OR_TRUNCATED	30

Appendix

Table 16: Breakdown of heuristic flags and LLM-identified error types during the audit. The primary errors were technical artifacts (e.g., prompt leaks) rather than clinical hallucinations.

Explore similar work

RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification
Jun 8, 2026Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang +2Healthcare Audio-Language Models
The Ohio State University · University of Southern California · University of Chicago
RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity
Feb 4, 2026Gaia A. Bertolino, Yuwei Zhang, Tong Xia +2Audio-Language Model Evaluation Audio QA
University of Cambridge · Cambridge, United Kingdom · Tsinghua University +3
RespFocus: Learning Temporal Evidence for Weakly Supervised Respiratory Sound Classification
Oct 11, 2026Yi Tian, Xiaohuan Chen, Yufei tang +5Respiratory Sound Classification Weakly Supervised Learning
Fuyao University of Science and Technology (fyust.edu.cn) · Xiamen University (xmu.edu.cn)

Jun 8, 2026J/K move · Enter open · S save

RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification

Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang, Shrikanth Narayanan+1

The Ohio State University · University of Southern California · University of Chicago

Healthcare Audio-Language Models Multimodal Foundation Models Multimodal Contrastive Learning Audio Classification Multimodal Learning+1

Open paper AskPDF arXiv

Respiratory diseases remain a leading cause of global mortality, where timely and accurate diagnosis is critical to improving patient outcomes and reducing healthcare burdens. While prior work has explored audio-based models for respiratory disease detection, such unimodal approaches often suffer from limited generalizability and diagnostic precision. In this paper, we propose RespiraMFM, a Multimodal Foundation Model that integrates respiratory sounds with patient medical history and symptoms to enhance diagnostic accuracy and disease detection capabilities. We introduce an effective contrastive alignment strategy for audio-text multimodal integration, allowing the model to learn better cross-modal representations between respiratory sounds and corresponding textual clinical information. We evaluate RespiraMFM across five major respiratory diseases using seven real-world datasets in both supervised fine-tuning and zero-shot settings, achieving a 9.15% improvement in AUROC on supervised tasks and a 20.98% gain on zero-shot tasks over existing baselines. These findings underscore the potential of our framework to advance early diagnosis and improve clinical decision-making in respiratory disease management.

Abstract

Explore similar work

RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification

RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity

RespFocus: Learning Temporal Evidence for Weakly Supervised Respiratory Sound Classification