Deep learning-based respiratory auscultation is currently hindered by two fundamental challenges: (i) inherent information loss, as converting signals into spectrograms discards transient acoustic events and clinical context; (ii) limited data availability, exacerbated by severe class imbalance. To bridge these gaps, we present Resp-Agent, an autonomous multimodal system orchestrated by a novel Active Adversarial Curriculum Agent (Thinker-A2CA). Unlike static pipelines, Thinker-A2CA serves as a central controller that actively identifies diagnostic weaknesses and schedules targeted synthesis in a closed loop. To address the representation gap, we introduce a modality-weaving Diagnoser that weaves clinical text with audio tokens via strategic global attention and sparse audio anchors, capturing both long-range clinical context and millisecond-level transients. To address the data gap, we design a flow matching Generator that adapts a text-only Large Language Model (LLM) via modality injection, decoupling pathological content from acoustic style to synthesize hard-to-diagnose samples. As a foundation for this work, we introduce Resp-229k, a benchmark corpus of 229k recordings paired with LLM-distilled clinical narratives. Extensive experiments demonstrate that Resp-Agent consistently outperforms prior approaches across diverse evaluation settings, improving diagnostic robustness under data scarcity and long-tailed class imbalance. Our code and data are available at https://github.com/zpforlove/Resp-Agent.
Figures & tables
Figure 1: Overview of Resp-Agent. The framework functions as a closed-loop system composed of three interacting modules: (a) Thinker: A compute-aware planner (Thinker-A 2 CA) that parses semantic intents and routes tasks to other agents based on recycled error profiles and calibrated confidence. (b) Generator: A synthesis module utilizing modality injection to condition the Resp-MLLM on both textual diagnosis and reference acoustic style, decoding discrete units via conditional flow matching. (c) Diagnoser: A clinical inference module employing modality weaving to fuse EHR summaries with audio features early in the network, leveraging sparse global attention for robust cross-modal reasoning.
Table 1: Resp-229k overview: split statistics and source datasets. The dataset identifiers correspond to UK COVID-19 ( Coppock et al., 2024 ; Budd et al., 2024 ; Pigoli et al., 2022 ) , ICBHI ( Rocha et al., 2017 ) , SPRSound ( Zhang et al., 2022 ) , COUGHVID ( Orlandic et al., 2021 ) , and KAUH ( Fraiwan et al., 2021 ) .
Figure 2: Detailed architecture of Resp-MLLM (Stage 1 of the Generator). The model functions as a style-conditioned multimodal unit generator. Top: A modality injection mechanism fuses textual diagnosis semantics with acoustic style embeddings (projected from temporally pooled BEATs features) to prompt the Qwen3-0.6B-Base backbone. Bottom: A leak-free conditioning strategy is employed during training: random mask sampling ( M≈10% ) prevents the model from peeking at oracle tokens, ensuring robust autoregressive prediction of discrete acoustic units.
Figure 3: Diagnoser Architecture: Modality Weaving with Strategic Global Attention. The framework comprises three key mechanisms: (1) Input-Level Modality Weaving: EHR text tokens and projected audio features (extracted via BEATs ) are fused into a single token-aligned sequence at the input layer, enabling native cross-modal interaction. (2) Strategic Global Attention: A Longformer backbone combines efficient sliding-window attention with a sparse set of global tokens, which includes textual sentinels and distributed Audio Anchors to model long-range dependencies with linear complexity. (3) Audio Anchor Mechanism: Anchors act as cross-modal hubs spaced at ≈ 80ms intervals, allowing distinct text symptoms (e.g., “wheeze”) to directly query transient acoustic events, thereby capturing fine-grained temporal structures without quadratic computational costs.
Method
Backbone
Pretraining Data
Sp (%)
Se (%)
Score (%)
SE+SA ( Yang et al., 2020 )
ResNet18
-
81.25
17.84
49.55
LungRN+NL ( Ma et al., 2020 )
ResNet - NL
-
63.20
41.32
52.26
RespireNet ( Gairola et al., 2021 )
ResNet34
IN
72.30
40.10
56.20
Chang et al. ( Chang et al., 2022 )
CNN8 - dilated
-
69.92
35.85
52.89
Ren et al. ( Ren et al., 2022 )
CNN8 - Pt
-
72.96
27.78
50.37
Wang et al. ( Wang and Wang, 2022 )
ResNeSt
IN
70.40
40.20
55.30
Table 2: RSC performance on the ICBHI dataset using the official 60–40% train–test split. In the Pretraining Data column, IN, AS, LA, HF, and SPR refer to ImageNet ( Deng et al., 2009 ) , AudioSet ( Gemmeke et al., 2017 ) , LAION - Audio - 630K ( Wu et al., 2023 ) , HF_Lung_V1 ( Hsu et al., 2021 ) , and SPRSound, respectively. * denotes the previous state-of-the-art ICBHI Score. The SOTA and second-best results are highlighted in bold and by underlining, respectively.
Setting
Method
B (k)
Acc
Macro-F1
Macro-F1 tail
LoSO Macro-F1 / tail
Planner policies under matched budget (Exp. 1)
Test-CD
No-Synth (CE)
0
0.849
0.212
0.074
0.237 / 0.086
Test-CD
Random
50
0.869
0.442
0.291
–
Test-CD
Class-Prior
50
0.876
0.512
0.349
0.473 / 0.334
Test-CD
Uncertainty-Static
50
0.881
0.546
0.376
–
Test-CD
Thinker-A 2 CA
50
0.887
0.598
0.421
0.532 / 0.383
Table 3: Summary of downstream Diagnoser performance on Test-CD under different planner policies and imbalance remedies. Macro-F1 tail is computed over the 8 rarest classes. “–” indicates that LoSO results are not reported for that setting due to the computational cost of retraining across all five folds.
Generator disentanglement (Exp. 6)
Test
Style-Sim ↑
P-Acc ↑
FAD ↓
Style-swap (avg over 4 styles)
0.91
97.9%
1.18
Content-swap (avg over 4 labels)
0.93
96.1%
1.19
Diagnoser ablations on Test-CD (Exp. 7)
Config
Acc
Macro-F1
Late Fusion, Raw Metadata, no anchors
0.780
0.145
Table 4: Compact summary of Generator content–style disentanglement
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Class name (unified)
Total count
Control Group
156,527
COVID-19
77,994
Pneumonia
1,909
COPD
820
Asthma
324
Bronchitis
188
Appendix
Table 5: Unified 16-class taxonomy and sample distribution for Resp-229k (raw N=238,074 ). The counts highlight the severe long-tail imbalance.
Model
Modality
Accuracy
Macro-F1
LSTM (main paper baseline)
Text
0.0912
0.0401
BERT-base
Text
0.1420
0.0710
RoBERTa-base
Text
0.1513
0.0742
Longformer-base (text-only)
Text
0.1585
0.0813
Conformer (audio-only)
Audio
0.7200
0.1935
Resp-Agent Diagnoser (Ours)
Audio + Text
0.8494
0.2118
Appendix
Table 6: Performance comparison of text-only, audio-only, and multimodal models on the cross-domain test set (original, imbalanced data). Text-only Transformers improve over LSTM but fail to match audio-only baselines, justifying the need for multimodal fusion.
Model / Fusion Strategy
Modality
Accuracy
Macro-F1
Conformer (audio-only, main paper)
Audio
0.7200
0.1935
Whisper-Small (audio-only)
Audio
0.7310
0.2010
Conformer + LSTM (Concat-MLP)
Audio + Text (Late)
0.8012
0.2003
Conformer + BERT (Concat-MLP)
Audio + Text (Late)
0.8124
0.2040
Conformer + BERT (Logit-Voting)
Audio + Text (Late)
0.8043
0.1992
Resp-Agent Diagnoser (Ours)
Audio + Text (Weaving)
0.8494
0.2118
Appendix
Table 7: Ablation on fusion strategies and audio encoder backbones (original, imbalanced data). Deep Modality Weaving outperforms shallow fusion methods, and the choice of fusion architecture outweighs marginal gains from changing the audio backbone.
Accuracy
Macro-F1
Δ vs. Conformer (audio-only)
Model
Original
Balanced
Original
Balanced
Δ Acc (Orig.)
Δ Acc (Bal.)
Δ F1 (Orig.)
Δ F1 (Bal.)
LSTM (text-only)
0.0912
0.3020
0.0401
0.2140
-0.6288
-0.4800
-0.1534
-0.3220
Conformer (audio-only)
0.7200
0.7820
0.1935
0.5360
0.0000
0.0000
0.0000
0.0000
Longformer (no anchors)
0.6495
0.7630
0.1890
0.5200
-0.0705
-0.0190
-0.0045
-0.0160
Ours: Resp-Agent (multimodal)
0.8494
0.8870
0.2118
0.5980
+0.1294
+0.1050
+0.0183
+0.0620
Appendix
Table 8: Summary of performance on Resp-229k under the original (imbalanced) and class-balanced regimes. The substantial gains in the balanced regime highlight the efficacy of the Generator, while the multimodal improvements confirm the value of the Diagnoser’s architecture.
Training Data Strategy
Test Acc.
Δ Acc
F1-Macro
Δ F1
Original Imbalanced
0.7200
–
0.1935
–
naive Augmentation Balanced
0.6914
-0.0286
0.1688
-0.0247
Appendix
Table 9: Degradation of Cross-Domain Performance with naive Augmentation. The model trained on the balanced set created by traditional augmentation techniques performs worse on unseen test data than the model trained on the original imbalanced set, highlighting the failure of these methods to produce generalizable synthetic data.
Generative Model
Cosine Similarity ↑
FAD ↓
c-WaveGAN
0.61±0.15
2.85
AudioLDM 2 (fine-tuned)
0.76±0.11
1.92
StableAudio Open (fine-tuned)
0.83±0.08
1.54
Resp-Agent (Ours)
0.92±0.04
1.13
Appendix
Table 10: Objective evaluation of individualized audio reconstruction. Our Generator (Resp-Agent) achieves the highest style adherence (Cosine Similarity) and best perceptual fidelity (FAD), outperforming strong generative baselines including a fine-tuned StableAudio Open model.
Training Set Strategy
Accuracy
F1-Macro
Relative Δ F1 (vs. Imbalanced)
Original Imbalanced
0.8494
0.2118
–
naive Augmentation Balanced
0.7520
0.1720
−0.0398
c-WaveGAN Balanced
0.8650
0.4520
+0.2402
AudioLDM 2 Balanced
0.8781
0.5265
+0.3147
StableAudio Open Balanced
0.8830
0.5620
+0.3502
Resp-Agent Balanced
0.8870
0.5980
+0.3862
Appendix
Table 11: Performance on the cross-domain Test-CD set using different balanced training sets (multimodal Longformer Diagnoser). Balancing with Resp-Agent yields the largest improvement in Macro-F1 over the imbalanced baseline, even compared to a strong StableAudio Open baseline.
Training Set Strategy
Accuracy
F1-Macro
Original Imbalanced
0.7200
0.1935
naive Augmentation Balanced
0.6914
0.1688
c-WaveGAN Balanced
0.7420
0.4010
AudioLDM 2 Balanced
0.7560
0.4760
StableAudio Open Balanced
0.7700
0.5050
Resp-Agent Balanced
0.7820
0.5360
Appendix
Table 12: Performance on the cross-domain Test-CD set using different balanced training sets (unimodal Conformer). StableAudio Open improves substantially over older baselines, but the Resp-Agent Balanced regime remains strongest.
K (Style Tokens)
Similarity (Cosine) ↑
FAD ↓
Longformer F1-Macro ↑
0 (Style Disabled)
0.80 ± 0.09
1.52
0.542
2
0.85 ± 0.08
1.38
0.563
4
0.87 ± 0.06
1.29
0.577
8 (Default)
0.92 ± 0.04
1.13
0.591
Appendix
Table 13: Ablation Study on the Number of Style Tokens (K). Performance improves consistently with a larger style prefix, validating the effectiveness of our style conditioning mechanism.
Decoder Type
Steps
FAD ↓
Similarity ↑
Inference Latency
DDPM
32
1.31
0.90
1.0 ×
CFM (Ours)
32
1.13
0.92
≈ 0.6 ×
Appendix
Table 14: Comparison of Decoder Paradigms. Conditional Flow-Matching (CFM) surpasses the traditional DDPM in both audio quality (FAD, Similarity) and inference speed.
Table 15: Quantitative results of the two-stage text summarization QA pipeline ( N=238,074 ). The low rewrite rate confirms the high fidelity of the initial synthesis.
Heuristic Category (Suspicious)
LLM-Identified Error Type (All Audited)
Type
Count
Error Type
Count
OVERLONG
3,031
OK
3,766
PROMPT_LEAK
300
OVERLONG_OR_PROMPT_LEAK
1,747
EMPTY_OR_TRUNCATED
25
OTHER_QUALITY_ISSUE
113
EMPTY_OR_TRUNCATED
30
Appendix
Table 16: Breakdown of heuristic flags and LLM-identified error types during the audit. The primary errors were technical artifacts (e.g., prompt leaks) rather than clinical hallucinations.
Respiratory diseases remain a leading cause of global mortality, where timely and accurate diagnosis is critical to improving patient outcomes and reducing healthcare burdens. While prior work has explored audio-based models for respiratory disease detection, such unimodal approaches often suffer from limited generalizability and diagnostic precision. In this paper, we propose RespiraMFM, a Multimodal Foundation Model that integrates respiratory sounds with patient medical history and symptoms to enhance diagnostic accuracy and disease detection capabilities. We introduce an effective contrastive alignment strategy for audio-text multimodal integration, allowing the model to learn better cross-modal representations between respiratory sounds and corresponding textual clinical information. We evaluate RespiraMFM across five major respiratory diseases using seven real-world datasets in both supervised fine-tuning and zero-shot settings, achieving a 9.15% improvement in AUROC on supervised tasks and a 20.98% gain on zero-shot tasks over existing baselines. These findings underscore the potential of our framework to advance early diagnosis and improve clinical decision-making in respiratory disease management.
Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang +2
The Ohio State University · University of Southern California · University of Chicago
As conversational multimodal AI tools are increasingly adopted to process patient data for health assessment, robust benchmarks are needed to measure progress and expose failure modes under realistic conditions. Despite the importance of respiratory audio for mobile health screening, respiratory audio question answering remains underexplored, with existing studies evaluated narrowly and lacking real-world heterogeneity across modalities, devices, and question types. We hence introduce the Respiratory-Audio Question-Answering (RA-QA) benchmark, including a standardized data generation pipeline, a comprehensive multimodal QA collection, and a unified evaluation protocol. RA-QA harmonizes public RA datasets into a collection of 9 million format-diverse QA pairs covering diagnostic and contextual attributes. We benchmark general audio-language models as well as domain-specific architectures, establishing reproducible reference points and showing how current approaches fail under heterogeneity.
Gaia A. Bertolino, Yuwei Zhang, Tong Xia +2
University of Cambridge · Cambridge, United Kingdom · Tsinghua University +3
Respiratory auscultation provides a low-cost means of assessing pulmonary function, yet transient crackles and sustained wheezes can be difficult to recognize in noisy clinical recordings. Conventional deep learning classifiers often operate as black boxes, offering limited insight into the temporal acoustic evidence supporting their predictions. We present RespFocus, a weakly supervised framework that uses multiple instance learning (MIL) to learn inspectable temporal evidence from cycle-level labels and integrate it into classification. We also introduce PedResp2026, a multicenter pediatric respiratory sound cohort collected in routine clinical environments with adjudicated cycle-level annotations. In classification experiments, RespFocus achieves four-class Scores of 69.55% on public ICBHI 2017 and 72.91% on PedResp2026. Evidence diagnostics and matched temporal interventions support the functional contribution of the learned evidence to model predictions. Together, these findings highlight RespFocus's potential as a tool for automated respiratory sound diagnosis.
Yi Tian, Xiaohuan Chen, Yufei tang +5
Fuyao University of Science and Technology (fyust.edu.cn) · Xiamen University (xmu.edu.cn)