RAMoEA-QA: Hierarchical Specialization for Robust Respiratory Audio Question Answering

Gaia A. Bertolino; Yuwei Zhang; Tong Xia; Domenico Talia; Cecilia Mascolo

cs.SDMar 6, 2026

RAMoEA-QA: Hierarchical Specialization for Robust Respiratory Audio Question Answering

Authors: Gaia A. Bertolino, Yuwei Zhang, Tong Xia, Domenico Talia, Cecilia Mascolo

Organizations: University of Cambridge, United Kingdom · Tsinghua University, China · University of Calabria, Italy

Abstract

Conversational generative AI is increasingly explored in healthcare, where models must integrate heterogeneous patient signals and support diverse interaction styles while producing clinically meaningful outputs. In respiratory care, non-invasive audio recordings captured with sensing devices offer a scalable route to screening and longitudinal monitoring, but heterogeneity is particularly acute: recordings vary across devices, environments, and acquisition protocols, and queries may vary in intent, answer format, and prediction objective. Existing biomedical audio-language question answering systems for respiratory assessment are starting to emerge, but they are typically built as single-path models, processing all inputs through the same acoustic and language pathway despite variation in recording conditions and query types. They are also usually evaluated in relatively limited settings, leaving open their robustness under realistic distribution shifts, including changes in acquisition domains, modality, and clinical task. To address this gap, we introduce RAMoEA-QA, the first RA QA model designed to support input-dependent specialization across heterogeneous recordings and query types within a unified hierarchical two-stage framework. We study this design in a unified RA QA setting spanning clinical and self-recorded, multi-device acquisition settings, question formats, and both discrete and continuous targets. Across in-domain and controlled-shift evaluations, RAMoEA-QA improves over matched monolithic baselines and routing controls, reaching 0.72 in in-domain test accuracy (vs. 0.61 and 0.67 for single-path baselines) on discriminative tasks, while also achieving the best regression performance and stronger average transfer under dataset, modality, and task shifts, including gains of up to 23 percentage points in accuracy on the COPD modality-shift setting.

Figures & tables

Task Family	Dataset	Task	Audio condition	Query-types
Discriminative	Coswara	Asthma	Deep and shallow breathing	SV
	COUGHVID	Diagnosis (Healthy, Symptomatic, Covid)	Coughing	All
	ICBHI	Diagnosis (URTI, COPD)	Four clinical recording devices	All
	KAUH	Diagnosis (Healthy, LRTI, COPD)	Sound types and positions	All
	Respiratory@TR	COPD Severity	Recording positions	OE, MC
Regression	MM-Lung	Spirometry (FEV1)	Deep breathing	OE

Table 2

Model	Task-level metrics				Text-level metrics
	Discriminative tasks		Regression tasks		Global
	Accuracy $\uparrow$	Macro-F1 $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	Token F1 $\uparrow$	BERTScore $\uparrow$	METEOR $\uparrow$
Pengi	0.22	0.21	NA	NA	0.02	-0.01	2.61
LTU*	0.18	0.11	14.36	36.20	0.34	0.90	52.64
GAMA	0.45	0.28	2.83	4.21	0.43	0.76	70.63
CareAQA-CT	0.61 $\pm$ 0.01	0.53 $\pm$ 0.06	2.61 $\pm$ 0.16	3.93 $\pm$ 0.14	0.83 $\pm$ 0.01	0.87 $\pm$ 0.01	84.89 $\pm$ 0.90

Table 3

Shift	Dataset	Task	CareAQA-operaCT		CareAQA-operaGT		RAMoEA-QA (ours)
			Accuracy $\uparrow$	MacroF1 $\uparrow$	Accuracy $\uparrow$	MacroF1 $\uparrow$	Accuracy $\uparrow$	MacroF1 $\uparrow$
$\Delta M$	Coswara	Asthma	0.90 ± 0.13	0.90 ± 0.13	0.66 ± 0.47	0.75 ± 0.35	0.87 ± 0.02	0.87 ± 0.02
$\Delta M$	Coswara	COPD	0.68 ± 0.44	0.75 ± 0.34	0.66 ± 0.47	0.75 ± 0.35	0.91 ± 0.02	0.92 ± 0.02
$\Delta D$	UK COVID-19	Asthma	0.84 ± 0.21	0.84 ± 0.21	0.66 ± 0.47	0.74 ± 0.35	0.88 ± 0.07	0.88 ± 0.06
$\Delta D$	UK COVID-19	COVID	0.56 ± 0.32	0.65 ± 0.20	0.66 ± 0.47	0.74 ± 0.35	0.84 ± 0.04	0.84 ± 0.04
$\Delta TM$	Coswara	Pneumonia	0.77 ± 0.31	0.79 ± 0.28	0.33 ± 0.00	0.50 ± 0.00	0.83 ± 0.05	0.84 ± 0.04

Table 6

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Source	Access	License
UK COVID-19	IC	https://zenodo.org/records/10043978	OGL 3.0
CoughVID	EPFL	https://zenodo.org/records/4048312	CC BY 4.0
ICBHI		https://bhichallenge.med.auth.gr	CC0
HF Lung		https://gitlab.com/techsupportHF/HF_Lung_V1	CC BY 4.0
HF Lung		https://gitlab.com/techsupportHF/HF_Lung_V1_IP	CC BY-NC 4.0
Coswara	IISc	https://github.com/iiscleap/Coswara-Data	CC BY 4.0

Appendix

Table 7

Task	Dataset	Task	Modalities	QA $N$
Discr.	Coswara	Asthma *	Deep and shallow breathing	204 / 64 / 64
	COUGHVID	Diagnosis (Healthy, Symptomatic, Covid-19)	Coughing	225 / 45 / 45
	ICBHI	Diagnosis (URTI, COPD)	Four clinical recording devices	423 / 32 / 32
	KAUH	Diagnosis (Healthy, LRTI, COPD)	6 sound types and 7 chest positions	225 / 45 / 45
	Respiratory@TR	COPD Severity	11 recording positions	210 / 70 / 70
Regr.	MM-Lung	Spirometry (FEV1)	Deep breathing	32 / 4 / 4

Appendix

Table 8

Model	Prediction	Parsed value	MAE $\downarrow$
GAMA	The patient’s respiratory rate is 24 breaths per minute.	24.0	12.60
LTU*	The patient’s respiratory rate cannot be accurately determined based on the given audio clip alone.	NA	NA
CareAQA-CT	The patient’s respiratory rate is 12.22 breaths/minute.	12.22	0.82
CareAQA-GT	The patient’s respiratory rate is 12.22 breaths/minute.	12.22	0.82
RAMoEA-QA	The patient’s respiratory rate is 11.33 breaths/minute.	11.33	0.07

Appendix

Table 18

Model	Prediction	Parsed label	Correct
GAMA	The patient has a severe case of pneumonia.	pneumonia	No
LTU*	The diagnosis for the patient cannot be determined based on the given information.	NA	No
CareAQA-CT	The diagnosis for the patient is chronic obstructive pulmonary disease.	COPD	No
CareAQA-GT	The diagnosis for the patient is chronic obstructive pulmonary disease.	COPD	No
RAMoEA-QA	The diagnosis for the patient is a lower respiratory tract infection.	LRTI	Yes

Appendix

Table 19

Model	Single-Verify		Open-ended		Multiple-choice
	Macro F1 $\uparrow$	BERTScore $\uparrow$	Macro F1 $\uparrow$	BERTScore $\uparrow$	Macro F1 $\uparrow$	BERTScore $\uparrow$
PENGI	0.40	0.03	0.00	-0.09	0.00	-0.08
GAMA	0.48	0.85	0.05	0.55	0.38	0.60
CareAQA-CT	0.54 $\pm$ 0.10	0.87 $\pm$ 0.02	0.49 $\pm$ 0.05	0.86 $\pm$ 0.00	0.59 $\pm$ 0.08	0.85 $\pm$ 0.02
CareAQA-GT	0.62 $\pm$ 0.28	0.91 $\pm$ 0.01	0.50 $\pm$ 0.06	0.87 $\pm$ 0.00	0.59 $\pm$ 0.08	0.84 $\pm$ 0.00
RAMoEA-QA	0.82 $\pm$ 0.03	0.94 $\pm$ 0.00	0.42 $\pm$ 0.04	0.83 $\pm$ 0.02	0.61 $\pm$ 0.01	0.85 $\pm$ 0.03

Appendix

Table 20

Setting	Accuracy $\uparrow$	MAE $\downarrow$	BERTScore $\uparrow$
operaCT + 2 LoRA	0.61	2.68	0.87
operaGT + 2 LoRA	0.70	2.58	0.89
operaCT / operaGT + 1 LoRA	0.68	2.55	0.86
operaCT / operaGT + 1 LoRA (double rank)	0.70	2.55	0.88
RAMoEA-QA	0.72	2.29	0.90

Appendix

Table 21

Setting	Accuracy $\uparrow$	MAE $\downarrow$	BERTScore $\uparrow$
operaCT / operaGT + 2 LoRA	0.72	2.29	0.90
operaCT / operaGT + 8 LoRA	0.75	2.79	0.91
operaCT / operaGT + 4 LoRA	0.71	2.54	0.91
operaCT / operaGT / operaCE + 2 LoRA	0.86	2.55	0.94

Appendix

Table 22

Model	Accuracy $\uparrow$	Macro F1 $\uparrow$	MAE $\downarrow$
Majority Vote	0.61	0.54	2.62
RAMoEA-QA	0.72	0.67	2.29

Appendix

Table 23

Question-type	Metric	MM Lung		NoseMic
		FEV1	FVC	Respiratory rate
Open-ended	MSE $\downarrow$	$1.11\pm 0.30$	$3.42\pm 0.28$	$27.79\pm 2.34$
Open-ended	RMSE $\downarrow$	$1.05\pm 0.14$	$1.85\pm 0.08$	$5.27\pm 0.22$

Appendix

Table 2: Per-dataset breakdown for in-domain testing (mean

\pm

std over runs) for the baseline using operaCT on regressive tasks.

Question-type	Metric	MM Lung		NoseMic
		FEV1	FVC	Respiratory rate
Open-ended	MSE $\downarrow$	$1.13\pm 0.28$	$2.73\pm 0.69$	$33.23\pm 2.01$
Open-ended	RMSE $\downarrow$	$1.06\pm 0.13$	$1.65\pm 0.21$	$5.76\pm 0.17$

Appendix

Table 3: Per-dataset breakdown for in-domain testing (mean

\pm

std over runs) for the baseline using operaGT on regressive tasks.

Question-type	Metric	MM Lung		NoseMic
		FEV1	FVC	Respiratory rate
Open-ended	MSE $\downarrow$	$0.96\pm 0.13$	$2.42\pm 1.00$	$26.11\pm 8.09$
Open-ended	RMSE $\downarrow$	$0.98\pm 0.07$	$1.54\pm 0.33$	$5.08\pm 0.80$

Appendix

Table 4: Per-dataset breakdown for in-domain testing (mean

\pm

std over runs) for RAMoEA-QA on regressive tasks.

Metric	Subset	Global	Coswara	CoughVid	ICBHI	KAUH	Respiratory@TR
Accuracy $\uparrow$	Global	$0.68\pm 0.16$	$0.75\pm 0.35$	$0.62\pm 0.06$	$0.50\pm 0.00$	$0.54\pm 0.18$	$0.88\pm 0.04$
	Single-verify	$0.70\pm 0.21$	$0.75\pm 0.35$	$0.67\pm 0.00$	$0.50\pm 0.00$	$0.67\pm 0.24$	$0.75\pm 0.07$
	Open-ended	$0.60\pm 0.06$	-	$0.33\pm 0.00$	$0.50\pm 0.00$	$0.50\pm 0.24$	$1.00\pm 0.00$
	Multiple-choice	$0.68\pm 0.06$	-	$0.83\pm 0.24$	$0.50\pm 0.00$	$0.33\pm 0.00$	$1.00\pm 0.00$
Macro F1 $\uparrow$	Global	$0.59\pm 0.21$	$0.67\pm 0.47$	$0.55\pm 0.08$	$0.33\pm 0.00$	$0.43\pm 0.23$	$0.87\pm 0.05$
	Single-verify	$0.62\pm 0.28$	$0.67\pm 0.47$	$0.62\pm 0.00$	$0.33\pm 0.00$	$0.58\pm 0.35$	$0.74\pm 0.09$

Appendix

Table 27

Metric	Subset	Global	Coswara	CoughVid	ICBHI	KAUH	Respiratory@TR
Accuracy $\uparrow$	Global	$0.61\pm 0.02$	$0.59\pm 0.12$	$0.65\pm 0.14$	$0.56\pm 0.09$	$0.47\pm 0.04$	$0.79\pm 0.02$
	Single-verify	$0.60\pm 0.03$	$0.59\pm 0.12$	$0.72\pm 0.16$	$0.62\pm 0.18$	$0.53\pm 0.20$	$0.58\pm 0.04$
	Open-ended	$0.60\pm 0.06$	-	$0.33\pm 0.00$	$0.50\pm 0.00$	$0.50\pm 0.24$	$1.00\pm 0.00$
	Multiple-choice	$0.68\pm 0.06$	-	$0.83\pm 0.24$	$0.50\pm 0.00$	$0.33\pm 0.00$	$1.00\pm 0.00$
Macro F1 $\uparrow$	Global	$0.54\pm 0.06$	$0.50\pm 0.24$	$0.58\pm 0.18$	$0.43\pm 0.14$	$0.40\pm 0.03$	$0.77\pm 0.01$
	Single-verify	$0.54\pm 0.10$	$0.50\pm 0.24$	$0.69\pm 0.20$	$0.53\pm 0.28$	$0.51\pm 0.17$	$0.54\pm 0.02$

Appendix

Table 28

Metric	Subset	Global	Coswara	CoughVid	ICBHI	KAUH	Respiratory@TR
Accuracy $\uparrow$	Global	$0.72\pm 0.02$	$0.86\pm 0.09$	$0.54\pm 0.14$	$0.58\pm 0.15$	$0.68\pm 0.02$	$0.79\pm 0.02$
	Single-verify	$0.81\pm 0.00$	$0.86\pm 0.09$	$0.78\pm 0.16$	$0.72\pm 0.22$	$1.00\pm 0.00$	$0.57\pm 0.04$
	Open-ended	$0.46\pm 0.02$	-	$0.00\pm 0.00$	$0.44\pm 0.09$	$0.33\pm 0.00$	$1.00\pm 0.00$
	Multiple-choice	$0.62\pm 0.10$	-	$0.61\pm 0.24$	$0.44\pm 0.09$	$0.39\pm 0.08$	$1.00\pm 0.00$
Macro F1 $\uparrow$	Global	$0.67\pm 0.03$	$0.86\pm 0.09$	$0.51\pm 0.18$	$0.47\pm 0.18$	$0.60\pm 0.02$	$0.75\pm 0.01$
	Single-verify	$0.79\pm 0.01$	$0.86\pm 0.09$	$0.76\pm 0.19$	$0.67\pm 0.29$	$1.00\pm 0.00$	$0.51\pm 0.03$

Appendix

Table 29

Audio QA

Audio question answering uses spoken, musical, and environmental sounds to answer questions grounded in acoustic evidence.

See all 45 papers

Feb 4, 2026

RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity

As conversational multimodal AI tools are increasingly adopted to process patient data for health assessment, robust benchmarks are needed to measure progress and expose failure modes under realistic conditions. Despite the importance of respiratory audio for mobile health screening, respiratory audio question answering remains underexplored, with existing studies evaluated narrowly and lacking real-world heterogeneity across modalities, devices, and question types. We hence introduce the Respiratory-Audio Question-Answering (RA-QA) benchmark, including a standardized data generation pipeline, a comprehensive multimodal QA collection, and a unified evaluation protocol. RA-QA harmonizes public RA datasets into a collection of 9 million format-diverse QA pairs covering diagnostic and contextual attributes. We benchmark general audio-language models as well as domain-specific architectures, establishing reproducible reference points and showing how current approaches fail under heterogeneity.

Gaia A. Bertolino, Yuwei Zhang, Tong Xia +2

University of Cambridge · Cambridge, United Kingdom · Tsinghua University +3

Audio-Language Model Evaluation Audio QA Healthcare Audio Understanding Multimodal QA Audio-Language Models+2

Ask PDF

Feb 16, 2026

Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis

Deep learning-based respiratory auscultation is currently hindered by two fundamental challenges: (i) inherent information loss, as converting signals into spectrograms discards transient acoustic events and clinical context; (ii) limited data availability, exacerbated by severe class imbalance. To bridge these gaps, we present Resp-Agent, an autonomous multimodal system orchestrated by a novel Active Adversarial Curriculum Agent (Thinker-A

^2

CA). Unlike static pipelines, Thinker-A

^2

CA serves as a central controller that actively identifies diagnostic weaknesses and schedules targeted synthesis in a closed loop. To address the representation gap, we introduce a modality-weaving Diagnoser that weaves clinical text with audio tokens via strategic global attention and sparse audio anchors, capturing both long-range clinical context and millisecond-level transients. To address the data gap, we design a flow matching Generator that adapts a text-only Large Language Model (LLM) via modality injection, decoupling pathological content from acoustic style to synthesize hard-to-diagnose samples. As a foundation for this work, we introduce Resp-229k, a benchmark corpus of 229k recordings paired with LLM-distilled clinical narratives. Extensive experiments demonstrate that Resp-Agent consistently outperforms prior approaches across diverse evaluation settings, improving diagnostic robustness under data scarcity and long-tailed class imbalance. Our code and data are available at https://github.com/zpforlove/Resp-Agent.

Pengfei Zhang, Tianxin Xie, Minghao Yang +1

The Hong Kong University of Science and Technology (Guangzhou)

Respiratory Sound Classification Audio Generation Synthetic Data Augmentation Multimodal Classification

Ask PDF

Feb 27, 2026

StethoLM: Audio Language Model for Cardiopulmonary Analysis Across Clinical Tasks

Listening to heart and lung sounds - auscultation - is one of the first and most fundamental steps in a clinical examination. Despite being fast and non-invasive, it demands years of experience to interpret subtle audio cues. Recent deep learning methods have made progress in automating cardiopulmonary sound analysis, yet most are restricted to simple classification and offer little clinical interpretability or decision support. We present StethoLM, the first audio-language model specialized for cardiopulmonary auscultation, capable of performing instruction-driven clinical tasks across the full spectrum of auscultation analysis. StethoLM integrates audio encoding with a medical language model backbone and is trained on StethoBench, a comprehensive benchmark comprising 77,027 instruction-response pairs synthesized from 16,125 labeled cardiopulmonary recordings spanning seven clinical task categories: binary classification, detection, reporting, reasoning, differential diagnosis, comparison, and location-based analysis. Through multi-stage training that combines supervised fine-tuning and direct preference optimization, StethoLM achieves substantial gains in performance and robustness on out-of-distribution data. Our work establishes a foundation for instruction-following AI systems in clinical auscultation.

Yishan Wang, Tsai-Ning Wang, Mathias Funk +1

Eindhoven University of Technology

Audio-Language Models Respiratory Sound Classification

Ask PDF

Abstract

Audio QA

Explore similar work

RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity

Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis

StethoLM: Audio Language Model for Cardiopulmonary Analysis Across Clinical Tasks