MedSonor: Learning to Use Clinical Audio in Medical LLMs

Geyang Yu; Junying Chen; Benyou Wang

cs.SDOct 14, 2026

MedSonor: Learning to Use Clinical Audio in Medical LLMs

Authors: Geyang Yu, Junying Chen, Benyou Wang

Organizations: The Chinese University of Hong Kong, Shenzhen

Abstract

Clinical audio conveys medical information that transcripts may not preserve, but using it in medical responses requires interpreting it alongside clinical requests and dialogue context. Existing medical audio supervision largely targets classification or recognition, while general audio-language models lack specialized medical knowledge and direct feedback for open-ended responses. We present MedSonor, a medical dialogue model integrating medical knowledge, spoken dialogue, and clinical audio understanding. We pair authentic medical audio with clinical requests and combine these audio-integrated dialogues with collected medical audio and synthesized spoken dialogues for supervised fine-tuning, followed by two stages of rubric-guided reinforcement learning. We introduce MedVoice-Dial and MedVoice?AID to evaluate responses within supplied multi-turn medical contexts and to requests involving clinical audio, respectively. Across seven medical benchmarks, MedSonor outperforms the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference evaluation, while broadly retaining general speech capabilities.

Figures & tables

Audio category	Key cues	Clinical relevance
Lung sounds	Wheezes, crackles, breath patterns	Respiratory auscultation.
Snoring	Spectral and temporal patterns	Upper-airway obstruction research.
Cardiac audio	Heart sounds and murmurs	Murmur detection and characterization.
Cough	Cough events and acoustics	Symptom assessment and screening research.
Medical speech	Symptoms, history, concerns	History taking and clinical reasoning.
Spoken intent	Questions and requests	Consultation goals and follow-up.

Table 1: Clinical roles of medical audio, spanning physiological sounds, spoken information, and vocal cues. Acoustic findings require clinical context and do not establish diagnoses alone.

Work	Audio modeling	Dialogue and response tasks
HeAR ( Baur et al., 2024 )	Health-acoustic representations	Consultation outside reported scope
Qwen2.5-Omni ( Xu et al., 2025 )	General audio understanding	General-purpose dialogue
AMIE ( Tu et al., 2024 )	Clinical sound interpretation outside reported scope	Text-based clinical consultation
SpeechMedAssist ( Chen et al., 2026 )	Cough and other vocal cues	Spoken medical consultation
StethoLM ( Wang et al., 2026 )	Cardiopulmonary sound interpretation	Instruction-driven clinical tasks

Table 2: Reported scope of representative models. Entries describe task coverage, not equivalent performance or clinical validation.

Figure 1: Data construction: standardized medical audio, synthesized spoken multi-turn dialogues with turn-level rubrics, and single-turn dialogues incorporating authentic medical audio.

Figure 2: Post-training overview. Medical SFT initializes the policy; successive medical dialogue RL and audio-integrated dialogue RL stages optimize severity-guarded rubric rewards.

Panel of Figure 3 — Figure 3: Physician preferences on 30 MedVoice-AID cases, based on final judgments from two attending and four resident physicians. Left: aggregate preference scores with 95% uncertainty intervals. Right: direct preference credits (%) for the row model against the column model, with matching model order. Colors span 0–100%, centered at 50%. Scoring and uncertainty estimation appear in Appendix F .

Panel of Figure 4 — Figure 4: Arena-style preference evaluation on 100 MedVoice-AID examples from 47 case families, judged by Gemini-3.8-Flash. Left: Bradley–Terry scores with 95% intervals from 2,000 case-family bootstrap resamples; Qwen2.5-Omni-7B is fixed at 1000. Right: direct preference credits (%) for the row model against the column model, with model order matched across both panels. The heatmap colorbar spans 0–100%, centered at 50%; colors do not indicate statistical significance. Ties receive half credit, and the two presentation orders are averaged.

	ASR		Dialogue		Open-ended instruction
Model	Libri	AISHELL	IFEval	CommonEval	OpenHermes	WildSpeech
Qwen2.5-Omni-7B	2.26	2.93	35.55	1.77	3.49	3.35
MedSonor (SFT)	2.60	2.13	47.58	3.40	3.47	4.87
MedSonor	2.78	1.73	45.34	3.30	3.36	4.76

Table 5: General speech capability retention after medical post-training. Libri and AISHELL report WER/CER (%; lower is better); IFEval reports %; CommonEval and OpenHermes report scores out of 5; WildSpeech reports a score out of 10. GPT-5.5 scores average three independent judgments.

Figure 5: Ablations of medical SFT and training sources. Base is Qwen2.5-Omni-7B. SFT and RL-only omit RL and medical SFT, respectively. Source ablations remove one SFT source while retaining both RL stages. Panels report GPT-5.4 rubric scores (higher is better) with independently scaled, truncated axes. Purple lines indicate the full pipeline. Complete numerical results are reported in Appendix H , Table 19 .

Model	$\textbf{Clean}(\boldsymbol{\alpha=0})$	Moderate noise ( $\boldsymbol{\alpha=0.2})$	$\textbf{Strong noise}(\boldsymbol{\alpha=0.6})$
Qwen2.5-Omni-7B	0.3341	0.2554	0.2212
Baichuan-Omni-1.5	0.3490	0.3039	0.2140
GLM-4-Voice-9B	0.2839	0.2310	0.2076
Step-Audio-2-mini	0.3442	0.3173	0.2645
LLaMA-3.1-8B-Omni	0.1411	0.1858	0.1560
MedSonor	0.7900	0.7480	0.7282

Table 6: Medical response quality under MS-SNSD babble noise (higher is better).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10

ID	User role	Scenario
1	Doctor	Doctor and patient conversing in a consultation room, where the doctor seeks medical AI assistance midway.
2	Patient	Patient at home in a bedroom, directly having a voice consultation dialogue with medical AI.
3	Nurse	Nurse at an emergency triage desk receiving patients and consulting a medical AI system.
4	Elderly patient	Elderly person at a community health center communicating with medical AI through a device.
5	Paramedic	Ambulance doctor in a moving vehicle urgently seeking medical AI support.
6	Pharmacist	Pharmacist at a hospital pharmacy window consulting medical AI when encountering medication-related questions.

Appendix

Table 9: Clinical interaction scenarios used for dialogue synthesis.

Table 12

Setting	SFT	Medical dialogue RL	Audio-integrated dialogue RL
Training examples	18,657	6,927	4,546
Learning rate	$8\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-7}$
Per-device batch size	8	4	4
Completions per prompt	–	10	10

Appendix

Table 12: Training data sizes and key hyperparameters for SFT and the two RL stages.

Setting / metric	Dialogue judge	Audio judge
Training examples	112,514	22,872
Validation examples	2,900	3,771
Parse success	100.00	100.00
Overall accuracy	94.76	89.90
Positive-item accuracy	92.37	92.31
Negative-item accuracy	99.39	85.77

Appendix

Table 13: Local judge data sizes and agreement with held-out GPT-5.4 labels. Counts are response–criterion pairs; metrics are percentages.

Benchmark	Audio	Current-turn transcript	Supplied history
LiveMedBench Clean-TTS	Spoken query	None	None
MMedFD-Lite	Spoken query	None	None
MedVoice-AID	Composed audio request	None	None
MedVoice-Dial	Current and preceding user speech	Original TTS source text	User audio and source text; reference-assistant text
ACI-Bench	Final patient turn	None	Text dialogue
HealthDial	Current user turn	None	Text dialogue

Appendix

Table 14: Policy-visible inputs within each evaluation setting. Compared models receive the same task content, encoded using their respective model/API formats. Current-target references and scoring rubrics remain judge-only.

Supervision source	Training examples
Collected medical audio	7,569
Synthesized spoken dialogue	5,088
Audio-integrated dialogue	6,000
Total	18,657

Appendix

Table 15: Composition of the final multimodal medical SFT training manifest following selection and assembly. Counts are training examples.

Source domain	Training examples
Lung and respiratory sounds	2,697
Pediatric respiratory sounds	386
Heart sounds	365
Cough and respiratory-health audio	590
Environmental health-related sounds	34
Voice and motor-speech disorders	414

Appendix

Table 16: Source-domain breakdown of the collected medical-audio SFT stream in the final training manifest.

Source (linked reference)	Count	Source (linked reference)	Count
HF Lung V1	2,575	CREMA-D	200
ICBHI 2017	122	RAVDESS	139
SPRSound	386	DAIC-WOZ	27
PhysioNet/CinC 2016	202	EATD-Corpus	73
CirCor DigiScope	163	TAME Pain	167
Coswara	146	Donate-a-Cry	88

Appendix

Table 17: Audio-data provenance and references. Each linked dataset name points to its source repository or dataset record. Counts are selected SFT training examples; they do not represent the full size of the source datasets.

ID	Use	Output contract
P1	Topic generation	One concise clinical topic per department–scenario pair
P2	Spoken dialogue synthesis	JSON alternating user and assistant turns
P3	Audio-integrated case synthesis	JSON case, audio script, reference, and signed rubric
P4	Policy input	Multimodal messages with audio markers and transcript fields
P5	Rubric generation and repair	Instance-specific signed rubric JSON
P6	Local item judge	Binary labels for every positive and negative item

Appendix

Table 18: Prompt inventory for data construction, training, and evaluation.

Training pipeline	MedVoice-AID	MedVoice-Dial	ACI-Bench	HealthDial	ThReadMed-QA
Qwen2.5-Omni-7B	0.3341	0.3237	0.2002	0.4489	0.1800
SFT	0.7666	0.4257	0.2168	0.4106	0.2034
RL-only	0.3033	0.4198	0.1297	0.3489	0.1641
w/o collected medical audio	0.7552	0.4743	0.1732	0.4665	0.2705
w/o synthesized spoken dialogue	0.7505	0.4587	0.2181	0.4570	0.2096
w/o audio-integrated dialogue	0.5957	0.4725	0.1836	0.4749	0.3058

Appendix

Table 19: Ablation of multimodal medical supervised fine-tuning and its training sources. SFT omits both RL stages. RL-only starts directly from Qwen2.5-Omni-7B, while the source-removal variants omit the named SFT source before applying the same medical dialogue RL followed by audio-integrated dialogue RL. The full pipeline applies SFT followed by these two RL stages. All values are GPT-5.4 signed instance-rubric scores under the same evaluation protocol.

Medical LLMs

Medical large language models are AI systems adapted to clinical and biomedical data for health reasoning, information extraction, and communication.

See all 397 papers

Au-M-ol: A Unified Model for Medical Audio and Language Understanding
Apr 25, 2026Meizhu Liu, Nistha Mitra, Paul Li +3Medical Speech Recognition Speech Processing
Oracle AI Science · Neuramill
A Dataset of Robot-Patient and Doctor-Patient Medical Dialogues for Spoken Language Processing Tasks
May 26, 2026Heriberto Cuayahuitl, Grace JangHealthcare Speech Processing
School of Engineering and Physical Sciences University of Lincoln, UK · Lincoln Medical School Universities of Lincoln and Nothingham, UK
MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio
May 1, 2026Harshit Rajgarhia, Shuubham Ojha, Asif Shaik +4Audio QA Healthcare
Centific Global Solutions Inc. · University of Maryland, College Park, MD, USA.

Apr 25, 2026J/K move · Enter open · S save

Au-M-ol: A Unified Model for Medical Audio and Language Understanding

Meizhu Liu, Nistha Mitra, Paul Li, Amine Abdaoui+2

Oracle AI Science · Neuramill

Medical Speech Recognition Speech Processing Audio Understanding Audio-Language Models Automatic Speech Recognition Medical LLMs

Open paper AskPDF arXiv

In this work, we present Au-M-ol, a novel multimodal architecture that extends Large Language Models (LLMs) with audio processing. It is designed to improve performance on clinically relevant tasks such as Automatic Speech Recognition (ASR). Au-M-ol has three main components: (1) an audio encoder that extracts rich acoustic features from medical speech, (2) an adaptation layer that maps audio features into the LLM input space, and (3) a pretrained LLM that performs transcription and clinical language understanding. This design allows the model to interpret spoken medical content directly, improving both accuracy and robustness. In experiments, Au-M-ol reduces Word Error Rate (WER) by 56% compared to state-of-the-art baselines on medical transcription tasks. The model also performs well in challenging conditions, including noisy environments, domain-specific terminology, and speaker variability. These results suggest that Au-M-ol is a strong candidate for real-world clinical applications, where reliable and context-aware audio understanding is essential.

Abstract

Medical LLMs

Explore similar work

Au-M-ol: A Unified Model for Medical Audio and Language Understanding

A Dataset of Robot-Patient and Doctor-Patient Medical Dialogues for Spoken Language Processing Tasks

MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio