Clinical audio conveys medical information that transcripts may not preserve, but using it in medical responses requires interpreting it alongside clinical requests and dialogue context. Existing medical audio supervision largely targets classification or recognition, while general audio-language models lack specialized medical knowledge and direct feedback for open-ended responses. We present MedSonor, a medical dialogue model integrating medical knowledge, spoken dialogue, and clinical audio understanding. We pair authentic medical audio with clinical requests and combine these audio-integrated dialogues with collected medical audio and synthesized spoken dialogues for supervised fine-tuning, followed by two stages of rubric-guided reinforcement learning. We introduce MedVoice-Dial and MedVoice?AID to evaluate responses within supplied multi-turn medical contexts and to requests involving clinical audio, respectively. Across seven medical benchmarks, MedSonor outperforms the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference evaluation, while broadly retaining general speech capabilities.
Figures & tables
Audio category
Key cues
Clinical relevance
Lung sounds
Wheezes, crackles, breath patterns
Respiratory auscultation.
Snoring
Spectral and temporal patterns
Upper-airway obstruction research.
Cardiac audio
Heart sounds and murmurs
Murmur detection and characterization.
Cough
Cough events and acoustics
Symptom assessment and screening research.
Medical speech
Symptoms, history, concerns
History taking and clinical reasoning.
Spoken intent
Questions and requests
Consultation goals and follow-up.
Table 1: Clinical roles of medical audio, spanning physiological sounds, spoken information, and vocal cues. Acoustic findings require clinical context and do not establish diagnoses alone.
Table 2: Reported scope of representative models. Entries describe task coverage, not equivalent performance or clinical validation.
Figure 1: Data construction: standardized medical audio, synthesized spoken multi-turn dialogues with turn-level rubrics, and single-turn dialogues incorporating authentic medical audio.
Figure 2: Post-training overview. Medical SFT initializes the policy; successive medical dialogue RL and audio-integrated dialogue RL stages optimize severity-guarded rubric rewards.
Figure 3: Physician preferences on 30 MedVoice-AID cases, based on final judgments from two attending and four resident physicians. Left: aggregate preference scores with 95% uncertainty intervals. Right: direct preference credits (%) for the row model against the column model, with matching model order. Colors span 0–100%, centered at 50%. Scoring and uncertainty estimation appear in Appendix F .
Figure 4: Arena-style preference evaluation on 100 MedVoice-AID examples from 47 case families, judged by Gemini-3.8-Flash. Left: Bradley–Terry scores with 95% intervals from 2,000 case-family bootstrap resamples; Qwen2.5-Omni-7B is fixed at 1000. Right: direct preference credits (%) for the row model against the column model, with model order matched across both panels. The heatmap colorbar spans 0–100%, centered at 50%; colors do not indicate statistical significance. Ties receive half credit, and the two presentation orders are averaged.
ASR
Dialogue
Open-ended instruction
Model
Libri
AISHELL
IFEval
CommonEval
OpenHermes
WildSpeech
Qwen2.5-Omni-7B
2.26
2.93
35.55
1.77
3.49
3.35
MedSonor (SFT)
2.60
2.13
47.58
3.40
3.47
4.87
MedSonor
2.78
1.73
45.34
3.30
3.36
4.76
Table 5: General speech capability retention after medical post-training. Libri and AISHELL report WER/CER (%; lower is better); IFEval reports %; CommonEval and OpenHermes report scores out of 5; WildSpeech reports a score out of 10. GPT-5.5 scores average three independent judgments.
Figure 5: Ablations of medical SFT and training sources. Base is Qwen2.5-Omni-7B. SFT and RL-only omit RL and medical SFT, respectively. Source ablations remove one SFT source while retaining both RL stages. Panels report GPT-5.4 rubric scores (higher is better) with independently scaled, truncated axes. Purple lines indicate the full pipeline. Complete numerical results are reported in Appendix H , Table 19 .
Model
Clean(α=0)
Moderate noise ( α=0.2)
Strong noise(α=0.6)
Qwen2.5-Omni-7B
0.3341
0.2554
0.2212
Baichuan-Omni-1.5
0.3490
0.3039
0.2140
GLM-4-Voice-9B
0.2839
0.2310
0.2076
Step-Audio-2-mini
0.3442
0.3173
0.2645
LLaMA-3.1-8B-Omni
0.1411
0.1858
0.1560
MedSonor
0.7900
0.7480
0.7282
Table 6: Medical response quality under MS-SNSD babble noise (higher is better).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Table 10
ID
User role
Scenario
1
Doctor
Doctor and patient conversing in a consultation room, where the doctor seeks medical AI assistance midway.
2
Patient
Patient at home in a bedroom, directly having a voice consultation dialogue with medical AI.
3
Nurse
Nurse at an emergency triage desk receiving patients and consulting a medical AI system.
4
Elderly patient
Elderly person at a community health center communicating with medical AI through a device.
5
Paramedic
Ambulance doctor in a moving vehicle urgently seeking medical AI support.
6
Pharmacist
Pharmacist at a hospital pharmacy window consulting medical AI when encountering medication-related questions.
Appendix
Table 9: Clinical interaction scenarios used for dialogue synthesis.
Table 12
Setting
SFT
Medical dialogue RL
Audio-integrated dialogue RL
Training examples
18,657
6,927
4,546
Learning rate
8×10−6
5×10−6
5×10−7
Per-device batch size
8
4
4
Completions per prompt
–
10
10
Appendix
Table 12: Training data sizes and key hyperparameters for SFT and the two RL stages.
Setting / metric
Dialogue judge
Audio judge
Training examples
112,514
22,872
Validation examples
2,900
3,771
Parse success
100.00
100.00
Overall accuracy
94.76
89.90
Positive-item accuracy
92.37
92.31
Negative-item accuracy
99.39
85.77
Appendix
Table 13: Local judge data sizes and agreement with held-out GPT-5.4 labels. Counts are response–criterion pairs; metrics are percentages.
Benchmark
Audio
Current-turn transcript
Supplied history
LiveMedBench Clean-TTS
Spoken query
None
None
MMedFD-Lite
Spoken query
None
None
MedVoice-AID
Composed audio request
None
None
MedVoice-Dial
Current and preceding user speech
Original TTS source text
User audio and source text; reference-assistant text
ACI-Bench
Final patient turn
None
Text dialogue
HealthDial
Current user turn
None
Text dialogue
Appendix
Table 14: Policy-visible inputs within each evaluation setting. Compared models receive the same task content, encoded using their respective model/API formats. Current-target references and scoring rubrics remain judge-only.
Supervision source
Training examples
Collected medical audio
7,569
Synthesized spoken dialogue
5,088
Audio-integrated dialogue
6,000
Total
18,657
Appendix
Table 15: Composition of the final multimodal medical SFT training manifest following selection and assembly. Counts are training examples.
Source domain
Training examples
Lung and respiratory sounds
2,697
Pediatric respiratory sounds
386
Heart sounds
365
Cough and respiratory-health audio
590
Environmental health-related sounds
34
Voice and motor-speech disorders
414
Appendix
Table 16: Source-domain breakdown of the collected medical-audio SFT stream in the final training manifest.
Source (linked reference)
Count
Source (linked reference)
Count
HF Lung V1
2,575
CREMA-D
200
ICBHI 2017
122
RAVDESS
139
SPRSound
386
DAIC-WOZ
27
PhysioNet/CinC 2016
202
EATD-Corpus
73
CirCor DigiScope
163
TAME Pain
167
Coswara
146
Donate-a-Cry
88
Appendix
Table 17: Audio-data provenance and references. Each linked dataset name points to its source repository or dataset record. Counts are selected SFT training examples; they do not represent the full size of the source datasets.
ID
Use
Output contract
P1
Topic generation
One concise clinical topic per department–scenario pair
P2
Spoken dialogue synthesis
JSON alternating user and assistant turns
P3
Audio-integrated case synthesis
JSON case, audio script, reference, and signed rubric
P4
Policy input
Multimodal messages with audio markers and transcript fields
P5
Rubric generation and repair
Instance-specific signed rubric JSON
P6
Local item judge
Binary labels for every positive and negative item
Appendix
Table 18: Prompt inventory for data construction, training, and evaluation.
Training pipeline
MedVoice-AID
MedVoice-Dial
ACI-Bench
HealthDial
ThReadMed-QA
Qwen2.5-Omni-7B
0.3341
0.3237
0.2002
0.4489
0.1800
SFT
0.7666
0.4257
0.2168
0.4106
0.2034
RL-only
0.3033
0.4198
0.1297
0.3489
0.1641
w/o collected medical audio
0.7552
0.4743
0.1732
0.4665
0.2705
w/o synthesized spoken dialogue
0.7505
0.4587
0.2181
0.4570
0.2096
w/o audio-integrated dialogue
0.5957
0.4725
0.1836
0.4749
0.3058
Appendix
Table 19: Ablation of multimodal medical supervised fine-tuning and its training sources. SFT omits both RL stages. RL-only starts directly from Qwen2.5-Omni-7B, while the source-removal variants omit the named SFT source before applying the same medical dialogue RL followed by audio-integrated dialogue RL. The full pipeline applies SFT followed by these two RL stages. All values are GPT-5.4 signed instance-rubric scores under the same evaluation protocol.
In this work, we present Au-M-ol, a novel multimodal architecture that extends Large Language Models (LLMs) with audio processing. It is designed to improve performance on clinically relevant tasks such as Automatic Speech Recognition (ASR). Au-M-ol has three main components: (1) an audio encoder that extracts rich acoustic features from medical speech, (2) an adaptation layer that maps audio features into the LLM input space, and (3) a pretrained LLM that performs transcription and clinical language understanding. This design allows the model to interpret spoken medical content directly, improving both accuracy and robustness. In experiments, Au-M-ol reduces Word Error Rate (WER) by 56% compared to state-of-the-art baselines on medical transcription tasks. The model also performs well in challenging conditions, including noisy environments, domain-specific terminology, and speaker variability. These results suggest that Au-M-ol is a strong candidate for real-world clinical applications, where reliable and context-aware audio understanding is essential.
Large Language Models (LLMs) have brought huge improvements to Artificial Intelligence (AI), which can be applied to general-purpose tasks. However, their application to textual or spoken medical consultations is still an open research problem. This paper proposes MeDial-Speech, a novel speech dataset for training and evaluating Med-AIs that can carry out consultations with patients. It was collected in realistic environments from robot-patient and doctor-patient dialogues, contains 111+ hours of speech data (without data augmentation), and covers four health conditions: Lewy body dementia, heart failure, shoulder pain, and angina. In addition, we propose a dialogue benchmark via sentence selection (with 20 options) to evaluate three state-of-the-art LLMs: GPT-5 mini, DeepSeek-V3, and Claude Sonnet 4. Experimental results reveal that Claude Sonnet 4 is the best in sentence selection, with 71.1% accuracy using manual transcriptions and 74.7% using automatic transcriptions, and that all LLMs are highly overconfident in their probabilistic predictions, regardless of selecting correct or incorrect sentences in medical dialogues. This dataset is free of charge for non-commercial purposes at: https://huggingface.co/datasets/hcuayahu/MeDial-Speech
Heriberto Cuayahuitl, Grace Jang
School of Engineering and Physical Sciences University of Lincoln, UK · Lincoln Medical School Universities of Lincoln and Nothingham, UK
Medical audio data is difficult to collect due to privacy regulations and high annotation costs arising from domain expertise. Thus, existing benchmarks tend to underrepresent complex medical audio scenarios. To address this challenge, we present MedMosaic, a medical audio question-answering dataset designed to benchmark language and audio reasoning models under realistic clinical constraints. MedMosaic features a diverse range of medical audio types, including condition-related physiological sounds, carefully constructed synthetic voices to mimic speech with artifacts as well as real short and long length clinical conversations to model varying context lengths. The dataset also features a total of 46,701 question-answer pairs, spanning categories such as multiple-choice, sequential multi-turn, and open-ended question-answers, enabling systematic evaluation of multi-hop reasoning and answer generation capabilities. Benchmarking 13 audio and multimodal reasoning models reveals that reasoning remains challenging for all evaluated systems, with substantial performance variation across question types. In particular, even state-of-the-art model like Gemini-2.5-pro can only achieve 68.1% accuracy approximately. These findings underscore persistent limitations in medical reasoning and highlight the need for more robust, domain-specific multimodal reasoning models. A sample of benchmark data is available here: https://shorturl.at/Lyp33
Harshit Rajgarhia, Shuubham Ojha, Asif Shaik +4
Centific Global Solutions Inc. · University of Maryland, College Park, MD, USA.