Spoken medical question answering can fail when automatic speech recognition (ASR) corrupts a clinically important term before retrieval. We present MedAstra, a framework that maps frozen WavLM speech representations through a lightweight adapter into a frozen E5 text embedding space. This enables a dense retrieval path that accepts a waveform without a query transcript. Retrieved passages and speech-linked textual metadata support answer generation with entity alternatives, diagnostic contrasts, relation heuristics, and bounded retrieval refinement. ASR-less refers to the initial speech retrieval interface; lexical retrieval, string-based entity correction, and the text generator require textual inputs. The deterministic speech embedding does not constitute a calibrated model of acoustic uncertainty. Reported retrieval results reach 87.6 Recall@10 for the speech-only retriever and 90.4 for the hybrid configuration, compared with 81.0 for transcript-first dense retrieval. Medical QA and ablations provide descriptive system comparisons, with input-provenance and evaluation limitations stated explicitly. We distinguish controlled synthetic acoustic tests from real clinical speech, qualify component attribution, and specify the remaining comparisons needed to assess speech-only QA and clinical generalization.
Figures & tables
Figure 1: Motivation for waveform-based medical evidence retrieval. Crohn’s/crown’s illustrates a possible ASR substitution, not a recorded benchmark case. The right panel depicts design goals, not established clinical reliability.
Figure 2: MedAstra ’s three-stage design. The waveform drives dense retrieval; lexical signals, reranking, string correction, and generation require text. Entity alternatives, diagnostic contrasts, and relation heuristics supply context without guaranteeing answer support.
Method
R@1
R@5
R@10
BM25 transcript
41.8 ± 0.6
63.4 ± 0.7
72.1 ± 0.8
Whisper + dense
52.3 ± 0.5
73.5 ± 0.6
81.0 ± 0.7
SpeechRAG-style
56.9 ± 0.4
77.8 ± 0.5
84.3 ± 0.6
SEAL-style
58.2 ± 0.4
79.1 ± 0.5
85.0 ± 0.5
MedAstra speech
60.7 ± 0.3
81.5 ± 0.4
87.6 ± 0.4
MedAstra hybrid
63.2 ± 0.2
84.1 ± 0.3
90.4 ± 0.3
Table 1: Previously reported SpokenSQuAD retrieval percentages (mean ± supplied SD). The hybrid row’s text-query source is unverified; it is not labeled ASR-free.
Figure 3: Reported Recall@10 under additive noise. SpeechRAG is the adapted baseline; invariance names the reported training condition. Corpus/query construction and adversary wiring remain unverified (Section 4 ).
Method
PubMedQA
MedQA
MedMCQA
LLM only
63.8 ± 0.7
48.7 ± 0.8
46.2 ± 0.9
Text RAG
69.5 ± 0.6
54.6 ± 0.7
51.8 ± 0.7
Contextual ASR
72.1 ± 0.5
57.2 ± 0.6
54.0 ± 0.6
MedRAG-style
75.2 ± 0.4
60.9 ± 0.5
57.4 ± 0.5
MedAstra
78.6 ± 0.3
64.3 ± 0.4
61.2 ± 0.4
Table 2: Previously reported answer-accuracy percentages. The supplied revision names Qwen2.5-7B-Instruct + SFT as the primary decoder, but its checkpoint and run records are unavailable. Panel (a) is a full-system comparison. Panel (b) does not repeat the baselines for each decoder. Undocumented faithfulness and relation-consistency scores are withheld.
Variant
R@10
MedQA
Entity acc.
Full
90.4 ± 0.3
64.3 ± 0.4
89.6 ± 0.3
– ASR-less retrieval
81.0 ± 0.6
57.2 ± 0.7
78.5 ± 0.6
– diagnostic graph
89.1 ± 0.4
60.7 ± 0.5
86.3 ± 0.4
– relation heuristic
89.5 ± 0.4
62.1 ± 0.5
85.7 ± 0.4
– iteration
88.2 ± 0.5
61.4 ± 0.5
87.1 ± 0.4
– style invariance
88.9 ± 0.4
63.2 ± 0.4
88.2 ± 0.3
Table 3: Previously reported component-removal results on different tasks, not one pooled test set. Numerical values are retained; matched controls and exact evaluation counts remain unverified.
Setting
MedQA accuracy
Latency (s)
Single shot
61.4 ± 0.5
1.42 ± 0.03
2 rounds
64.3 ± 0.4
1.96 ± 0.05
3 rounds
64.5 ± 0.4
2.74 ± 0.06
2 rounds + stop
64.2 ± 0.4
1.71 ± 0.04
Table 4: Previously reported iteration trade-off. Hardware, timing boundaries, and the stopping implementation for these measurements are unverified; these are not portable latency guarantees.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Operation
Required query-side input
Speech/text dense search
Projected waveform vector zs
BM25 and text reranking
Explicit text query q
Phonetic/edit correction
Text token and candidate vocabulary
Diagnostic contrasts
Extracted textual manifestations
Current answer generator
Text question, choices, readable evidence
Later-round refinement
Original text query and generated terms
Appendix
Table 5: Input contract. Stored evidence text does not supply a missing query transcript. A condition using ASR text for the last five operations is ASR-assisted answering even if its dense query is acoustic.
Component
Recoverable implementation
Historical provenance
Speech/text encoders
microsoft/wavlm-base-plus ; intfloat/e5-base ; both frozen; dimensions 768
Weights 1, 0.2, 0.1; temperature 0.07; augmentation probability 0.5; GRL 1.0
Configuration snapshot needed per run
Adapter optimization
AdamW; learning rate 2×10−4 ; weight decay 0.01; batch 8; gradient clipping 1.0; configured warmup 200 capped at 10% of steps; AMP on GPU
Epochs, seeds, hardware and elapsed times needed
Audio
Mono; 16 kHz; maximum 8 seconds; truncate then zero-pad
Duration/truncation counts not recorded
Appendix
Table 6: Code-derived settings and missing run provenance. Approximate frozen model sizes are 94M (WavLM) and 109M (E5); exact adapter/head counts follow from layer dimensions.
Public release has Google-TTS passages and text questions; waveform-query adaptation, relevance mapping, evaluated IDs/counts and noise settings missing
VoxPopuli-Med
Author-derived parliamentary subset
Subset IDs, selection rule, language/speaker/acoustic conditions and counts; no clinical status established
PubMedQA
Biomedical yes/no/maybe QA
Used subset/split IDs; availability of source abstract context; TTS provenance; task-specific parser
MedQA-USMLE
Exam QA
Release/option variant, split IDs, retained counts, TTS and question-input provenance
MedMCQA
Exam QA
Evaluation split with accessible gold labels, retained counts, TTS provenance
Appendix
Table 7: Known artifact roles and outstanding provenance. All reported language evaluation is English. No missing count is estimated from a public release total.
Public artifact/source
Terms stated by upstream
LibriSpeech (SLR12)
CC BY 4.0
SpokenSQuAD dataset
CC BY-SA 4.0
PubMedQA repository
MIT; underlying article terms need separate checking
MedQA repository
MIT; externally hosted exam/textbook sources need separate checking
MedMCQA repository
MIT; actual downloaded release requires confirmation
VoxPopuli data
CC0; original recordings also have Parliament notice
Appendix
Table 8: Public upstream terms checked on October 10, 2026. Links identify the primary source; missing historical versions and corpus permissions remain unresolved.
Label
Rule
SUPPORTED
All clinically material claims are entailed by the visible evidence; no material contradiction or unsupported recommendation.
PARTIAL
The main claim has evidence support, but at least one material detail is unlicensed or insufficiently established.
UNSUPPORTED
The main claim lacks evidence or contradicts the visible evidence. A correct benchmark option can still receive this label.
UNSAFE
The output adds personalized diagnosis, triage or treatment advice beyond the evidence and task framing. This is an overreach label, not a validated clinical harm score.
Appendix
Table 9: Prospective evidence-support rubric. Apply UNSAFE first, then UNSUPPORTED, PARTIAL, or SUPPORTED.
Retrieval
Clean
20 dB
10 dB
5 dB
ASR + dense
81.0 ± 0.5
75.7 ± 0.6
63.9 ± 0.7
49.4 ± 0.8
SpeechRAG
84.3 ± 0.4
81.6 ± 0.5
75.2 ± 0.6
66.1 ± 0.7
MedAstra
87.6 ± 0.3
84.2 ± 0.4
78.5 ± 0.5
69.7 ± 0.6
+ style
88.9 ± 0.2
86.7 ± 0.3
82.4 ± 0.4
75.8 ± 0.5
Appendix
Table 10: Reported Recall@10 under additive noise, including all original supplied SDs. ASR uses Whisper plus dense retrieval; SpeechRAG is the adapted baseline; the last two rows omit/add style training. These statistics inherit the missing run/query provenance in Section 4 .
Method
Entity acc.
Corr. F1
Rare recall
ASR transcript
71.6 ± 0.8
64.3 ± 0.7
58.9 ± 0.9
MedSpeak correction
82.4 ± 0.5
77.9 ± 0.5
71.6 ± 0.6
MedAstra – relation
85.7 ± 0.4
81.5 ± 0.4
76.2 ± 0.5
MedAstra
89.6 ± 0.3
85.8 ± 0.3
81.9 ± 0.4
Appendix
Table 11: Reported objective stress-test percentages (mean ± supplied SD), transcribed unchanged from the original manuscript. Entity metrics and diagnosis top-1 use different suites. Construction, rare-term definition, run counts, and evaluated membership remain unverified; no safety or population-level claim follows.
Figure 4: Restored objective panels of the clinical grounding plot: entity accuracy and diagnosis top-1, with the original means and supplied SDs. The suites have different denominators and missing run provenance; these are descriptive benchmark results, not clinical validation.
Figure 5: Restored accuracy/latency trade-off with unchanged reported means and supplied SDs, not confidence intervals. Undocumented faithfulness and relation-consistency curves are withheld. Timing provenance and the historical stopping implementation remain unverified.
Failure class
Distinguishing observation
Retrieval miss
Relevant item absent from the returned ranks
Entity confusion
Query or answer substitutes the reference term
Lexical mismatch
Exact term available but omitted by retrieval
Evidence insufficiency
Visible snippets lack the decisive finding
Unsupported answer
Generated claim lacks visible evidence support
Relation distortion
Answer misstates a relation in the evidence
Appendix
Table 12: Failure taxonomy for analysis; no completed human annotation study is claimed. Classes may co-occur; no frequencies are claimed.
End-to-end speech-to-text retrieval is becoming an important capability for voice agents, allowing spoken queries to retrieve relevant documents without intermediate transcription. Training these retrievers typically depends on large collections of spoken query-document pairs that are costly to acquire and difficult to extend across domains and languages. We introduce \ours, which replaces such task-specific supervision with speech-transcript pairs from standard ASR corpora. A lightweight adapter projects the output of a frozen speech encoder into the retrieval space of a frozen pretrained text embedding model. \ours is competitive with cascaded ASR-then-retrieve systems while reducing inference latency, and outperforms existing publicly available speech embedding models. It also transfers to text-to-speech and speech-to-speech retrieval without additional training. Controlled analyses validate the speech-transcript alignment paradigm, demonstrating robustness to reduced training data and consistent gains from larger corpora. Further ablations of loss functions and trainable modules establish practical guidelines for the framework. Model weights and codes are to be released.
Sangkwon Park, Dongwook Lee, Youngho Cho +3
ECE, Seoul National University · IPAI, Seoul National University · University of Seoul +1
Spoken Question Answering (Spoken QA) presents a challenging cross-modal problem: effectively aligning acoustic queries with textual knowledge while avoiding the latency and error propagation inherent in cascaded ASR-based systems. In this paper, we introduce Attention-guided Evidence Grounding (AEG), a novel end-to-end framework that leverages the internal cross-modal attention of Speech Large Language Models (SpeechLLMs) to explicitly locate and ground key evidence in the model's latent space. To address the diffuse attention distribution in pre-trained models, we propose Learning to Focus on Evidence (LFE), a supervised fine-tuning paradigm that calibrates the model's attention mechanism to distinguish query-relevant segments from irrelevant context. Experiments on SQuAD, HotpotQA, and MuSiQue demonstrate that AEG reduces hallucinations and achieves strong efficiency gains, outperforming large-scale cascaded baselines (Whisper-Large-v3 + Reranker) while reducing inference latency by approximately 62%.
Overall word error rate hides clinically critical errors: a transcript can be 95% correct and still swap one drug for another. The usual fix weights errors on medical entities, and almost always depends on an evaluation-time named-entity recognition (NER) model or cloud API, which makes the metric's denominator a versioned black box. We present MedWER, an evaluation protocol and open-source tool for medical ASR whose denominator is a fixed, license-clean term list: 19,373 drug, diagnosis, symptom, and injury-mechanism entries projected from public sources. The protocol couples a pinned text normalizer with a phrase-aware term-restricted WER, the MedWER, so the only versioned component is a normalizer dependency held at an exact release and checked against committed golden fixtures. Coverage is validated against an independent provincial drug-benefit file the list was not built from; the matching heuristic is calibrated against ground-truth entity spans. Baselines for Moonshinebase, Whisperbase.en, and MedASR on two open benchmarks are scored with the released tool and reported with 95% confidence intervals from resampled per-utterance scores.