MedAstra: ASR-less Speech Retrieval with Diagnostic Grounding for Medical QA
Abstract
Spoken medical question answering can fail when automatic speech recognition (ASR) corrupts a clinically important term before retrieval. We present MedAstra, a framework that maps frozen WavLM speech representations through a lightweight adapter into a frozen E5 text embedding space. This enables a dense retrieval path that accepts a waveform without a query transcript. Retrieved passages and speech-linked textual metadata support answer generation with entity alternatives, diagnostic contrasts, relation heuristics, and bounded retrieval refinement. ASR-less refers to the initial speech retrieval interface; lexical retrieval, string-based entity correction, and the text generator require textual inputs. The deterministic speech embedding does not constitute a calibrated model of acoustic uncertainty. Reported retrieval results reach 87.6 Recall@10 for the speech-only retriever and 90.4 for the hybrid configuration, compared with 81.0 for transcript-first dense retrieval. Medical QA and ablations provide descriptive system comparisons, with input-provenance and evaluation limitations stated explicitly. We distinguish controlled synthetic acoustic tests from real clinical speech, qualify component attribution, and specify the remaining comparisons needed to assess speech-only QA and clinical generalization.
Figures & tables
| Method | R@1 | R@5 | R@10 |
|---|---|---|---|
| BM25 transcript | 41.8 0.6 | 63.4 0.7 | 72.1 0.8 |
| Whisper + dense | 52.3 0.5 | 73.5 0.6 | 81.0 0.7 |
| SpeechRAG-style | 56.9 0.4 | 77.8 0.5 | 84.3 0.6 |
| SEAL-style | 58.2 0.4 | 79.1 0.5 | 85.0 0.5 |
| MedAstra speech | 60.7 0.3 | 81.5 0.4 | 87.6 0.4 |
| MedAstra hybrid | 63.2 0.2 | 84.1 0.3 | 90.4 0.3 |
| Method | PubMedQA | MedQA | MedMCQA |
|---|---|---|---|
| LLM only | 63.8 0.7 | 48.7 0.8 | 46.2 0.9 |
| Text RAG | 69.5 0.6 | 54.6 0.7 | 51.8 0.7 |
| Contextual ASR | 72.1 0.5 | 57.2 0.6 | 54.0 0.6 |
| MedRAG-style | 75.2 0.4 | 60.9 0.5 | 57.4 0.5 |
| MedAstra | 78.6 0.3 | 64.3 0.4 | 61.2 0.4 |
| Variant | R@10 | MedQA | Entity acc. |
|---|---|---|---|
| Full | 90.4 0.3 | 64.3 0.4 | 89.6 0.3 |
| – ASR-less retrieval | 81.0 0.6 | 57.2 0.7 | 78.5 0.6 |
| – diagnostic graph | 89.1 0.4 | 60.7 0.5 | 86.3 0.4 |
| – relation heuristic | 89.5 0.4 | 62.1 0.5 | 85.7 0.4 |
| – iteration | 88.2 0.5 | 61.4 0.5 | 87.1 0.4 |
| – style invariance | 88.9 0.4 | 63.2 0.4 | 88.2 0.3 |
| Setting | MedQA accuracy | Latency (s) |
|---|---|---|
| Single shot | 61.4 0.5 | 1.42 0.03 |
| 2 rounds | 64.3 0.4 | 1.96 0.05 |
| 3 rounds | 64.5 0.4 | 2.74 0.06 |
| 2 rounds + stop | 64.2 0.4 | 1.71 0.04 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Operation | Required query-side input |
|---|---|
| Speech/text dense search | Projected waveform vector |
| BM25 and text reranking | Explicit text query |
| Phonetic/edit correction | Text token and candidate vocabulary |
| Diagnostic contrasts | Extracted textual manifestations |
| Current answer generator | Text question, choices, readable evidence |
| Later-round refinement | Original text query and generated terms |
| Component | Recoverable implementation | Historical provenance |
|---|---|---|
| Speech/text encoders | microsoft/wavlm-base-plus ; intfloat/e5-base ; both frozen; dimensions 768 | Checkpoint revisions not recorded |
| Adapter | , ReLU, dropout 0.1, LayerNorm; 1,182,720 trainable parameters | Trained checkpoints absent |
| Style head | , dropout 0.1; 197,892 parameters | Enabled in configuration; run states absent |
| Alignment/invariance | Weights 1, 0.2, 0.1; temperature 0.07; augmentation probability 0.5; GRL 1.0 | Configuration snapshot needed per run |
| Adapter optimization | AdamW; learning rate ; weight decay 0.01; batch 8; gradient clipping 1.0; configured warmup 200 capped at 10% of steps; AMP on GPU | Epochs, seeds, hardware and elapsed times needed |
| Audio | Mono; 16 kHz; maximum 8 seconds; truncate then zero-pad | Duration/truncation counts not recorded |
| Artifact | Stated role | Missing information |
|---|---|---|
| LibriSpeech | Paired alignment supervision | Used splits, retained counts, audio durations, speaker overlap; supplied config selects train.100[:2000] |
| SpokenSQuAD | Speech retrieval/noise analysis | Public release has Google-TTS passages and text questions; waveform-query adaptation, relevance mapping, evaluated IDs/counts and noise settings missing |
| VoxPopuli-Med | Author-derived parliamentary subset | Subset IDs, selection rule, language/speaker/acoustic conditions and counts; no clinical status established |
| PubMedQA | Biomedical yes/no/maybe QA | Used subset/split IDs; availability of source abstract context; TTS provenance; task-specific parser |
| MedQA-USMLE | Exam QA | Release/option variant, split IDs, retained counts, TTS and question-input provenance |
| MedMCQA | Exam QA | Evaluation split with accessible gold labels, retained counts, TTS provenance |
| Public artifact/source | Terms stated by upstream |
|---|---|
| LibriSpeech (SLR12) | CC BY 4.0 |
| SpokenSQuAD dataset | CC BY-SA 4.0 |
| PubMedQA repository | MIT; underlying article terms need separate checking |
| MedQA repository | MIT; externally hosted exam/textbook sources need separate checking |
| MedMCQA repository | MIT; actual downloaded release requires confirmation |
| VoxPopuli data | CC0; original recordings also have Parliament notice |
| Label | Rule |
|---|---|
| SUPPORTED | All clinically material claims are entailed by the visible evidence; no material contradiction or unsupported recommendation. |
| PARTIAL | The main claim has evidence support, but at least one material detail is unlicensed or insufficiently established. |
| UNSUPPORTED | The main claim lacks evidence or contradicts the visible evidence. A correct benchmark option can still receive this label. |
| UNSAFE | The output adds personalized diagnosis, triage or treatment advice beyond the evidence and task framing. This is an overreach label, not a validated clinical harm score. |
| Retrieval | Clean | 20 dB | 10 dB | 5 dB |
|---|---|---|---|---|
| ASR + dense | 81.0 0.5 | 75.7 0.6 | 63.9 0.7 | 49.4 0.8 |
| SpeechRAG | 84.3 0.4 | 81.6 0.5 | 75.2 0.6 | 66.1 0.7 |
| MedAstra | 87.6 0.3 | 84.2 0.4 | 78.5 0.5 | 69.7 0.6 |
| + style | 88.9 0.2 | 86.7 0.3 | 82.4 0.4 | 75.8 0.5 |
| Method | Entity acc. | Corr. F1 | Rare recall |
|---|---|---|---|
| ASR transcript | 71.6 0.8 | 64.3 0.7 | 58.9 0.9 |
| MedSpeak correction | 82.4 0.5 | 77.9 0.5 | 71.6 0.6 |
| MedAstra – relation | 85.7 0.4 | 81.5 0.4 | 76.2 0.5 |
| MedAstra | 89.6 0.3 | 85.8 0.3 | 81.9 0.4 |
| Failure class | Distinguishing observation |
|---|---|
| Retrieval miss | Relevant item absent from the returned ranks |
| Entity confusion | Query or answer substitutes the reference term |
| Lexical mismatch | Exact term available but omitted by retrieval |
| Evidence insufficiency | Visible snippets lack the decisive finding |
| Unsupported answer | Generated claim lacks visible evidence support |
| Relation distortion | Answer misstates a relation in the evidence |
Medical QA
Medical question answering develops and evaluates AI systems that answer clinical, patient-facing, and health-information questions.
See all 96 papers