Medical question answering collapses in Hausa: a pre-registered pilot on three small open models
Organizations: Mewar University
Abstract
A system that reads nothing and always answers the most frequent letter scores \BaselineP{} on our Hausa medical multiple-choice set. \NBelowBaseline{} of the \NModels{} small open models we evaluate do not beat it. We lead with that because the usual summary of a multilingual result -- a percentage-point drop from English -- does not say whether what survives the drop is performance at all. On a paired English--Hausa set of \NEval{} MedMCQA items, three instruction-tuned models of 3--4B parameters lose between \GapMedGemma{} and \GapQwen{} points of accuracy, every gap significant after Holm correction. Back-translating the Hausa items into English recovers between half and two-thirds of the gap for the two general-purpose models but not demonstrably for the medically adapted one, so the deficit is substantially, though not wholly, a matter of the language of presentation. Answers are also far less stable: agreement across three option orderings falls from \ConsEnQwen{} to \ConsHaQwen{} in the worst case. An exploratory error analysis of \ENRows{} discordant items attributes most of the damage to medical vocabulary lost in translation (\ETerm{}, 95% CI \ETermLo{}--\ETermHi{}), usually in the question stem. A fourth registered hypothesis proposed closing the gap with self-generated preference pairs. Its pilot gate found nothing to train on: asked to reason in Hausa rather than to pick a letter, the model echoes the prompt back, switches into Swahili, or simply stops without answering; only \GateParse{} of chains are parseable. We report that intervention as not run, which is itself the sharper result -- a forced-choice benchmark cannot see a model that has no generative command of the language at all. All hypotheses, thresholds and decision rules were registered before any model was run, and all deviations are logged.
Figures & tables
| all 240 chains | the 144 unparseable | |
|---|---|---|
| echoes the prompt | 127 (52.9%) | 63 (43.8%) |
| carries a Swahili marker | 78 (32.5%) | 44 (30.6%) |
| cut off at the budget | 12 (5.0%) | 6 (4.2%) |
| degenerate repetition | 1 (0.4%) | 0 (0.0%) |
Medical QA
Medical question answering develops and evaluates AI systems that answer clinical, patient-facing, and health-information questions.
See all 98 papers