cs.CLNov 16, 2026

Medical question answering collapses in Hausa: a pre-registered pilot on three small open models

Authors: Abbas Sani

Organizations: Mewar University

Abstract

A system that reads nothing and always answers the most frequent letter scores \BaselineP{} on our Hausa medical multiple-choice set. \NBelowBaseline{} of the \NModels{} small open models we evaluate do not beat it. We lead with that because the usual summary of a multilingual result -- a percentage-point drop from English -- does not say whether what survives the drop is performance at all. On a paired English--Hausa set of \NEval{} MedMCQA items, three instruction-tuned models of 3--4B parameters lose between \GapMedGemma{} and \GapQwen{} points of accuracy, every gap significant after Holm correction. Back-translating the Hausa items into English recovers between half and two-thirds of the gap for the two general-purpose models but not demonstrably for the medically adapted one, so the deficit is substantially, though not wholly, a matter of the language of presentation. Answers are also far less stable: agreement across three option orderings falls from \ConsEnQwen{} to \ConsHaQwen{} in the worst case. An exploratory error analysis of \ENRows{} discordant items attributes most of the damage to medical vocabulary lost in translation (\ETerm{}, 95% CI \ETermLo{}--\ETermHi{}), usually in the question stem. A fourth registered hypothesis proposed closing the gap with self-generated preference pairs. Its pilot gate found nothing to train on: asked to reason in Hausa rather than to pick a letter, the model echoes the prompt back, switches into Swahili, or simply stops without answering; only \GateParse{} of chains are parseable. We report that intervention as not run, which is itself the sharper result -- a forced-choice benchmark cannot see a model that has no generative command of the language at all. All hypotheses, thresholds and decision rules were registered before any model was run, and all deviations are logged.

Figures & tables

Explore similar work

CardsList