Conditional Rank Allocation for Taxonomy-Aware Medical Language Model Adaptation

Guangyuan Dong; Ziwei Hong; Xuehao Zhou; Zidong Yu; Bingchen Liu; Kehan Liu; Chuang Liu; Rong Fu

cs.AIOct 5, 2026

Conditional Rank Allocation for Taxonomy-Aware Medical Language Model Adaptation

Authors: Guangyuan Dong, Ziwei Hong, Xuehao Zhou, Zidong Yu, Bingchen Liu, Kehan Liu, Chuang Liu, Rong Fu, +1 more

Organizations: National University of Singapore, Singapore · University of Pennsylvania, USA · Department of Urology, Shanghai General Hospital, Shanghai, China · Shanghai Jiao Tong University School of Medicine, Shanghai, China · Syracuse University, USA · Shandong University, China

Abstract

Medical question answering spans specialties and clinical operations that may benefit from different adaptation directions. We propose ARBOR, a parameter-efficient method that selects rank-one components from a shared low-rank basis for each question. An additive gate combines question representations, specialty tags, operation tags, and their interaction; a learned coefficient scales the adapter residual. An illustrative separation under orthogonal, equiprobable subtasks shows how conditional selection can avoid an approximation floor faced by a fixed update with the same active rank. This result motivates the design without asserting a corresponding bound for medical corpora. On Qwen3-8B across CMB, CMExam, MedQA, and MedMCQA, five-seed experiments yield 69.69% mean accuracy across benchmarks, exceeding LoRA r16 and MoELoRA by 1.26 and 1.30 percentage points, respectively. The reported advantage over LoRA r16 increases from 0.08 to 1.94 points as training expands from one to seven specialties. Tag perturbations and atom masking support the usefulness of clinical routing, while atom clusters align with the supplied specialty labels (adjusted Rand index 0.62). Calibration, transfer, and measured costs further characterize the method. These findings support structured conditional adaptation for medical QA, while leaving clinical safety and broader deployment untested.

Figures & tables

Fig. 1: Arbor overview. Clinical priors and frozen question features feed four additive routing branches. Straight-through top-

\kappa

selection activates four of sixteen stored rank-one atoms; a learned coefficient scales their contribution alongside the frozen backbone. Mask and scale are shared across target layers. The training objective and question-level routing record are shown below. Highlighted mask positions are schematic.

Axis	Accuracy	Coverage	Majority	Agreement
Specialty (7)	91.4%	100%	28.6%	0.86
Operation (8)	84.6%	100%	24.2%	0.79

TABLE I: Clinical-prior validation on 500 annotated questions. Agreement is inter-annotator

\kappa

Method	Trainable (M)	CMB	CMExam	MedQA	MedMCQA	Macro
Frozen Qwen3-8B	–	71.68	68.95	62.86	54.86	64.59
LoRA r4	10.91	$77.62\pm.21$	$75.67\pm.24$	$62.91\pm.42$	$57.19\pm.38$	$68.35\pm.18$
LoRA r16	43.65	$77.79\pm.19$	$75.98\pm.22$	$62.60\pm.45$	$57.34\pm.41$	$68.43\pm.21$
LoRA r24	65.47	$77.95\pm.23$	$75.88\pm.25$	$62.71\pm.44$	$57.62\pm.37$	$68.54\pm.22$
DoRA r24	66.87	$78.48\pm.20$	$75.86\pm.28$	$62.19\pm.47$	$56.90\pm.44$	$68.36\pm.17$
HydraLoRA	44.90	$78.02\pm.22$	$75.92\pm.24$	$63.15\pm.46$	$57.34\pm.40$	$68.61\pm.20$

TABLE II: Qwen3-8B medical QA accuracy (%). Trainable methods share the CMB-source protocol; values are mean

\pm

standard deviation over five seeds. Macro is the unweighted benchmark average. Bold marks the best trainable method. Adapter libraries compose pretrained adapters and are reported separately without across-seed uncertainty.

Fig. 2: Accuracy improvements under matched source training. (a) Benchmark gains over three baselines share a 0–2.5 percentage-point radial scale, with zero at the inner ring; the center reports Arbor ’s macro accuracy. (b) Paired macro differences against seven trainable baselines, with reported 95% intervals across five training seeds. All intervals exclude zero.

Fig. 3: Clinical routing and component analysis. (a) Tag-source controls with other components held fixed. (b) Aggregate accuracy under test-time tag noise (25 runs per level), with the reported training-heterogeneity endpoints shown separately. (c) Component ablations relative to full Arbor ; annotation uncertainty denotes standard deviation across five seeds. (d) Alignment with supplied specialty labels. Only available numerical summaries are shown; heterogeneity endpoints are not interpolated.

Method	Params	Active	Train	Latency	Memory
	(M)	rank	(min)	(ms)	(GB)
LoRA r4	10.91	4	38	416	19.6
LoRA r16	43.65	16	41	418	19.8
LoRA r24	65.47	24	44	421	20.0
DoRA	66.87	24	58	435	21.0
HydraLoRA	44.90	16	49	452	20.6

TABLE III: Reported Qwen3-8B costs on one A100-80GB. Parameters are total trainable parameters; memory is peak inference memory. Latency is per question under greedy decoding.

Explore similar work

Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering
Jun 30, 2026Hao Gong, Ruilin Gong, Yining HuangMedical Visual Question Answering Clinical Reasoning Training
University of Chinese Academy of Sciences · Henan University of Technology · Meta Emergence Laboratory
TriageRA-CCF: Source-Side Clinical Confidence and Coverage Signals for Adaptive Rank Budgeting in Medical LLMs
Jun 28, 2026Shucan Ji, Yining Huang, Hongliang GuoLarge Language Model Adaptation Token Budget Allocation
College of Computer Science, Sichuan University, Chengdu, China · Meta Emergence Laboratory
Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection
Jul 31, 2026Chaimae Abouzahir, Musa Khan, Hala Ali-Hassan +7Multilingual Healthcare Q&A Large Language Model Adaptation
New York University Abu Dhabi · Cleveland Clinic Abu Dhabi

Jun 30, 2026, cs.CLJ/K move · Enter open · S save

Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering

Hao Gong, Ruilin Gong, Yining Huang

University of Chinese Academy of Sciences · Henan University of Technology · Meta Emergence Laboratory

Medical Visual Question Answering Clinical Reasoning Training Question-Answering Benchmarks Question Clinical Parameter-Efficient Adaptation

Open paper AskPDF arXiv

Medical multiple-choice question answering requires parameter-efficient adaptation across heterogeneous knowledge domains and reasoning operations. A medication question, a diagnostic decision, a public-health item, and a nursing-action item may require different low-rank updates, while some recall items should preserve the base model's representation with only mild adapter intervention. We propose BiRG-LoRA, a single-adapter rank-gated LoRA method for medical question answering. BiRG-LoRA keeps one LoRA module per target layer but makes its rank dimension input-conditioned: for each question, a biaxial gate combines hidden semantic evidence with specialty/profession priors, clinical-operation priors, and their interaction to select a sparse top-

k

subset of rank atoms. A scalar injection coefficient further controls the strength of the selected adapter update. Under a matched Qwen3-8B CMB-source protocol, BiRG-LoRA achieves the highest four-benchmark macro-average accuracy among trainable PEFT baselines and matched routing controls: 69.31% averaged over CMB, CMExam, MedQA, and MedMCQA. It improves over MoELoRA by 0.89 percentage points while using 28.1% fewer trainable parameters; a paired, benchmark-stratified bootstrap over final predictions gives a 95% confidence interval of [0.42, 1.37] for this macro-average gain. Basic controls show that BiRG-LoRA also improves over vanilla LoRA r16 and active-rank-matched LoRA r4 by 0.83 macro points, and an evaluation-time weak-axis perturbation check suggests that performance is not brittle to moderate tag noise. The results support a bounded claim: clinically structured rank allocation improves cross-benchmark medical QA under a matched single-seed protocol, while training-seed variance remains future work.

Abstract

Explore similar work

Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering

TriageRA-CCF: Source-Side Clinical Confidence and Coverage Signals for Adaptive Rank Budgeting in Medical LLMs

Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection