Multimodal Medical Image Parsing with Task-Specific Low-Rank Adapters

Taha Sabir; Alberto Montero Molina; Thinh Quang Ly; Veronika Dmitrenko; Mohammad Yaqub; Tausifa Jan Saleem

cs.CVOct 14, 2026

Multimodal Medical Image Parsing with Task-Specific Low-Rank Adapters

Authors: Taha Sabir, Alberto Montero Molina, Thinh Quang Ly, Veronika Dmitrenko, Mohammad Yaqub, Tausifa Jan Saleem

Organizations: Department of Computer Science, Gettysburg College, Gettysburg, PA 17325, USA · Higher School of Computing (ESCOM), Instituto Politécnico Nacional (IPN), Mexico City 07738, Mexico · Department of Computer Science and Engineering, Michigan State University, East Lansing, MI 48824, USA · Faculty of Computer Science and Faculty of Economic Sciences, HSE University, Moscow, Russia ·

Abstract

Adapting a single vision-language model to medical image parsing requires training choices that reflect the demands of each task. We study task-specific adaptation with Qwen3.5-9B for FLARE 2026 Task 3, covering six task families and seven imaging modalities. Our system couples a shared, frozen 4-bit backbone with LoRA adapters selected from the input question. We customize adapter ranks and prompts, rebalance rare classification targets, and supplement regression training with dermatology classification examples. On the fixed internal holdout, the task-specific adapters achieve a dataset-macro balanced accuracy of 0.7791 for classification, a dataset-macro micro-F1 of 0.6286 for multi-label classification, a counting mean absolute error (MAE) of 11.73, a dataset-macro detection F1 of 0.8914, a dataset-macro regression MAE of 9.78, and a report-generation CRIMSON score of 0.8204. On the organizer's 1,703-example hidden-validation set, the submitted system achieves 0.8496 classification balanced accuracy, 0.5338 sample-averaged multi-label F1, and 14.5950 regression MAE. The final 30.1 GB container averages 3.1 seconds per case, including model loading, and produces identical outputs across repeated verification runs. The benefits vary across tasks, supporting the use of task-specific adapter configurations and image budgets within a shared medical vision-language model. Code is available at https://github.com/TahaSabir0/FLARE26-Hexapus.

Figures & tables

Figure 1: Routed inference with a shared quantized backbone. The question selects a specialist or the general fallback, together with the image budget and prompt used for answer generation.

Route	Data scope	Pixels	LoRA r/ $\alpha$	Prompt	Output
Classification	Full labeled pool, oversampled	$2048^{2}$	32/64	Bare	Label string
Multi-label classification	Full labeled pool	$2048^{2}$	64/128	Medical	Label set
Counting	Full labeled pool	$2048^{2}$	8/16	Counting	Numeric string
Detection	Full labeled pool	$2048^{2}$	8/16	Bare	Nested boxes
Regression	A $\cup$ B + bcn20000 donor mixture	$2048^{2}$	8/16	Bare	Numeric string
Report generation	A $\cup$ B	$768^{2}$	8/16	Bare	Free text

Table 1: Per-route configuration of the proposed system. Pixel values denote maximum image-area budgets, not forced square resizing.

Figure 2: Image-grouped development partitions and configuration-screening subsets. Percentages are approximate;

A^{\prime}

and

B^{\prime}

each contain 30% of their parent partition. Public-validation images overlapping the organizer training data were removed.

Component	Setting
System	Ubuntu 22.04.5 LTS (A5000 workstation)
CPU / RAM	32 cores / 256 GB (A5000 workstation)
GPUs	NVIDIA RTX A5000 (24 GB)
	NVIDIA RTX PRO 6000 Blackwell (96 GB)
Framework	PyTorch 2.11.0, CUDA 12.8
Training library	LLaMA-Factory 0.9.5

Table 2: Development environment.

Parameter	Setting
Epochs	3
Effective batch size	8
Optimizer	AdamW
Initial learning rate	$2\times 10^{-4}$
Learning rate schedule	Linear decay, 3% warm-up
Maximum sequence length	8,192 tokens

Table 3: Shared training settings for the QLoRA adapters.

Figure 3: Zero-shot and adapted Qwen3.5-9B on internal and public-validation data, using the protocols in Section 3.1 . BA denotes balanced accuracy; multi-label F1 is micro-averaged within each dataset. Task scores are dataset macros, except single-dataset counting and report generation. Report means use the same successfully scored cases for both models; no format-assisted report run was performed. Public validation has no regression cases. Arrows indicate the better direction.

Task	Dataset	Metric	$n$	Resolution	Qwen3.5-9B
Classification	CMMD	BA	357	$2048^{2}$	0.5970
	BCN20000	BA	286	$2048^{2}$	0.6900
	Dental	BA	420	$2048^{2}$	0.6700
	Neojaundice	BA	60	$2048^{2}$	0.8100
	Bone marrow	BA	72	$2048^{2}$	0.8660
	BUS-UCLM	BA	52	$2048^{2}$	0.9190

Table 4: Per-dataset results of the route-matched Qwen3.5-9B adapters on the internal holdout. The report score uses all 781 adapted-model scores; Fig. 3 uses the 780 reports jointly scored with zero-shot.

Task	Metric	$n$	Score
Classification	Balanced accuracy $\uparrow$	1,126	0.8496
Multi-label classification	Sample-averaged F1 $\uparrow$	477	0.5338
Regression	MAE $\downarrow$	100	14.5950

Table 5: Official hidden-validation performance of the submitted Qwen3.5-9B system. Scores are pooled by task.

(a) General versus task-specific adapters
Task	Metric	General	Specialist
Classification	BA $\uparrow$	0.7348	0.7695
Multi-label classification	F1 $\uparrow$	0.6157	0.6130
Counting	MAE $\downarrow$	14.3400	13.4500
Detection	F1 $\uparrow$	0.8876	0.9194
Regression	MAE $\downarrow$	12.1600	10.1800

Table 6: Configuration ablations on the internal holdout. (a) General adapter versus task specialists at matched

768^{2}

input. (b) Resolution comparisons for task specialists. Regression is the mean MAE over two datasets.

Panel of Figure 4 — Figure 4: Representative successes or low-error predictions (left) and failures (right). Classification and detection use non-overlapping validation-public cases; regression uses the internal holdout because public validation has no regression cases. Blue boxes show ground truth and orange boxes show predictions. All examples use task-specific $A\cup B$ adapters.

Measure	Observed	Limit or check
Mean runtime (incl. model loading)	3.1 s/case	$\leq 6.57$ s/case
Container image size	30.1 GB	$\leq 35$ GB
Repeated-run identity	32/32 cases	Identical outputs

Table 7: Final container verification on 32 cases covering all six tasks.

Balanced accuracy	Multi-label F1	Detection F1	MAE
0.8585	0.5310	0.8633	18.5286

Table 8: Organizer-reported final-testing results for Hexapus.

Requirements	Answer
A meaningful title	Yes
The number of authors ( $\leq$ 6)	6
Author affiliations and ORCID	Yes
Corresponding author email is presented	Yes
Validation scores are presented in the abstract	Yes
Introduction includes at least three parts: background, related work, and motivation	Yes

Table 9: Checklist Table. Please fill out this checklist table in the answer column.

Medical Image Analysis

Medical image analysis uses computational methods to interpret clinical images, measure anatomy and disease, and support diagnosis and treatment.

See all 291 papers

Two-Stage Mixture-of-LoRA for Multi-Task Medical Vision-Language Learning
Sep 13, 2026Zhanghao Chen, Yuanyuan Li, Zhenyu Lu +3VLM Adaptation Medical VLMs
School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · School of Biological Science & Medical Engineering, Southeast University, Nanjing 211189, China
Expandable Modality Experts for Medical Vision-Language Learning
Oct 21, 2026DM BAPPY, Eunsu Kim, Hanjoo Lee +2Medical VLMs Continual Learning for VLMs
Korea University · The Catholic University of Korea
Semi-Supervised Few-Shot Adaptation of Vision-Language Models
Mar 3, 2026Julio Silva-Rodríguez, Ender KonukogluVLM Adaptation Few-Shot Image Classification
Computer Vision Lab, ETH Zurich, Zurich, Switzerland

Sep 13, 2026J/K move · Enter open · S save

Two-Stage Mixture-of-LoRA for Multi-Task Medical Vision-Language Learning

Zhanghao Chen, Yuanyuan Li, Zhenyu Lu, Shuo Gao+2

School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · School of Biological Science & Medical Engineering, Southeast University, Nanjing 211189, China

VLM Adaptation Medical VLMs Mixture of Experts Low-Rank Adaptation

Open paper AskPDF arXiv

Medical vision-language models (VLMs) allow a single model to perform clinical image analysis tasks ranging from diagnosis classification to report generation. However, joint adaptation is challenged by heterogeneous output formats, conflicting task gradients, and imbalanced training data. Hence, we present Two-Stage Mixture-of-LoRA, a framework built on MedGemma-1.5-4B. The framework uses a shared-specific Mixture-of-LoRA architecture comprising one shared LoRA and six task-specific expert LoRAs, together with a two-stage training procedure. In Stage 1, we jointly train the shared LoRA and all task-specific expert LoRAs on all tasks. In Stage 2, we first freeze the backbone, the shared LoRA, and all non-target experts, and refine one task expert at a time. Classification and regression then receive an additional modality-balanced continuation, in which smaller modality groups are repeated to match the largest group. In the FLARE 2026 Task 3 test sets, the proposed method achieves 0.85 balanced accuracy for classification, 0.48 micro-F1 for multi-label classification, 0.79 detection F1, and 17.39 regression MAE. Code is available at https://github.com/YuanYL03/MICCAI-FLARE-2026-Challenge-Task3-2D.

Abstract

Medical Image Analysis

Explore similar work

Two-Stage Mixture-of-LoRA for Multi-Task Medical Vision-Language Learning

Expandable Modality Experts for Medical Vision-Language Learning

Semi-Supervised Few-Shot Adaptation of Vision-Language Models