PACE-MoE: Adaptive Domain Sampling and Domain-Specific LoRA for Medical Visual Grounding

Zhenxuan Wang; Jisheng Dang; Ronghao Lin; Bimei Wang; Jingwen Zhao; Wenjun Ma; Bin Hu; Qi Tian

cs.CVOct 20, 2026

PACE-MoE: Adaptive Domain Sampling and Domain-Specific LoRA for Medical Visual Grounding

Authors: Zhenxuan Wang, Jisheng Dang, Ronghao Lin, Bimei Wang, Jingwen Zhao, Wenjun Ma, Bin Hu, Qi Tian,

Organizations: School of Information Science & Engineering, Lanzhou University, Lanzhou 730000, China · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China · Cloud and AI BU, Huawei, Shenzhen, Guangdong 518129, China ·

Abstract

Medical visual grounding requires models to align clinical descriptions with spatial evidence across heterogeneous imaging domains. However, existing multi-domain post-training strategies typically rely on fixed domain sampling and shared parameter updates, overlooking the need to adapt both which domains receive training and where their updates are applied. To address this issue, we propose PACE-MoE, a policy-guided framework that jointly performs adaptive domain sampling and domain-specific adaptation for medical visual grounding. PACE dynamically assesses the training value of each imaging domain from current-policy feedback and progressively adjusts domain coverage through domain admission, replay, and final all-domain balancing. Meanwhile, domain-specific LoRA adapters route updates to the corresponding imaging domains while preserving the pretrained backbone and visual encoder. This coordinated design aligns training-data allocation with parameter adaptation without modifying the original reward formulation or Group Relative Policy Optimization (GRPO) objective. Experiments across seven medical imaging domains show that PACE-MoE consistently improves grounding performance over fixed-domain training, achieving a 3.64% relative improvement in mean box IoU over the Vanilla baseline. Under matched domain-specific adapter architectures, adaptive sampling alone further improves mean box IoU by 2.07%, demonstrating the benefit of policy-guided domain allocation. These results establish coordinated data-parameter routing as an effective strategy for multi-domain medical visual grounding.

Figures & tables

Fig. 1: Motivation for domain-aware post-training. (a) A fixed data mixture does not respond to the current model, and one shared LoRA adapter receives updates from all domains. (b) PACE-MoE uses policy feedback to adapt the sampling distribution

p_{s}(g\mid\mathcal{S}_{t})

. The selected domain label

g

also identifies the adapter to update. The pretrained backbone remains shared and frozen.

Fig. 2: Overview of PACE-MoE. (a) Architecture of PACE-MoE. Current-policy rollouts on a fixed seven-domain probe buffer estimate reward, within-query dispersion, and output validity. These statistics update the domain sampling distribution through domain admission, replay, and a final all-domain stage. The sampled label

g_{t,b}

selects a query from

\mathcal{D}_{g}

and activates the corresponding rank-16 FFN LoRA adapter

\phi_{g}

. (b) Optimization pipeline. The selected adapter is updated using the shared spatial reward and GRPO objective, while the MedReasoner backbone and MedSAM2 remain frozen. At each probe event, the updated policy is evaluated again to determine the next domain sampling distribution.

Method	Adaptive sampling	Domain-specific adapter	Reward changed	Replay	Advantage reweighting
Vanilla GRPO	$\times$	$\times$	$\times$	$\times$	$\times$
Balanced GRPO	$\times$	$\times$	$\times$	$\times$	$\times$
Fixed curriculum	$\times$	$\times$	$\times$	$\checkmark$	$\times$
DRPO-style	$\times$	$\times$	$\times$	$\times$	$\checkmark$
Domain-MoE	$\times$	$\checkmark$	$\times$	$\times$	$\times$
PACE-GRPO (Ours)	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$

TABLE I: Comparison of domain sampling and parameter-sharing strategies.

Item	Value
Initialization	Released MedReasoner-Soft-7B
Reasoner backbone / segmenter	Lingshu-7B / frozen MedSAM2
RL implementation	veRL + vLLM
Updates / queries per update	25 / 4
Responses per query	2
Probe responses per query $K_{p}$	2

TABLE II: Configuration of the 25-update experiments.

Fig. 3: Qualitative comparison across seven imaging domains using the same test image and query for each pair. Top: Vanilla-25. Bottom: PACE-MoE. Solid green and dashed orange boxes denote ground truth and predictions, respectively. CT (a) and X-ray (c) retain the full image; the other columns use identical crops for both methods that retain all compared boxes. IoU is reported as a percentage and computed in the original image coordinates.

Method	$N$	Box IoU $\uparrow$	pDice ${}_{\rm rel}\uparrow$	pDice ${}_{\rm std}\uparrow$
Released Soft-7B [ 1 ]	–	34.6734	30.4282	21.8461
Vanilla-25 [ 1 ]	8	$34.8485\pm 0.1502$	$30.5631\pm 0.0632$	$21.8746\pm 0.0565$
PACE-25	8	$35.2297\pm 0.3645$	$30.8478\pm 0.2076$	$22.1075\pm 0.1303$
MoE-Vanilla-25	5	$35.3871\pm 0.1226$	$30.9523\pm 0.2502$	$21.9571\pm 0.0899$
PACE-MoE-25 (Ours)	5	$\mathbf{36.1182\pm 0.1341}$	$\mathbf{31.6718\pm 0.2877}$	$\mathbf{22.2067\pm 0.0929}$

TABLE III: Multi-seed system comparison and sampling controls after 25 updates. Shared-adapter rows report eight seeds; domain-specific rows report five seeds (1, 18, 42, 123, and 456). Grounding scores are mean

\pm

sample standard deviation, in %. Cross-architecture rows compare complete systems; within-architecture pairs assess adaptive sampling.

Method	Box IoU $\uparrow$	pDice ${}_{\rm rel}\uparrow$	pDice ${}_{\rm std}\uparrow$
Released Soft-7B [ 1 ]	34.6734	30.4282	21.8461
Vanilla-25 [ 1 ]	$34.8485\pm 0.1502$	$30.5631\pm 0.0632$	$21.8746\pm 0.0565$
PACE-25 (Ours)	$35.2297\pm 0.3645$	$30.8478\pm 0.2076$	$22.1075\pm 0.1303$
Relative gain (%)	$+1.09$	$+0.93$	$+1.06$
Paired $t$ -test $p$	0.0023	0.0023	0.0003

TABLE IV: Shared-adapter results after 25 updates on 1,297 test cases. Scores (%): mean

\pm

sample standard deviation over 8 seeds (1, 18, 36, 42, 123, 456, 999, 2026). Gains are relative to Vanilla-25; paired tests use original scores.

Panel of Figure 8 — Fig. 4: Paired seed-level changes from fixed to adaptive domain sampling. Scores are in %. Each line connects runs with the same seed and initialization. All 24 metric–seed pairs improve with adaptive sampling.

Method	Box IoU $\uparrow$	pDice ${}_{\rm rel}\uparrow$	pDice ${}_{\rm std}\uparrow$
Released Soft-7B [ 1 ]	34.6734	30.4282	21.8461
Vanilla-25 [ 1 ]	35.0129	30.5933	21.9745
Balanced-25	34.9135	30.6818	22.0257
Fixed-25	34.7962	30.7027	22.0253
DRPO-style-25 [ 23 ]	34.6774	30.7705	21.8747
PACE-NoReplay-25	34.8871	30.8690	21.9824

TABLE V: Seed-42 control study. Grounding scores are in %. Every adapted method starts from the same checkpoint and uses 25 optimizer updates.

Panel of Figure 10 — Fig. 5: Seed-42 relative box-IoU gains by imaging domain, using each domain’s Vanilla-25 score as the baseline. Six domains improve; MRI changes by $-0.03\%$ .

Imaging domain	$N$	Vanilla-25	PACE-25	Gain (%)
CT	528	25.5192	26.6257	+4.34
Dermoscopy	51	55.7754	56.8357	+1.90
Mammography	23	22.3928	24.6799	+10.21
Endoscopy	85	44.5062	45.6580	+2.59
MRI	397	36.6831	36.6709	-0.03
Ultrasound	53	36.1560	37.8294	+4.63

TABLE VI: Seed-42 domain-level box IoU (%) on the 1,297-case test split. Gain is the relative change from Vanilla-25, in %. The macro average weights the seven domains equally, whereas the overall result is sample-weighted.

Method	Box IoU $\uparrow$	pDice ${}_{\rm rel}\uparrow$	pDice ${}_{\rm std}\uparrow$
Vanilla	34.9320 ${\pm}$ 0.1069	30.5628 ${\pm}$ 0.0612	21.8899 ${\pm}$ 0.0774
w/o replay	34.9364 ${\pm}$ 0.0970	30.7735 ${\pm}$ 0.0837	21.8937 ${\pm}$ 0.0775
w/o final balance	34.9302 ${\pm}$ 0.1692	30.6962 ${\pm}$ 0.1120	21.8361 ${\pm}$ 0.0855
PACE (full)	35.4867 ${\pm}$ 0.3088	30.9714 ${\pm}$ 0.0993	22.2168 ${\pm}$ 0.1527

TABLE VII: PACE ablations after 25 updates across seeds 42, 1, and 18. Grounding scores are mean

\pm

standard deviation, in %.

Panel of Figure 13 — Fig. 6: Paired ablations across seeds 42, 1, and 18: mean $\pm 1$ sample standard deviation of paired box-IoU changes, both normalized by the common Vanilla-25 mean (34.9320%) and expressed as percentages.

Method	IoU $\uparrow$	pDice ${}_{\rm rel}\uparrow$	pDice ${}_{\rm std}\uparrow$
Released Soft-7B [ 1 ]	34.6734	30.4282	21.8461
Vanilla-200 [ 1 ]	34.4138	30.2541	22.0571
PACE-200	34.5867	30.3731	21.8042
PACE-MoE-200	34.6691	30.3822	21.9208

TABLE VIII: Seed-42 extended-training analysis after 200 post-training updates. Grounding scores are in %.

Medical Image Grounding

Medical image grounding links clinical phrases, questions, and diagnoses to the image regions or slices that support them.

See all 80 papers

Mar 30, 2026

MedLoc-R1: Performance-Aware Curriculum Reward Scheduling for GRPO-Based Medical Visual Grounding

Medical visual grounding serves as a crucial foundation for fine-grained multimodal reasoning and interpretable clinical decision support. Despite recent advances in reinforcement learning (RL) for grounding tasks, existing approaches such as Group Relative Policy Optimization~(GRPO) suffer from severe reward sparsity when directly applied to medical images, primarily due to the inherent difficulty of localizing small or ambiguous regions of interest, which is further exacerbated by the rigid and suboptimal nature of fixed IoU-based reward schemes in RL. This leads to vanishing policy gradients and stagnated optimization, particularly during early training. To address this challenge, we propose MedLoc-R1, a performance-aware reward scheduling framework that progressively tightens the reward criterion in accordance with model readiness. MedLoc-R1 introduces a sliding-window performance tracker and a multi-condition update rule that automatically adjust the reward schedule from dense, easily obtainable signals to stricter, fine-grained localization requirements, while preserving the favorable properties of GRPO without introducing auxiliary networks or additional gradient paths. Experiments on three medical visual grounding benchmarks demonstrate that MedLoc-R1 consistently improves both localization accuracy and training stability over GRPO-based baselines. Our framework offers a general, lightweight, and effective solution for RL-based grounding in high-stakes medical applications. Code & checkpoints are available at \hyperlink{}{https://github.com/MembrAI/MedLoc-R1}.

Guangjing Yang, Ziyuan Qin, Chaoran Zhang +6

Beijing University of Posts and Telecommunications · Emory University · Peking University +1

Medical Image Grounding Reward Shaping Curriculum RL Sparse-Reward RL Group Relative Policy Optimization

Ask PDF

Feb 26, 2026

pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation

Parameter-efficient fine-tuning has demonstrated promising results across various visual adaptation tasks, such as classification and segmentation. Typically, prompt tuning techniques have harnessed knowledge from a single pre-trained model, whether from a general or a specialized medical domain. However, this approach typically overlooks the potential synergies that could arise from integrating diverse domain knowledge within the same tuning process. In this work, we propose a novel Mixture-of-Experts prompt tuning method called pMoE, which leverages the strengths of multiple expert domains through expert-specialized prompt tokens and the learnable dispatcher, effectively combining their expertise in a unified model framework. Our pMoE introduces expert-specific prompt tokens and utilizes a dynamic token dispatching mechanism at various prompt layers to optimize the contribution of each domain expert during the adaptation phase. By incorporating both domain knowledge from diverse experts, the proposed pMoE significantly enhances the model's versatility and applicability to a broad spectrum of tasks. We conduct extensive experiments across 47 adaptation tasks, including both classification and segmentation in general and medical domains. The results demonstrate that our pMoE not only achieves superior performance with a large margin of improvements but also offers an optimal trade-off between computational efficiency and adaptation effectiveness compared to existing methods.

Shentong Mo, Xufang Luo, Dongsheng Li

Carnegie Mellon University · Microsoft Research

Visual Prompt Learning Vision Foundation Model Adaptation Parameter-Efficient Fine-Tuning Mixture-of-Experts Models Multi-Source Domain Adaptation

Ask PDF

Apr 19, 2026

From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation

Visual prompting has emerged as a powerful method for adapting pre-trained models to new domains without updating model parameters. However, existing prompting methods typically optimize a single prompt per domain and apply it uniformly to all inputs, limiting their ability to generalize under intra and inter-domain variability, which is especially critical in the medical field. To address this, we propose APEX, an Adaptive Prompt EXtraction framework that retrieves input-specific prompts from a learnable prompt memory. The memory stores diverse, domain-discriminative prompt representations and is queried via domain features extracted from the Fourier spectrum. To learn robust and discriminative domain features, we introduce a novel Low-Frequency Feature Contrastive (LFC) learning framework that clusters representations from the same domain while separating those from different domains. Extensive experiments on two medical segmentation tasks demonstrate that APEX significantly improves generalization across both seen and unseen domains. Furthermore, it complements any existing backbones and consistently enhances performance, confirming its effectiveness as a plug-and-play prompting solution in medical fields. The code is available at https://github.com/cetinkayaevren/apex/

Evren Çetinkaya, Sangmin Lee, Jung Uk Kim +2

Technical University of Munich · Korea University · Kyung Hee University +1

Domain Generalization Medical Image Segmentation Visual Prompt Learning

Ask PDF

Abstract

Medical Image Grounding

Explore similar work

MedLoc-R1: Performance-Aware Curriculum Reward Scheduling for GRPO-Based Medical Visual Grounding

pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation

From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation