Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosis

Pei Liu; Xiangxiang Zeng; Tengfei Ma; Yucheng Xing; Xuanbai Ren; Yiping Liu

cs.CVMar 11, 2026

Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosis

Authors: Pei Liu, Xiangxiang Zeng, Tengfei Ma, Yucheng Xing, Xuanbai Ren, Yiping Liu

Organizations: College of Computer Science and Electronic Engineering, Hunan University · National University of Singapore

Abstract

Whole-Slide Images (WSIs) are widely used for estimating the prognosis of cancer patients. Current studies generally follow a cancer-specific learning paradigm. However, the available training samples for one cancer type are usually scarce in pathology. Consequently, the model often struggles to learn generalizable knowledge, thus performing worse on the tumor samples with inherent high heterogeneity. Although multi-cancer joint learning and knowledge transfer approaches have been explored recently to address it, they either rely on large-scale joint training or extensive inference across multiple models, posing new challenges in computational efficiency. To this end, this paper proposes a new scheme, Sparse Task Vector Mixup with Hypernetworks (STEPH). Unlike previous ones, it efficiently absorbs generalizable knowledge from other cancers for the target via model merging: i) applying task vector mixup to each source-target pair and then ii) sparsely aggregating task vector mixtures to obtain an improved target model, driven by hypernetworks. Extensive experiments on 13 cancer datasets show that STEPH improves over cancer-specific learning and an existing knowledge transfer baseline by 5.14% and 2.01%, respectively. Moreover, it is a more efficient solution for learning prognostic knowledge from other cancers, without requiring large-scale joint training or extensive multi-model inference. Code is publicly available at https://github.com/liupei101/STEPH.

Figures & tables

Figure 1: Current learning paradigms of modeling WSIs for survival analysis . Unlike previous approaches, the proposed STEPH efficiently utilizes the generalizable prognostic knowledge from other cancers by model merging.

Figure 2: Performance of different learning paradigms in WSI prognosis. We compare STEPH with cancer-specific learning and a representation transfer-based solution. Multi-cancer joint learning is absent as its training overheads and hardware specifications are orders of magnitude greater than those of others.

Figure 3: Sparse Task Vector Mixup with Hypernetworks ( STEPH ) for efficient knowledge transfer in WSI prognosis. After computing task vectors, STEPH first applies mixup to each paired $(\tau_{t},\tau_{s})$ to absorb prognostic knowledge from cross-cancer models. Then, the most beneficial mixtures are selected and aggregated to derive $\mathcal{M}_{t}^{*}$ for prediction. Hypernetworks drive these steps by steering task vectors.

Figure 4: Loss landscape of task vector mixup on

t

. Red vectors in (a) and (b) depict the range of TVM with lower loss. Figure (c) shows the minimum loss that can be reached by TVM.

Figure 5: Alignment of

\tau_{\text{mix}}

with

\tau_{t}

in dominant subspaces (95%) for the two layers of MIL encoder, measured by SAR [ 18 ] .

Method

BRCA

KIPAN

LUNG

GBM LGG

COAD READ

STES

UCEC

HNSC

SKCM

BLCA

LIHC

CESC

SARC

Avg.

- Traditional cancer-specific training on

\mathcal{M}_{t}

0.6648

0.8094

0.5496

0.7756

0.6725

0.6648

0.7098

0.6201

0.5708

0.6438

0.7265

0.6500

0.5312

Vanilla

(

\pm

0.032)

(

\pm

0.013)

(

\pm

0.034)

(

\pm

0.020)

(

\pm

0.028)

(

\pm

0.040)

(

\pm

0.043)

(

\pm

0.047)

(

\pm

0.037)

(

\pm

0.015)

(

\pm

0.048)

(

\pm

0.058)

(

\pm

0.056)

0.6609

Fine-tuned

0.6723

0.8080

0.5463

0.7744

0.6753

0.6694

0.7136

0.6207

0.5682

0.6384

0.7253

0.6367

0.5462

0.6611

(

\pm

0.021)

(

\pm

0.014)

(

\pm

0.034)

(

\pm

0.020)

(

\pm

0.028)

(

\pm

0.044)

(

\pm

0.039)

(

\pm

0.043)

(

\pm

0.032)

(

\pm

0.021)

(

\pm

0.046)

(

\pm

0.058)

(

\pm

0.030)

- Representation-based knowledge transfer from

\mathcal{M}_{s}

Table 1: Comparison with baselines in WSI prognosis tasks. It reports the mean and standard deviation of C-Index in five-fold cross-validation. Best in bold, second best underline.

Figure 6: More results of TVM loss landscape .

Figure 7: More results of the alignment of

\tau_{\text{mix}}

with

\tau_{t}

Figure 8: Visualization of TVM trajectory in training. STEPH is used to train the paired

\tau_{t}

and

\tau_{s}

(same as those in Fig. 4 and Fig. 6 ). A deeper point color indicates a larger epoch number.

Figure 9: Dynamics of

\lambda

and

w

in training. This example shows a case where

t=

BRCA and 12 cancer-specific models (

\mathcal{M}_{s}

) are merged with

\mathcal{M}_{t}

by STEPH to obtain

\mathcal{M}_{t}^{*}

Ablation		Avg.
- On task vector mixup
w/o mixup	fix $\lambda$ = 0 (only $\{\tau_{s_{i}}\}$ )	0.6860
	fix $\lambda$ = 0 ( $\tau_{t}\in\{\tau_{s_{i}}\}$ )	0.6895
	fix $\lambda$ = 1 (only $\tau_{t}$ )	0.6851
w/ mixup	trainable param. $\lambda$	0.6921
w/ mixup	hypernetwork-driven $\lambda$	0.6949

Table 2: Ablation study on STEPH .

Method		Avg.
AdaMerging	vanilla (w/ trainable param. $w$ )	0.5689
AdaMerging	w/ our hypernetwork-driven $w$	0.6877
TIES	w/ AM	0.6396
TIES	w/ our hypernetwork-driven $w$	0.6802
Surgery	w/ AM	0.5943
Surgery	w/ our hypernetwork-driven $w$	0.6668

Table 3: Our hypernetwork-driven aggregation scheme for existing model merging methods.

$\beta$	$0$	$0.01$	$0.05$	$0.1$
Avg.	0.6927	0.6933	0.6949	0.6909

Table 4: STEPH with different

\beta

(coefficient of

\mathcal{L}_{\text{mix}}

$\gamma$	$0$	$0.0025$	$0.005$	$0.01$	0.05
Avg.	0.6879	0.6896	0.6896	0.6865	0.6881

Table 5: STEPH with different

\gamma

(coefficient of

\mathcal{L}_{\text{agg}}

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# Patients	# WSIs	# Patches ( avg. )	# Patches ( max )	Survival rate
BRCA	1,035	1,106	10,694	58,176	92.4%
KIPAN	880	910	10,117	72,795	83.6%
LUNG	848	924	13,199	220,294	78.2%
GBMLGG	841	1,623	16,209	216,885	52.4%
COADREAD	568	577	6,458	38,322	86.6%
STES	512	539	8,429	31,458	72.7%

Appendix

Table S1 : Characteristics of WSI datasets .

Figure S1 : Dynamics of

\lambda

and

w

in training (

t=

KIPAN).

Figure S2 : Dynamics of

\lambda

and

w

in training (

t=

LUNG).

Method	BRCA	KIPAN	LUNG	GBM LGG	COAD READ	STES	UCEC	HNSC	SKCM	BLCA	LIHC	CESC	SARC	Avg.
ABMIL	0.6648	0.8094	0.5496	0.7756	0.6725	0.6648	0.7098	0.6201	0.5708	0.6438	0.7265	0.6500	0.5312	0.6609
ABMIL	( $\pm$ 0.032)	( $\pm$ 0.013)	( $\pm$ 0.034)	( $\pm$ 0.020)	( $\pm$ 0.028)	( $\pm$ 0.040)	( $\pm$ 0.043)	( $\pm$ 0.047)	( $\pm$ 0.037)	( $\pm$ 0.015)	( $\pm$ 0.048)	( $\pm$ 0.058)	( $\pm$ 0.056)	0.6609
TransMIL	0.6802	0.8121	0.5669	0.7873	0.6651	0.6582	0.7471	0.6315	0.5786	0.6173	0.7385	0.6365	0.5152	0.6642
TransMIL	( $\pm$ 0.026)	( $\pm$ 0.018)	( $\pm$ 0.021)	( $\pm$ 0.019)	( $\pm$ 0.059)	( $\pm$ 0.056)	( $\pm$ 0.072)	( $\pm$ 0.014)	( $\pm$ 0.039)	( $\pm$ 0.029)	( $\pm$ 0.044)	( $\pm$ 0.068)	( $\pm$ 0.042)	0.6642
ILRA	0.6645	0.8017	0.5374	0.7739	0.6147	0.6451	0.7442	0.5262	0.5586	0.6608	0.7231	0.6336	0.5351	0.6476
ILRA	( $\pm$ 0.026)	( $\pm$ 0.016)	( $\pm$ 0.062)	( $\pm$ 0.016)	( $\pm$ 0.041)	( $\pm$ 0.053)	( $\pm$ 0.057)	( $\pm$ 0.024)	( $\pm$ 0.042)	( $\pm$ 0.046)	( $\pm$ 0.044)	( $\pm$ 0.016)	( $\pm$ 0.055)	0.6476

Appendix

Table S2 : Comparison with MIL networks for WSI-based survival prediction on 13 cancer datasets.

Figure S3 : Additional results of task vector mixup .

$K$	$1$	$3$	$5$	$7$	$9$	$11$	$13$
Avg.	0.6882	0.6929	0.6949	0.6935	0.6932	0.6926	0.6912

Appendix

Table S3 : STEPH with different

K

(the number of selected task vector mixtures in sparse aggregation).

Method	Learnable param.	Test-time inference
Method	(M)	(GFLOPs)
ABMIL	1.20	23.6
TransMIL	2.95	60.3
ILRA	4.21	55.3
R 2 T-MIL	2.98	40.7
Patch-GCN	1.69	21.7

Appendix

Table S4 : Comparison with MIL networks in terms of model efficiency . A WSI with 10,000 patches is used in model inference.

Ablation	BRCA	KIPAN	LUNG	GBM LGG	COAD READ	STES	UCEC	HNSC	SKCM	BLCA	LIHC	CESC	SARC	Avg.
w/o mixup	0.7183	0.8264	0.5876	0.7745	0.6971	0.6781	0.7840	0.6420	0.5954	0.6387	0.7400	0.6886	0.5358	0.6851
( $\lambda=1$ )	( $\pm$ 0.051)	( $\pm$ 0.016)	( $\pm$ 0.039)	( $\pm$ 0.024)	( $\pm$ 0.024)	( $\pm$ 0.030)	( $\pm$ 0.069)	( $\pm$ 0.022)	( $\pm$ 0.037)	( $\pm$ 0.038)	( $\pm$ 0.046)	( $\pm$ 0.043)	( $\pm$ 0.070)	0.6851
w/o $\mathcal{H}_{\text{mix}}$	0.7323	0.8152	0.5901	0.7728	0.7044	0.6645	0.7852	0.6548	0.5903	0.6404	0.7579	0.6951	0.5944	0.6921
(param. $\lambda$ )	( $\pm$ 0.061)	( $\pm$ 0.019)	( $\pm$ 0.040)	( $\pm$ 0.024)	( $\pm$ 0.027)	( $\pm$ 0.047)	( $\pm$ 0.073)	( $\pm$ 0.041)	( $\pm$ 0.031)	( $\pm$ 0.038)	( $\pm$ 0.034)	( $\pm$ 0.066)	( $\pm$ 0.066)	0.6921
w/o sparse	0.7263	0.8212	0.5865	0.7746	0.7009	0.6640	0.7803	0.6526	0.5891	0.6433	0.7541	0.6937	0.5994	0.6912
w/o sparse	( $\pm$ 0.052)	( $\pm$ 0.016)	( $\pm$ 0.043)	( $\pm$ 0.021)	( $\pm$ 0.027)	( $\pm$ 0.055)	( $\pm$ 0.075)	( $\pm$ 0.038)	( $\pm$ 0.024)	( $\pm$ 0.041)	( $\pm$ 0.034)	( $\pm$ 0.063)	( $\pm$ 0.086)	0.6912

Appendix

Table S5 : Complete ablation results on 13 datasets.

Figure S4 : Visualization of attention maps given by models. The two WSIs are from the test set of COADREAD. Tumor (TUM).

Method	Model Avg.	AdaMerging	TIES AM	Surgery AM	Iso-C AM	STEPH
Time (s)	0.00	78.08	78.42	78.00	78.50	80.45
Time (s)	( $\pm$ 0.00)	( $\pm$ 0.92)	( $\pm$ 2.20)	( $\pm$ 1.34)	( $\pm$ 1.54)	( $\pm$ 1.63)

Appendix

Table S6 : Mean training time per epoch on BRCA.

$m$ ( $K$ )	6	7	8	9	10	11	12
Time (s)	72.91	72.98	74.51	75.62	75.60	75.74	76.69
Time (s)	( $\pm$ 1.20)	( $\pm$ 1.42)	( $\pm$ 3.00)	( $\pm$ 1.54)	( $\pm$ 1.62)	( $\pm$ 2.18)	( $\pm$ 1.79)

Appendix

Table S7 : Mean training time per epoch on BRCA with different numbers of source models (

m

Figure S5 : Loss of TVM between BLCA and others (w/ top

w

Whole-Slide Image Analysis

Whole-slide image analysis uses AI to interpret gigapixel pathology slides for diagnosis, prognosis, tissue mapping, and report generation.

See all 214 papers

SlideMix: Enhancing Whole Slide Image Analysis via Multimodal Shuffling
Aug 31, 2026Chad Wong, Sicheng Chen, Tianyi Zhang +4Data Augmentation Histopathology Image Classification
1Nhu Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Electrical & Computer Engineering, National University of Singapore · 3PuzzleLogic Pte Ltd, Singapore 229594, Singapore
Exploiting Spatial Structure for Transductive Few-Shot Classification of Whole-Slide Images
Sep 25, 2026Tiffanie Godelaine, Manon Dausort, Karim El Khoury +3Histopathology Image Classification Few-Shot Image Classification
ICTEAM, Universit´e Catholique de Louvain, Belgium
Hierarchical Multi-Scale Graph Learning with Knowledge-Guided Attention for Whole-Slide Image Survival Analysis
Feb 26, 2026Bin Xu, Yufei Zhou, Boling Song +6Hierarchical GNNs Cancer Survival Prediction
Nanjing University of Information Science and Technology, Nanjing, China · Case Western Reserve University, Cleveland, OH, USA · Georgia Institute of Technology and Emory University, Atlanta, GA, USA +4

Aug 31, 2026J/K move · Enter open · S save

SlideMix: Enhancing Whole Slide Image Analysis via Multimodal Shuffling

Chad Wong, Sicheng Chen, Tianyi Zhang, Enhui Chai+3

1Nhu Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Electrical & Computer Engineering, National University of Singapore · 3PuzzleLogic Pte Ltd, Singapore 229594, Singapore

Data Augmentation Histopathology Image Classification Multiple Instance Learning Whole-Slide Image Analysis

Open paper AskPDF arXiv

Histopathological whole slide images (WSIs) are central to cancer diagnosis, but their gigapixel scale, tissue heterogeneity, weak slide-level supervision, sparse diagnostic regions, and multi-scale evidence make robust automated analysis challenging. Multiple instance learning (MIL) is widely used to aggregate tile-level features into slide-level predictions, yet existing augmentation strategies often perturb tissue regions without preserving diagnostic relevance, slide context, or cross-scale structure. We propose SlideMix, a model-agnostic multimodal augmentation framework for MIL-based WSI analysis. SlideMix uses a retrieval-augmented vision-language model (VLM)-based Visual-Language Adaptive Region selector to identify diagnostically relevant regions and reduce weak-label noise. It then performs In-place Tile Shuffling within meaningful tissue regions to mix feature embeddings while preserving slide-level context. A VLM-based soft-labeling module supervises mixed samples, while a multi-factor, loss-driven online Curriculum-Learning Feedback scheme adaptively controls shuffle granularity, feature similarity, and shuffle ratio to promote cross-scale representation learning. Across 11 WSI datasets comprising 20,523 slides, 8 diagnostic tasks, and 10 WSI backbones, SlideMix improves accuracy and generalization in most settings and compares favorably with established augmentation baselines, providing a simple plug-and-play approach for more robust and scalable digital pathology models. Source code: https://github.com/Xia-Research-Lab/SlideMix

Abstract

Whole-Slide Image Analysis

Explore similar work

SlideMix: Enhancing Whole Slide Image Analysis via Multimodal Shuffling

Exploiting Spatial Structure for Transductive Few-Shot Classification of Whole-Slide Images

Hierarchical Multi-Scale Graph Learning with Knowledge-Guided Attention for Whole-Slide Image Survival Analysis