Organizations: Stony Brook University, Stony Brook, NY, USA · Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, MA, USA
Whole Slide Image (WSI) multimodal large language models (MLLMs) are difficult to build and deploy because gigapixel slides induce thousands of highly redundant visual tokens that must be learned from sparse slide-level language supervision. Existing slide-level pathology MLLMs typically combine heavy slide-level encoders with long visual prefixes, making development and deployment expensive under limited computational resources. We revisit this regime and show that WSI tile features are highly redundant at both global and local scales, while task-relevant evidence is sparse and query-dependent. We therefore introduce LoC-Path, a resource-efficient slide-level MLLM that compresses before fusion. LoC-Path uses a sparse token merger and a masked-autoencoder (MAE)-pretrained resampler to replace expensive slide-level encoding with a compact latent interface. A token importance scorer selects the most relevant latents and cross-attention routing adapters fuse them into a few LLM decoder layers. This design lowers both multimodal tuning cost and inference-time latency/memory by avoiding heavy slide-level encoding and long visual prefixes. Extensive experiments show that the inference TFLOPs of LoC-Path are 97% lower and inference memory is also 39% lower. LoC-Path enables efficient end-to-end development and deployment under limited computational resources while remaining competitive with prior slide-level MLLMs. Code is available at https://github.com/WinstonHuTiger/loc_path.
Figures & tables
Figure 1: WSI tile tokens are redundant globally and locally, and only a sparse subset is task-relevant. We analyze ∼ 2k TCGA WSIs using CONCH-V1 [ 48 ] . (a) A small prototype budget reconstructs tile features well. (b,c) Local-neighbor and pairwise statistics both show substantial redundancy. (d) A small token subset carries most report-aligned signal: 7.7% , 22.8% , and 35.2% of tokens explain 50% , 80% , and 90% of the positive mass.
Figure 2: Overview of LoC-Path . (a) End-to-end pipeline with STM+Resampler and routed fusion (TIS/CARA). STM and resampler are designed to reduce WSI’s tile-level tokens redundancy locally and globally; TIS and CARA modules are designed to fuse the visual tokens with LLM in a query-dependent manner through cross-attention. (b) Resampler’s MAE pretraining helps it learn to glance through the tile-level token sequence and then collectively compress it. See Sec. 3.1 and Sec. 3.2 for details.
Model
Cost
WSI-Bench: VQA
WSI-Bench: Report Gen.
Zero-shot
VQA+ clin. ↑
L
M
Mem. (GB) ↓
TFLOPs ↓
Morph.
Diagnosis
Treatment
Avg.
Clin.
NLU
Slide-Bench
Path-Bench
WSI-P
WSI-R
Acc
WSI-P
WSI-R
Acc
WSI-P
WSI-R
Acc
WSI-P
WSI-R
Avg
BLEU-1
BLEU-2
ROUGE-L
METEOR
Acc
Acc
General MLLMs
GPT-4o [ 30 ]
-
-
-
-
0.220
0.204
0.471
0.472
0.457
0.530
0.496
0.841
0.875
0.507
0.067
0.138
0.103
0.202
0.069
0.132
0.167
0.414
0.793
0.305
Quilt-LLaVA [ 57 ]
576
576
12.620
12.798
0.448
0.447
0.947
0.586
0.604
0.849
0.788
0.816
1.000
0.721
0.324
0.333
0.329
0.474
0.351
0.475
0.460
0.163
0.380
0.525
SlideChat [ 10 ]
20480
20480
11.583
267.672
0.269
0.281
0.870
0.319
0.385
0.767
0.654
0.577
0.333
0.495
0.123
0.168
0.146
0.439
0.107
0.161
0.137
0.546
0.847
0.320
Table 1: Detailed performance and efficiency comparison by task type, together with report generation and zero-shot generalization (Slide-Bench and Path-Bench subset). WSI-P/WSI-R denote WSI-Precision/Relevance and Acc denotes Accuracy; L is the visual-token length after reduction or resampling (raw length if none) and M is the number routed to the LLM. Mem. (GB) and TFLOPs are per forward pass for LLM generation only (visual inputs pre-extracted for SlideChat and WSI-LLaVA; token-reduction baselines share LoC-Path’s L=256 LLM-side path); SlideChat is re-measured at its full N=20,480 -token input. Clin. Avg is the mean of WSI-P and WSI-R for report generation. The last column is 21(VQAAvg+Clin.Avg) . Best results per column are in bold. ∗ denotes WSI-LLaVA reproduced under our unified evaluation pipeline (token-budget variants in Supp. Tab. S9 ). Details of MLLM-HWSI, TCP-LLaVA, and the raw-token row are in Supp. Sec. E.2 .
Figure 3: Coverage and visualization of latent attention preferences. Bottom-left: attention coverage at threshold 10−4 ; 80% coverage means the resampler attends to 80% of token positions on the slide. z0 shows the top-64 token locations attended by latent z0 (orange); blue indicates all token positions.
Model
Stage (I vs Others)
Vital Status
Feature Quality (mean)
Computational Cost
AUC ↑
Acc ↑
AUC ↑
Acc ↑
Feature diversity (std) ↓
Coverage ↑
Peak Res. (GB) ↓
TFLOPs ↓
Slide-level encoders
TITAN [ 16 ]
0.836
0.781
0.718
0.863
0.372 (0.189)
—
9.27
0.350
PRISM [ 58 ]
0.737
0.730
0.643
0.869
0.782 (0.226)
—
7.90
3.530
GigaPath [ 81 ]
0.728
0.688
0.709
0.845
0.382 (0.197)
—
4.02
10.340
COBRA [ 38 ]
0.772
0.750
0.642
0.851
0.722 (0.127)
—
2.73
2.611
Table 2: Patient-level MIL results on TCGA-BRCA (Stage, Vital Status), with feature quality and computational cost. Feature quality is measured on 2k WSIs sampled from TCGA based on tumor type. Feature diversity is the mean pairwise cosine among output latents (lower is more diverse; std across WSIs). Computational cost (peak reserved memory and TFLOPs) is measured during inference under max tile token number for each encoder; "—" denotes not applicable. Resampler architecture ablations are in Supp. Tab. S5 .
Figure 4: Deployment scaling. Inference latency (TTFT) and Δ peak reserved memory vs. input visual token length N ; LoC-Path stays nearly constant while SlideChat grows with visual context.
Figure 5: Post-hoc interpretation of TIS. (a) WSI; (b) Post-Resampler and (c) Post-TIS patch-mass maps (warmer = higher mass); (d) mass over 13 weak tissue labels; (e) mass over the report-relevant classes. Here TIS raises the relevant mass by +0.228 , mainly via blood-vessel enrichment. Labels are used only for interpretation (Supp. Sec. G.1 ).
Number of tile (patch) tokens extracted from a WSI.
T
Number of text tokens in the input sequence.
Dv
Tile feature dimension; X∈RN×Dv is the tile-token matrix.
(ci,ri) , C
2D grid coordinate of tile i ; C is the coordinate set.
Table S1 : Summary of key symbols and operators. This table collects definitions for symbols/operators used throughout the paper and appendix for clarity and reproducibility.
Stage
GPUs
Wall (h)
GPU-h
Time on 4 GPUs (h)
Global batch
Peak mem./GPU (GB)
MAE resampler pretraining
8
10.73
85.84
21.46
64
9.2
Stage 1 (alignment)
4
1.12
4.49
1.12
512
12.4
Stage 2 (instruction tuning)
6
19.53
117.20
29.30
192
16.8
Total
–
–
207.53
51.88
–
–
Qwen2.5-7B (raw tokens), estimated on four RTX 6000 Ada GPUs
Stage 1 (alignment)
4
13.3
53.2
13.3
–
–
Table S2 : Training cost of each LoC-Path stage (NVIDIA RTX 6000 Ada). Wall time is measured between the first and last logged iteration; “Time on 4 GPUs” rescales each stage to four GPUs. The raw-token baseline was trained on H100 GPUs; its cost on RTX 6000 Ada is estimated with Stage 1 measured and Stage 2 extrapolated from the measured per-iteration time (17.7–17.9 s over 15,360 iterations).
Endpoint
LoC-Path
WSI-LLaVA ∗
Δ [95% CI]
n
VQA average
0.736
0.746
−0.010[−0.023,0.003]
442
Clinical (report) average
0.419
0.413
+0.006[−0.027,0.039]
148
VQA + clinical
0.578
0.580
−0.002[−0.020,0.016]
442
Morphology (task average)
0.701
0.699
+0.002[−0.008,0.011]
170
Diagnosis (task average)
0.679
0.686
−0.007[−0.031,0.017]
369
Treatment (task average)
0.829
0.853
−0.024[−0.051,0.001]
84
Table S3 : Paired WSI-cluster bootstrap on the common evaluable subset (LoC-Path minus reproduced WSI-LLaVA; 10,000 replicates; n = WSI clusters).
Table S4 : Extended ablation results on WSI-Bench for LoC-Path (excluding report generation), expanding the study in Sec. 4.3 of the main paper. We analyze the impact of LLM scale, fusion design, TIS routing and its ranking loss, CARA insertion layers and gating, token merging, latent and routing budgets ( L , M ), tile-level encoders, and resampler attention patterns. Unless otherwise specified, the default configuration uses a 7B LLM, the CONCH-V1 [ 48 ] encoder, and STM+Resampler; the 14B variant uses Qwen2.5-14B-Instruct; the tile-level encoder ablations use CONCH-V15 [ 16 ] and DINOv2, each with a 768-dim resampler and three cross-attention layers; the dense resampler replaces the default Top- K cross-attention with dense attention. The top block (7B, L=256/128 , M=96 ) lists the two main-paper LoC-Path settings and serves as the reference. Metrics include open-ended WSI-Precision (WSI-P), WSI-Relevance (WSI-R), and closed-ended Accuracy (Acc) across morphological, diagnostic, and treatment-planning tasks. Best results within each ablation group are highlighted in bold .
Resampler Config.
Stage
Vital Status
AUC ↑
Acc ↑
AUC ↑
Acc ↑
Ours
0.763
0.734
0.696
0.845
w/o Reg. Loss
0.741
0.672
0.714
0.845
Num. Latents:
L=128
0.800
0.688
0.720
0.826
L=512
0.739
0.672
0.662
0.789
Table S5 : Ablation of resampler architecture. We evaluate the impact of latent length ( L ), cross-attention depth, and context layer type on BRCA stage and vital status tasks (AUC/Accuracy). Unless otherwise specified, the default configuration uses L=256 , MAE pre-training, and regularization terms.
Encoder
Pre-training
Stage (I vs Others)
Vital Status
Feature Quality (mean)
AUC ↑
Acc ↑
AUC ↑
Acc ↑
Feature diversity (std) ↓
Coverage ↑
CONCH-V1 [ 48 ]
MIL-NCE [ 51 ]
0.692
0.719
0.754
0.857
0.999 ( 3.0e-4 )
0.065
LLM-loss [ 46 ]
0.724
0.688
0.759
0.882
0.841 (0.049)
1.000
MoCoV3 [ 9 ]
0.705
0.594
0.554
0.863
0.999 ( 4.1e-5 )
0.288
Ours (no regularizer)
0.741
0.672
0.714
0.845
0.110 (0.462)
0.812
Ours (MAE)
0.763
0.734
0.696
0.845
0.346 (0.056)
0.894
Table S6 : Comparison of resampler pre-training strategies. We report AUC and Accuracy for BRCA stage (I vs Others) and vital status tasks. Latent quality is evaluated via feature diversity (mean pairwise cosine similarity among latents; lower is more diverse) and union coverage, calculated on 2k random samples from the WSI-Bench training set.
Setting
s
TFLOPs ↓
Mem (MB) ↓
WSI-Bench Avg ↑
No Merge
0
1.713
2254.5
0.735
Merge 2x
2
0.440
704.5
0.738
Merge 4x
4
0.121
607.1
0.737
Table S7 : STM impact on vision cost (TFLOPs, Memory) vs. final MLLM performance (WSI-Bench Avg).
L / M
WSI-Bench Avg
128 / 96
0.740
256 / 96
0.738
256 / 144
0.729
512 / 384
0.741
Table S8 : Ablation on latent and routing budgets. WSI-Bench average (excluding report generation) under different (L,M) settings.
Figure S1 : BCNB accuracy trend as the selected token count M increases.
Model
Morphological
Diagnosis
Treatment
Avg. ↑
P
R
Acc
P
R
Acc
P
R
Acc
WSI-LLaVA ( L=576 )
0.567
0.605
0.926
0.594
0.639
0.826
0.774
0.786
1.000
0.746
WSI-LLaVA ( L=256 )
0.568
0.604
0.924
0.588
0.632
0.843
0.759
0.780
1.000
0.744
WSI-LLaVA ( L=128 )
0.565
0.594
0.924
0.589
0.615
0.846
0.748
0.761
1.000
0.738
Table S9 : WSI-LLaVA token-pooling results on WSI-Bench (excluding report generation), reproduced under our unified evaluation pipeline. Metrics are open-ended WSI-Precision (P), WSI-Relevance (R), and closed-ended Accuracy (Acc) across morphological, diagnostic, and treatment-planning tasks.
Setting
Morph.
Diag.
Treat.
Closed macro
BCNB
PathBench
λTIS=0.01
0.913
0.754
0.917
0.861
0.490
0.797
λTIS=0.02
0.943
0.860
0.979
0.927
0.519
0.813
λTIS=0.05
0.922
0.764
0.875
0.854
0.476
0.786
λrank=0
0.934
0.832
0.979
0.915
0.507
0.797
λrank=0.05
0.933
0.857
0.979
0.923
0.515
0.810
λrank=0.5
0.943
0.860
0.979
0.927
0.519
0.813
Table S10 : Sensitivity to the fusion-loss weights (accuracy). Top: λTIS with λrank=0.5 ; bottom: λrank with λTIS=0.02 .
Subset
Stage
FROC
95% CI
All 129 test slides
Raw CONCH tiles
0.548 ± 0.044
[0.447,0.657]
STM groups
0.676 ± 0.028
[0.576,0.779]
Resampler back-projection
0.268 ± 0.043
[0.153,0.397]
Δ STM − raw
+0.126
[0.058,0.185]
Δ resampler − STM
−0.409
[−0.523,−0.304]
Small-lesion subset (31 tumor + 80 normal slides)
Raw CONCH tiles
0.624 ± 0.050
[0.507,0.744]
Table S11 : Official-mask FROC of linear lesion probes at each compression stage (mean ± SD over three seeds; 95% whole-slide bootstrap CI) and paired differences.
Class
Operational definition
malignant tissue
Regions showing morphology consistent with malignant neoplasia, such as marked atypia and/or infiltrative growth patterns.
benign tissue
Non-malignant tissue regions without evident malignant infiltrative morphology.
stroma
Stromal background components, including fibrous, desmoplastic, or other supporting connective tissue.
lymphocytes
Regions predominantly characterized by lymphocytic infiltration.
necrosis
Necrotic tissue regions with loss of viable architecture or cellular detail.
adipose tissue
Regions dominated by adipocytes (fat tissue).
Table S12: Thirteen coarse tissue classes used for weak semantic labeling in Fig. 5 . The taxonomy and operational definitions were specified with guidance from a practicing pathologist. These categories are used only for interpretability analysis and do not constitute definitive pathology labels.
Figure S2 : Diversity of resampler latents via Top- K attention. Each panel corresponds to one latent from the MAE-pretrained resampler. This resampler has 256 output latents. Orange dots mark the spatial locations of the Top-64 input tokens receiving the highest attention from that latent; the blue background shows all foreground token positions. Different latents specialize to distinct regions and structures while jointly covering the whole slide, qualitatively confirming the high coverage and diversity reported in Sec. 4.2 and Tab. S6 .
Figure S3 : Visualization of MAE resampler reconstruction. Left: Original tile-level feature map (visualized by feature norm). Middle: Masked input (70% mask ratio); black regions indicate missing tokens. Right: Features reconstructed by the MAE-pretrained resampler (256 latents). Despite heavy corruption, the model successfully recovers the global tissue layout and feature magnitude, demonstrating the efficacy of the compressed latent representation.
Figure S4 : STM merging behavior on a TCGA WSI. From left to right: (i) input WSI thumbnail, (ii) original tile grid with per-tile feature norms, and (iii) merged grid after applying STM with a local window (e.g., s=2 ). STM aggressively reduces the number of tiles while largely preserving the overall tissue footprint and coarse morphology, illustrating how local redundancy can be removed before resampling.
Figure S5 : Qualitative success and failure cases across WSI-Bench tasks. Top row: Successful cases in morphological analysis and diagnosis, showing high overlap with expert descriptions. Bottom row: Typical failure modes in treatment planning and report generation. While the model captures global intent, it may miss targeted options or hallucinate details. (Highlight colors mark corresponding content in the reference and our output.)
Koc¸ University, Department of Computer Engineering · Koc¸ University, Department of Pathology · Hacettepe University, Department of Artificial Intelligence and Data Engineering
Harbin Institute of Technology, Shenzhen, School of Computer Science and Technology · National University of Singapore, Department of Electronic and Computer Engineering