Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models

Chengyu Fang; Heng Guo; Zheng Jiang; Chunming He; Xiu Li; Minfeng Xu

cs.CVMar 26, 2026

Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models

Authors: Chengyu Fang, Heng Guo, Zheng Jiang, Chunming He, Xiu Li, Minfeng Xu

Organizations: Tsinghua University · DAMO Academy, Alibaba Group · Hupan Lab · Duke University

Abstract

Multimodal large language models are promising for clinical visual question answering tasks, but scaling to 3D imaging is hindered by high computational costs. Prior methods often rely on 2D slices or fixed-length token compression, disrupting volumetric continuity and obscuring subtle findings. We present Photon, a framework that represents 3D medical volumes with token sequences of variable length. Photon introduces instruction-conditioned token scheduling and surrogate gradient propagation to adaptively reduce tokens during both training and inference, which lowers computational cost while mitigating the attention dilution caused by redundant tokens. It incorporates a custom backpropagation rule with gradient restoration to enable differentiable optimization despite discrete token drop. To stabilize token compression and ensure reliable use of visual evidence, Photon further applies regularization objectives that mitigate language-only bias and improve reliability. Experiments on diverse medical visual question answering tasks show that Photon achieves state-of-the-art accuracy while reducing resource usage and accelerating both training and inference.

Figures & tables

Figure 1: Photon is a 3D-native framework that adaptively models medical volumes using variable-length tokens, accelerating both training and inference. It enables efficient clinical question answering by removing instruction-irrelevant tokens while maintaining better quantitative performance.

Figure 2: Photon’s pipeline: Phase 1 aligns the visual embedding layer, and Phase 2 finetunes all modules for task adaptation, learning token reduction threshold estimation through our backpropagation strategy. Modules in right upper with black contour are not updated during training.

Task	Metric	Zero-shot						Finetuned
		Qwen2.5-VL	RadFM	M3D-L2	M3D-P3	OmniV	Lingshu	M3D-L2	M3D-P3	Photon
		3B	13B	7B	4B	1.5B	7B	7B	4B	3B
Existence Detection	Accuracy	19.52	29.20	18.00	40.25	28.66	59.60	81.09	82.43	83.07
Stat. Temp. Diag.	Accuracy	0.00	44.11	25.47	25.40	22.96	6.02	51.20	49.30	52.86
Longit. Temp. Diag.	Accuracy	0.29	42.99	24.17	24.31	24.23	12.13	74.78	74.77	77.01
Medical Measurement	BLEU	1.78	3.34	15.95	2.55	2.52	2.50	30.54	33.52	37.74

Table 1: Comparison of zero-shot and finetuned performance across six tasks in 3D-RAD benchmark. violet and indigo indicate the best and the second best. Stat. Temp. Diag. = Static Temporal Diagnosis, Longit. Temp. Diag. = Longitudinal Temporal Diagnosis.

Type	Subtype	Multi-choice							Free-text
		Rand	Merlin	M3D-L2	M3D-P3	CT-CHAT	RadFM	Photon	Merlin	M3D-L2	M3D-P3	CT-CHAT	RadFM	Photon
		—	7B	7B	4B	7B	13B	3B	7B	7B	4B	7B	13B	3B
Meas.	lesion volume measurement ∗	0.250	0.253	0.815	0.825	0.833	0.815	0.825	0.079	0.085	0.079	0.075	0.112	0.265
	organ HU measurement ∗	0.250	0.254	0.638	0.640	0.637	0.647	0.642	0.487	0.490	0.491	0.513	0.608	0.777
	organ volume measurement ∗	0.250	0.262	0.747	0.754	0.750	0.755	0.756	0.526	0.535	0.528	0.549	0.583	0.717
Average		0.250	0.256	0.733	0.740	0.740	0.739	0.741	0.364	0.370	0.366	0.379	0.434	0.587

Table 2: Performance on DeepTumorVQA benchmark. Subtypes marked with ∗ indicate free-text numerical answers evaluated using MRA, higher is better. Meas. = Measurement, Recog. = Recognition, Vis. Rsn. = Visual Reasoning, Med. Rsn. = Medical Reasoning.

Methods	E.D.	S.T.D.	L.T.D.	Medical Measurement			Image Observation			Anomaly Detection			Infer Speed	Token Num
Methods	Acc.	Acc.	Acc.	BLEU	ROUGE	BERT	BLEU	ROUGE	BERT	BLEU	ROUGE	BERT	(Tok/s)	(Tok/case)
Qwen2.5-VL	81.97	47.62	75.36	37.00	38.57	96.04	52.72	56.48	93.63	42.01	47.36	91.93	2.30	7.0K
VisionZip	82.00	47.19	75.42	37.04	38.69	96.06	52.51	56.36	93.63	41.96	47.39	91.93	2.32	2.1K
VisionZip	81.99	47.74	75.93	37.11	38.68	96.07	52.72	56.60	93.65	41.98	47.36	91.92	2.23	3.5K
VisionZip	82.01	47.40	75.88	37.07	38.82	96.04	52.57	56.43	93.64	42.01	47.39	91.93	2.18	4.9K
HiPrune	81.99	48.08	75.50	37.00	38.59	96.05	52.56	56.46	93.64	42.08	47.32	91.93	0.76	2.1K

Table 3: Comparison of different token pruning strategies on 3D-RAD. E.D. = Existence Detection, S.T.D. = Static Temporal Diagnosis, L.T.D. = Longitudinal Temporal Diagnosis.

Figure 3: Visualization of Photon’s results. White regions indicate reduced tokens, while purple boxes highlight retained areas that carry clinically relevant information for answering the questions.

Task	E.D.	S.T.D.	L.T.D.	Medical Measurement			Image Observation			Anomaly Detection			Computational Metrics
Task	Acc.	Acc.	Acc.	BLEU	ROUGE	BERT	BLEU	ROUGE	BERT	BLEU	ROUGE	BERT	Train spd.	Train Tok.	Infer Spd.	Infer Mem.
Without task-related finetune.
Qwen2.5-VL	19.52	0.00	0.29	1.78	4.30	84.20	3.51	8.84	84.53	2.93	9.17	84.47	—	—	—	—
Vis. Ful. Ft.	0.00	0.00	0.00	5.02	9.91	84.82	1.02	2.96	81.46	0.83	2.61	80.83	—	—	—	—
Photon Phase 1	28.10	8.41	10.90	4.81	9.69	85.35	5.98	17.62	86.39	6.84	15.64	85.98	—	—	—	—
Finetuned with 3D-RAD datasets.

Table 4: Ablation study on 3D-RAD. Computational statistics include training speed (iterations/s), average retaining tokens, and inference speed (tokens/s), inference peak GPU memory. E.D. = Existence Detection, S.T.D. = Static Temporal Diagnosis, L.T.D. = Longitudinal Temporal Diagnosis.

Figure 4: Visualization Results of base model and different visual alignment methods. Our method achieves alignment while avoiding mode collapse. Vis. Ful. Ft. = Visual Modules Fully Finetuned.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Visualization of retention band triggers ( eq. 18 ) and the number of kept tokens. Left: training without robust regularization, middle: training without flip regularization, right: training with both regularizations. A lower activation frequency and smaller magnitude of the retention band indicate more stable pruning during training.

Figure 6: Visualization of retention rate of training time. Our token reduction method exhibits an average retention rate that fluctuates within a reasonable range before steadily stabilizing, indicating convergence toward an appropriate per-sample threshold estimation.

Metric	RadFM	M3D-P3	M3D-L2	Merlin	CT-CHAT	Photon	Photon Bootstrap
Sensitivity	75.2	0.0	0.0	50.2	0.0	81.2 $\pm$ 1.7	81.7 [78.6, 84.7]
Specificity	98.6	100.0	100.0	51.2	100.0	95.2 $\pm$ 0.1	95.3 [93.8, 96.6]
Accuracy	88.6	57.4	57.4	50.7	57.4	89.3 $\pm$ 0.7	89.5 [87.9, 91.1]

Appendix

Table 5: Clinical Metrics of Kidney tumor recognition (sensitivity, specificity, and accuracy (%)).

Method	Avg. keep tok.	Meas.	Recog.	Vis. Rsn.	Med. Rsn.	Overall
Photon OOD Multi-Choice	0.34K	0.745	0.599	0.636	0.677	0.648
Photon OOD Free-Text	0.40K	0.459	0.643	0.520	0.599	0.570

Appendix

Table 6: Photon performance on the AbdomenAtlas3.0-style OOD split of DeepTumorVQA. Results are reported for multi-choice and free-text variants (accuracy (%)).

Method	Avg. keep tok.	CT	MRI	X-ray	Other	Ultrasound	Overall
Qwen2.5-VL 3B Zero-Shot	7.00K	0.461	0.460	0.485	0.477	0.432	0.460
Qwen2.5-VL 3B + ITS Zero-Shot	3.28K	0.459	0.456	0.488	0.470	0.426	0.457

Appendix

Table 7: Zero-shot results on full MedFrameQA dataset. No additional medical fine-tuning is performed (accuracy (%)).

Setting	E.D.	S.T.D.	L.T.D.	Medical Measurement			Image Observation			Anomaly Detection			Avg. Train Tok.
Setting	Acc.	Acc.	Acc.	BLEU	ROUGE	BERT	BLEU	ROUGE	BERT	BLEU	ROUGE	BERT	Avg. Train Tok.
Effect of $\tau_{\text{ce}}$ (temperature for contrastive estimation).
$\tau_{\text{ce}}$ =0.10	82.44	51.66	77.66	38.65	39.87	96.20	51.11	56.77	93.61	41.95	46.91	91.86	0.42K
$\tau_{\text{ce}}$ =0.20	83.07	52.86	77.01	37.74	39.36	96.14	51.59	56.66	93.62	42.33	47.50	91.96	0.39K
$\tau_{\text{ce}}$ =0.50	82.88	52.13	77.71	38.59	39.77	96.25	51.61	56.16	93.53	41.73	46.81	91.87	0.32K
Effect of $\beta$ (surrogate gradient weight).

Appendix

Table 8: Ablation on key hyperparameters and robustness of Photon.

\tau_{\text{ce}}

controls the temperature for contrastive estimation;

\beta

balances surrogate gradient strength;

\epsilon_{\text{sat}}

defines the saturation suppression threshold. The last block tests robustness under noisy threshold perturbation and cross-model scaling.

Method	MedFrameQA							SLAKE
Method	Avg. keep rate.	CT	MRI	X-ray	Other	Ultrasound	Avg. Acc.	Avg. keep	Acc.	Match / Total
Qwen2.5VL 3B	1.000	0.744	0.745	0.893	0.667	0.486	0.727	1.000	0.8446	413/489
Ours 3B	0.432	0.761	0.787	0.857	0.750	0.486	0.745	0.375	0.8425	412/489

Appendix

Table 9: Token retention and performance on MedFrameQA and SLAKE (accuracy).

Method	Avg. keep rate	Question type accuracy								Avg. Acc
Method	Avg. keep rate	Causal-How	Causal-Why	Desc.-Count	Desc.-Loc	Desc.-Other	Temp.-Now	Temp.-Next	Temp.-Present	Avg. Acc
Qwen2.5VL 3B	1.000	0.7545	0.7738	0.6429	0.9358	0.8533	0.7760	0.7269	0.7097	0.7729
Ours 3B	0.473	0.7502	0.7693	0.6522	0.9358	0.8483	0.7631	0.7455	0.7527	0.7723

Appendix

Table 10: Token retention and performance on NExT-QA (accuracy).

Figure 7: More visualization results and failure cases by Photon.

Task	Image observation	Anomaly detection
Avg. retained tokens	485.3	479.6
Task	Medical measurement	Existence detection
Avg. retained tokens	499.5	455.9

Appendix

Table 11: Task-conditioned lung token retention (left) and organ-wise Jaccard dissimilarity (right).

Method	Global coverage	Mean per volume coverage
Ours 3B	0.831	0.818

Appendix

Table 12: Block-level lung coverage for Photon (3B) on lung-related instructions in 3D-RAD.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models
Mar 12, 2026Shengyuan Liu, Zanting Ye, Yunrui Lin +6Efficient VLM Inference 3D Medical Imaging
Chinese University of Hong Kong · Westlake University · Southern Medical University +4
MedARC: Training-Free Adaptive Redundancy Compression of Visual Tokens for 3D Medical Vision-Language Models
Jul 29, 2026Yitao Zhu, Mengjun Liu, Yingji Fu +2Efficient VLM Inference Medical VLMs
Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong · School of Integrated Circuits and Electronics, Beijing Institute of Technology, China · 3Mental Health Research Center, The Hong Kong Polytechnic University, Hong Kong +1
CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding
Aug 5, 2026Zhenyu Yi, Qiang Hu, Zhenhao Li +33D Medical Imaging Medical VQA
School of Biomedical Engineering, Shanghai Jiao Tong University · 2Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology

Mar 12, 2026J/K move · Enter open · S save

MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models

Shengyuan Liu, Zanting Ye, Yunrui Lin, Chen Hu+5

Chinese University of Hong Kong · Westlake University · Southern Medical University+4

Efficient VLM Inference 3D Medical Imaging Medical VLMs Visual Token Pruning

Open paper AskPDF arXiv

While specialized Medical Vision-Language Models (VLMs) have achieved remarkable success in interpreting 2D and 3D medical modalities, their deployment for 3D volumetric data remains constrained by significant computational inefficiencies. Current architectures typically suffer from massive anatomical redundancy due to the direct concatenation of consecutive 2D slices and lack the flexibility to handle heterogeneous information densities across different slices using fixed pruning ratios. To address these challenges, we propose MedPruner, a training-free and model-agnostic hierarchical token pruning framework specifically designed for efficient 3D medical image understanding. MedPruner introduces a two-stage mechanism: an Inter-slice Anchor-based Filtering module to eliminate slice-level temporal redundancy, followed by a Dynamic Information Nucleus Selection strategy that achieves adaptive token-level compression by quantifying cumulative attention weights. Extensive experiments on three 3D medical benchmarks and across three diverse medical VLMs reveal massive token redundancy in existing architectures. Notably, MedPruner enables models such as MedGemma-1.5 to maintain or even exceed their original performance while retaining fewer than 5% of visual tokens, thereby reducing visual-token overhead and validating the necessity of dynamic token selection for practical clinical deployment. Our code is available at https://github.com/CUHK-AIM-Group/MedPruner.

Abstract

Medical VLMs

Explore similar work

MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models

MedARC: Training-Free Adaptive Redundancy Compression of Visual Tokens for 3D Medical Vision-Language Models

CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding