SurgPace: Complexity-Aware Progressive Distillation for Surgical Multimodal Large Language Models
Organizations: Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · Department of Radiation Oncology, Stanford University, Palo Alto, USA · Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Chair of Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany ·
Abstract
Multimodal large language models (MLLMs) have shown strong potential in robotic surgery for question answering, scene understanding, and grounded assistance, but their high computational cost limits real clinical deployment. Knowledge distillation provides a practical way to build compact surgical MLLMs, yet direct distillation is not well suited to surgical data because surgical samples vary greatly in difficulty, visual ambiguity, and required clinical context, making uniform teacher-student transfer unstable. In addition, the role of supervision changes during training: early stages require stronger teacher guidance for multimodal alignment, while later stages rely more on task supervision for effective optimization. To address these challenges, we propose SurgPace, a complexity-aware continuous curriculum framework for progressive distillation of surgical vision-language models. SurgPace estimates sample complexity using predictive confidence and Lorentz geometric consistency, and uses the fused score to guide a continuous easy-to-hard training process. We further introduce hierarchical representation alignment, which aligns the student with the teacher at the feature, logit, and probability levels, while gradually shifting the training focus from teacher imitation to task supervision. This design separates sample-level curriculum from objective-level scheduling, leading to more stable and effective distillation. We evaluate SurgPace on Surg-396K, a benchmark built from four public surgical datasets and covering five dialogue paradigms. SurgPace consistently outperforms strong MLLM distillation baselines, demonstrating its effectiveness for building compact, memory-efficient surgical MLLMs with faster decoding under controlled benchmarks. Code is available at https://github.com/hezhicheng2002/SurgPace.
Figures & tables
| Task family | Training instances | Proportion (%) |
| Single-phrase QA | 156,983 | 34.03 |
| Visual QA | 109,960 | 23.84 |
| Detailed description | 29,960 | 6.49 |
| Region-based QA | 92,287 | 20.00 |
| Grounding QA | 72,107 | 15.63 |
| Total | 461,297 | 100.00 |
| Method | Single Phrase QA | Visual QA | |||||||
| Acc. | Prec. | Rec. | F1 | BLEU-3 | CIDEr | METEOR | ROUGE-1 | ROUGE-L | |
| MiniLLM [ 11 ] | 0.8265 | 0.8384 | 0.8446 | 0.8402 | 0.3246 | 0.4676 | 0.7608 | 0.7521 | 0.7083 |
| EfficientVLM [ 49 ] | 0.8296 | 0.8412 | 0.8475 | 0.8431 | 0.3801 | 0.4830 | 0.7705 | 0.7632 | 0.7204 |
| ULD [ 3 ] | 0.8307 | 0.8406 | 0.8458 | 0.8421 | 0.3215 | 0.4683 | 0.7613 | 0.7539 | 0.7098 |
| LLaVA-KD [ 4 ] | 0.8241 | 0.8361 | 0.8421 | 0.8378 | 0.3689 | 0.4797 | 0.7686 | 0.7599 | 0.7152 |
| LLaVA-MoD [ 41 ] | 0.8279 | 0.8388 | 0.8466 | 0.8415 | 0.3582 | 0.4649 | 0.7757 | 0.7599 | 0.7000 |
| Method | Detailed Description | Region-Based QA | Grounding QA | ||||
| ROUGE-L | CIDEr | METEOR | ROUGE-1 | ROUGE-L | AP50 | mIoU | |
| MiniLLM [ 11 ] | 0.3345 | 0.2931 | 0.5782 | 0.5964 | 0.5566 | 0.5857 | 0.5104 |
| EfficientVLM [ 49 ] | 0.3284 | 0.3034 | 0.5875 | 0.6038 | 0.5661 | 0.5454 | 0.4887 |
| ULD [ 3 ] | 0.3134 | 0.3038 | 0.5921 | 0.6013 | 0.5600 | 0.5708 | 0.5022 |
| LLaVA-KD [ 4 ] | 0.3154 | 0.3046 | 0.5918 | 0.5978 | 0.5520 | 0.5636 | 0.4993 |
| LLaVA-MoD [ 41 ] | 0.3317 | 0.3053 | 0.5786 | 0.5943 | 0.5493 | 0.5742 | 0.5019 |
| Method | Single Phrase QA | Visual QA | Description | Region-Based QA | Grounding QA | |||||
| Acc. | F1 | BLEU-3 | CIDEr | METEOR | ROUGE-L | CIDEr | METEOR | AP50 | mIoU | |
| EfficientVLM | ||||||||||
| Ours (Student) | ||||||||||
| Offline score setting | 16-metric avg. | Change |
| Original teacher logits | 0.6955 | – |
| Temperature calibration ( ) | 0.6926 | ( ) |
| Noise on centered logits ( ) | 0.6887 | ( ) |
| Audit dimension | EfficientVLM | Ours (Student) | EndoChat (Teacher) a |
| Overall (0–100) | 64.92 | 72.70 | 63.97 |
| Instrument | 3.02 | 3.40 | 3.66 |
| Tissue | 2.91 | 3.19 | 2.79 |
| Action/motion | 2.65 | 2.62 | 1.36 |
| Direction | 2.24 | 2.45 | 1.75 |
| Spatial | 2.29 | 2.91 | 2.61 |
| Flagged error type | Efficient VLM | Ours (Student) | EndoChat (Teacher) |
| Instrument identity/ count | 127 | 83 | 46 |
| Tissue/target relationship | 87 | 70 | 109 |
| Action/motion | 110 | 151 | 273 |
| Direction | 218 | 183 | 227 |
| Spatial grounding/ relation | 233 | 132 | 143 |
| Hallucination/ unsupported content | 171 | 151 | 92 |
| Metric | 13B Teacher (FP16) | 3B Student (FP16) | Reduction / speedup |
| Parameters | 13.02B | 3.22B | 4.04 fewer |
| Active text parameters | 13.02B | 2.78B | 4.69 fewer |
| Request FLOP proxy for text backbone | 4.68 fewer | ||
| Loaded GPU memory | 26.03 GB | 6.46 GB | 4.03 less |
| Peak GPU memory | 26.20 GB | 6.52 GB | 4.02 less |
| Prefill / TTFT | 31.95 ms | 18.55 ms | 1.72 faster |
| Variant | Single Phrase QA | Visual QA | Detailed Description | Region-Based QA | Grounding QA | |||||
| Acc. | F1 | BLEU-3 | CIDEr | METEOR | ROUGE-L | CIDEr | METEOR | AP50 | mIoU | |
| 0.8826 | 0.8908 | 0.3662 | 0.5102 | 0.7799 | 0.3548 | 0.3260 | 0.5984 | 0.9595 | 0.8350 | |
| 0.8997 | 0.9066 | 0.3737 | 0.5099 | 0.7817 | 0.3587 | 0.3257 | 0.5988 | 0.9614 | 0.8420 | |
| 0.8925 | 0.8992 | 0.3741 | 0.5109 | 0.7821 | 0.3561 | 0.3296 | 0.6018 | 0.9618 | 0.8390 | |
| 0.8971 | 0.9041 | 0.3754 | 0.5111 | 0.7834 | 0.3591 | 0.3289 | 0.6017 | 0.9624 | 0.8439 | |
| 0.9004 | 0.9069 | 0.3764 | 0.5135 | 0.7841 | 0.3621 | 0.3311 | 0.6010 | 0.9625 | 0.8471 | |
| Variant | Single Phrase QA | Visual QA | Detailed Description | Region-Based QA | Grounding QA | |||||
| Acc. | F1 | BLEU-3 | CIDEr | METEOR | ROUGE-L | CIDEr | METEOR | AP50 | mIoU | |
| mixed | 0.8959 | 0.9033 | 0.3520 | 0.4897 | 0.7702 | 0.3601 | 0.3309 | 0.5997 | 0.9420 | 0.8434 |
| hard-to-easy | 0.8896 | 0.8964 | 0.3758 | 0.5097 | 0.7808 | 0.3565 | 0.3276 | 0.6009 | 0.9607 | 0.8357 |
| easy-to-hard | 0.9004 | 0.9069 | 0.3764 | 0.5135 | 0.7841 | 0.3621 | 0.3311 | 0.6010 | 0.9625 | 0.8471 |
| Variant | Single Phrase QA | Visual QA | Detailed Description | Region-Based QA | Grounding QA | |||||
| Acc. | F1 | BLEU-3 | CIDEr | METEOR | ROUGE-L | CIDEr | METEOR | AP50 | mIoU | |
| One-pass | 0.6343 | 0.7369 | 0.3238 | 0.4692 | 0.7738 | 0.3481 | 0.3098 | 0.5744 | 0.9351 | 0.8049 |
| Baby step | 0.8543 | 0.8655 | 0.3123 | 0.4799 | 0.7648 | 0.3449 | 0.3105 | 0.5932 | 0.9280 | 0.8016 |
| Root | 0.8874 | 0.8949 | 0.3749 | 0.5106 | 0.7798 | 0.3589 | 0.3280 | 0.5984 | 0.9597 | 0.8360 |
| Geom | 0.8853 | 0.8933 | 0.3778 | 0.5095 | 0.7795 | 0.3587 | 0.3232 | 0.5968 | 0.9612 | 0.8396 |
| Linear | 0.9004 | 0.9069 | 0.3764 | 0.5135 | 0.7841 | 0.3621 | 0.3311 | 0.6010 | 0.9625 | 0.8471 |
| Variant | Single Phrase QA | Visual QA | Detailed Description | Region-Based QA | Grounding QA | |||||
| Acc. | F1 | BLEU-3 | CIDEr | METEOR | ROUGE-L | CIDEr | METEOR | AP50 | mIoU | |
| Objective schedule only | 0.8928 | 0.9000 | 0.3783 | 0.5099 | 0.7805 | 0.3543 | 0.3289 | 0.5980 | 0.9619 | 0.8365 |
| + static complexity score | 0.8978 | 0.9044 | 0.3692 | 0.5096 | 0.7790 | 0.3577 | 0.3295 | 0.6051 | 0.9613 | 0.8399 |
| Full (score + pacing) | 0.9004 | 0.9069 | 0.3764 | 0.5135 | 0.7841 | 0.3621 | 0.3311 | 0.6010 | 0.9625 | 0.8471 |
| Variant | Single Phrase QA | Visual QA | Detailed Description | Region-Based QA | Grounding QA | |||||
| Acc. | F1 | BLEU-3 | CIDEr | METEOR | ROUGE-L | CIDEr | METEOR | AP50 | mIoU | |
| Discrete (3,3,4) | 0.8999 | 0.9065 | 0.3762 | 0.5120 | 0.7826 | 0.3592 | 0.3294 | 0.5966 | 0.9612 | 0.8409 |
| Discrete (2,6,2) | 0.8963 | 0.9033 | 0.3685 | 0.5103 | 0.7801 | 0.3581 | 0.3284 | 0.5961 | 0.9596 | 0.8374 |
| Continuous | 0.9004 | 0.9069 | 0.3764 | 0.5135 | 0.7841 | 0.3621 | 0.3311 | 0.6010 | 0.9625 | 0.8471 |
| Variant | Single Phrase QA | Visual QA | Detailed Description | Region-Based QA | Grounding QA | |||||
| Acc. | F1 | BLEU-3 | CIDEr | METEOR | ROUGE-L | CIDEr | METEOR | AP50 | mIoU | |
| w/o latent alignment | 0.8909 | 0.8986 | 0.3745 | 0.5102 | 0.7796 | 0.3568 | 0.3284 | 0.6001 | 0.9607 | 0.8361 |
| w/o entropy regularization | 0.8843 | 0.8920 | 0.3780 | 0.5097 | 0.7794 | 0.3532 | 0.3231 | 0.5967 | 0.9577 | 0.8301 |
| full alignment | 0.9004 | 0.9069 | 0.3764 | 0.5135 | 0.7841 | 0.3621 | 0.3311 | 0.6010 | 0.9625 | 0.8471 |
| Model | GPU-hours |
| MiniLLM | 38.481 |
| EfficientVLM | 75.777 |
| ULD | 37.890 |
| LLaVA-KD | 165.514 |
| LLaVA-MoD | 81.835 |
| CKD-MDSR | 45.956 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | Target of assessment | Applicability and scale |
|---|---|---|
| Instrument identity and count | Whether the instruments are correctly identified and counted. | First five dimensions: applicable=false if the reference contains no assessable information; otherwise 4 = fully correct, 3 = minor omission/imprecision, 2 = material error with partial correctness, 1 = mostly incorrect, 0 = contradictory or absent. |
| Tissue/target relationship | Whether each instrument is correctly related to the tissue or target. | Same applicability rule and 0–4 scale as above. |
| Action or motion | Whether the described action, motion, or state is supported by the reference. | Same applicability rule and 0–4 scale as above. |
| Anatomical/instrument direction | Whether anatomical or instrument direction is correct. | Same applicability rule and 0–4 scale as above. |
| Spatial grounding or relations | Whether positions and relations among instruments and targets are grounded in the reference. | Same applicability rule and 0–4 scale as above. |
| Hallucination/unsupported content | Whether the candidate introduces details unsupported by the reference. | Always applicable; reverse severity: 4 = no unsupported content, 3 = minor unsupported detail, 2 = material unsupported content, 1 = major hallucination, 0 = severe or pervasive hallucination. |
| Field | Recorded value |
| Protocol | Six-dimension blinded clinical audit |
| Script version | 2026-08-18-clinical-audit |
| Judge model | qwen-max |
| Endpoint | Alibaba Cloud DashScope compatible-mode route |
| Temperature | 0 |
| Reasoning effort | none |
VLM Distillation
Vision-language model distillation transfers multimodal capabilities to smaller, faster models for efficient deployment.
See all 133 papers