SurgPace: Complexity-Aware Progressive Distillation for Surgical Multimodal Large Language Models
Authors: Zhicheng He, Xiaohan Xing, Hongqiu Wang, Kun Yuan, Guankun Wang, Yanheng Li, Haoxuan Che, Nassir Navab, Hongliang Ren, Long Bai
Organizations: Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · Department of Radiation Oncology, Stanford University, Palo Alto, USA · Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Chair of Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany · XGEN Labs, Hong Kong SAR, China
Multimodal large language models (MLLMs) have shown strong potential in robotic surgery for question answering, scene understanding, and grounded assistance, but their high computational cost limits real clinical deployment. Knowledge distillation provides a practical way to build compact surgical MLLMs, yet direct distillation is not well suited to surgical data because surgical samples vary greatly in difficulty, visual ambiguity, and required clinical context, making uniform teacher-student transfer unstable. In addition, the role of supervision changes during training: early stages require stronger teacher guidance for multimodal alignment, while later stages rely more on task supervision for effective optimization. To address these challenges, we propose SurgPace, a complexity-aware continuous curriculum framework for progressive distillation of surgical vision-language models. SurgPace estimates sample complexity using predictive confidence and Lorentz geometric consistency, and uses the fused score to guide a continuous easy-to-hard training process. We further introduce hierarchical representation alignment, which aligns the student with the teacher at the feature, logit, and probability levels, while gradually shifting the training focus from teacher imitation to task supervision. This design separates sample-level curriculum from objective-level scheduling, leading to more stable and effective distillation. We evaluate SurgPace on Surg-396K, a benchmark built from four public surgical datasets and covering five dialogue paradigms. SurgPace consistently outperforms strong MLLM distillation baselines, demonstrating its effectiveness for building compact, memory-efficient surgical MLLMs with faster decoding under controlled benchmarks. Code is available at https://github.com/hezhicheng2002/SurgPace.
Figures & tables
Figure 1: Overview of the proposed SurgPace framework. Given surgical multimodal data, a teacher model first estimates sample complexity offline using complementary confidence and geometric consistency signals, which are permissively fused so that lower scores indicate easier samples. The resulting complexity estimates are then used to guide online curriculum training, where easier samples are emphasized earlier and harder samples are gradually incorporated as training progresses. During this process, the student model learns from the teacher through hierarchical representation alignment at the probability, logit, and feature levels, while a progressive objective schedule adjusts the balance of supervision over time. Together, these components enable stable and efficient compression of surgical MLLMs into a lightweight student model with reduced computational demands.
Figure 2: Overview of the proposed complexity-aware curriculum framework for progressive distillation. Teacher confidence and Lorentz geometric consistency are fused into a permissive complexity score, which guides continuous curriculum pacing from easy-first sampling to balanced training. This pacing is coupled with progressive objective interpolation and hierarchical representation alignment, jointly establishing a more structured optimization process that enables stable and efficient compression of surgical vision-language models.
Figure 3: Qualitative visualization of a representative same-image case across five dialogue paradigms. For the same surgical scene, the figure compares ground truth, teacher, and student outputs for single-phrase QA, visual QA, region-based QA, grounding QA, and detailed description. The example shows that the student preserves the main clinical semantics and spatial relationships while remaining compact.
Figure 4: Radar chart of task mean scores computed from Tables 2 and 3 . Each axis averages the metrics within one dialogue paradigm.
Task family
Training instances
Proportion (%)
Single-phrase QA
156,983
34.03
Visual QA
109,960
23.84
Detailed description
29,960
6.49
Region-based QA
92,287
20.00
Grounding QA
72,107
15.63
Total
461,297
100.00
Table 1: Distribution of the processed Surg-396K split used in this study.
Method
Single Phrase QA
Visual QA
Acc.
Prec.
Rec.
F1
BLEU-3
CIDEr
METEOR
ROUGE-1
ROUGE-L
MiniLLM [ 11 ]
0.8265
0.8384
0.8446
0.8402
0.3246
0.4676
0.7608
0.7521
0.7083
EfficientVLM [ 49 ]
0.8296
0.8412
0.8475
0.8431
0.3801
0.4830
0.7705
0.7632
0.7204
ULD [ 3 ]
0.8307
0.8406
0.8458
0.8421
0.3215
0.4683
0.7613
0.7539
0.7098
LLaVA-KD [ 4 ]
0.8241
0.8361
0.8421
0.8378
0.3689
0.4797
0.7686
0.7599
0.7152
LLaVA-MoD [ 41 ]
0.8279
0.8388
0.8466
0.8415
0.3582
0.4649
0.7757
0.7599
0.7000
Table 2: Comparison experiments against state-of-the-art MLLM distillation methods on Single Phrase QA and Visual QA under a fixed 64,856-example evaluation source and greedy decoding.
Method
Detailed Description
Region-Based QA
Grounding QA
ROUGE-L
CIDEr
METEOR
ROUGE-1
ROUGE-L
AP50
mIoU
MiniLLM [ 11 ]
0.3345
0.2931
0.5782
0.5964
0.5566
0.5857
0.5104
EfficientVLM [ 49 ]
0.3284
0.3034
0.5875
0.6038
0.5661
0.5454
0.4887
ULD [ 3 ]
0.3134
0.3038
0.5921
0.6013
0.5600
0.5708
0.5022
LLaVA-KD [ 4 ]
0.3154
0.3046
0.5918
0.5978
0.5520
0.5636
0.4993
LLaVA-MoD [ 41 ]
0.3317
0.3053
0.5786
0.5943
0.5493
0.5742
0.5019
Table 3: Comparison experiments against state-of-the-art MLLM distillation methods on Detailed Description, Region-Based QA, and Grounding QA under the same fixed evaluation protocol as Table 2 .
Method
Single Phrase QA
Visual QA
Description
Region-Based QA
Grounding QA
Acc.
F1
BLEU-3
CIDEr
METEOR
ROUGE-L
CIDEr
METEOR
AP50
mIoU
EfficientVLM
0.8312±0.0014
0.8447±0.0014
0.3741±0.0056
0.4850±0.0018
0.7705±0.0016
0.3259±0.0057
0.3130±0.0084
0.5871±0.0007
0.5686±0.0229
0.5032±0.0136
Ours (Student)
0.8915±0.0101
0.8990±0.0092
0.3822±0.0050
0.5119±0.0014
0.7843±0.0042
0.3589±0.0033
0.3292±0.0038
0.5999±0.0016
0.9604±0.0031
0.8424±0.0069
Table 4: Robustness comparison across three seeds between SurgPace and the strongest baseline, EfficientVLM, using the same ten task-spanning metrics reported in the ablation studies. Each cell gives the mean, with the sample standard deviation shown below, over seeds 0, 1, and 42 under the fixed common evaluation protocol.
Offline score setting
16-metric avg.
Change
Original teacher logits
0.6955
–
Temperature calibration ( T=1.3 )
0.6926
−0.0029 ( −0.4% )
Noise on centered logits ( σrel=0.05 )
0.6887
−0.0068 ( −1.0% )
Table 5: Sensitivity to perturbations of the teacher logits used only for offline difficulty scoring. All settings use EndoChat, seed 42, identical online distillation, and the same training and common evaluation protocols.
Audit dimension
EfficientVLM
Ours (Student)
EndoChat (Teacher) a
Overall (0–100)
64.92
72.70
63.97
Instrument
3.02
3.40
3.66
Tissue
2.91
3.19
2.79
Action/motion
2.65
2.62
1.36
Direction
2.24
2.45
1.75
Spatial
2.29
2.91
2.61
Table 7: Blinded clinical audit based on reference text across six dimensions using Qwen-3.8-Max on 300 shared stratified Detailed Description samples per system. Scores range from 0 to 4; higher is better, and a hallucination score of 4 denotes no unsupported content.
Flagged error type
Efficient VLM
Ours (Student)
EndoChat (Teacher)
Instrument identity/ count
127
83
46
Tissue/target relationship
87
70
109
Action/motion
110
151
273
Direction
218
183
227
Spatial grounding/ relation
233
132
143
Hallucination/ unsupported content
171
151
92
Table 8: Flagged error counts from the shared blinded clinical audit. Counts are reported out of the same 300 stratified Detailed Description samples for each system.
Metric
13B Teacher (FP16)
3B Student (FP16)
Reduction / speedup
Parameters
13.02B
3.22B
4.04 × fewer
Active text parameters
13.02B
2.78B
4.69 × fewer
Request FLOP proxy for text backbone
5.00×1012
1.07×1012
4.68 × fewer
Loaded GPU memory
26.03 GB
6.46 GB
4.03 × less
Peak GPU memory
26.20 GB
6.52 GB
4.02 × less
Prefill / TTFT
31.95 ms
18.55 ms
1.72 × faster
Table 10: Controlled text generation efficiency on one NVIDIA A100 80GB GPU. The 13B teacher and 3B student are both evaluated in FP16 for a precision-matched comparison; both use batch size 1, 128 input tokens, 64 output tokens, greedy decoding, three warm-up runs, and ten measured repeats.
Variant
Single Phrase QA
Visual QA
Detailed Description
Region-Based QA
Grounding QA
Acc.
F1
BLEU-3
CIDEr
METEOR
ROUGE-L
CIDEr
METEOR
AP50
mIoU
sconf
0.8826
0.8908
0.3662
0.5102
0.7799
0.3548
0.3260
0.5984
0.9595
0.8350
slorentz
0.8997
0.9066
0.3737
0.5099
0.7817
0.3587
0.3257
0.5988
0.9614
0.8420
smax
0.8925
0.8992
0.3741
0.5109
0.7821
0.3561
0.3296
0.6018
0.9618
0.8390
smean
0.8971
0.9041
0.3754
0.5111
0.7834
0.3591
0.3289
0.6017
0.9624
0.8439
smin
0.9004
0.9069
0.3764
0.5135
0.7841
0.3621
0.3311
0.6010
0.9625
0.8471
Table 11: Ablation on complexity score fusion in the offline scoring stage. The variants correspond to Eq. ( 10 ), which fuses the teacher-derived confidence score sconf computed in the standard Euclidean probability space and the Lorentz geometric score slorentz into the final complexity score si . The adopted final configuration is highlighted in yellow.
Variant
Single Phrase QA
Visual QA
Detailed Description
Region-Based QA
Grounding QA
Acc.
F1
BLEU-3
CIDEr
METEOR
ROUGE-L
CIDEr
METEOR
AP50
mIoU
mixed
0.8959
0.9033
0.3520
0.4897
0.7702
0.3601
0.3309
0.5997
0.9420
0.8434
hard-to-easy
0.8896
0.8964
0.3758
0.5097
0.7808
0.3565
0.3276
0.6009
0.9607
0.8357
easy-to-hard
0.9004
0.9069
0.3764
0.5135
0.7841
0.3621
0.3311
0.6010
0.9625
0.8471
Table 12: Ablation on curriculum order . Different variants change the ordering policy induced by the complexity-aware pacing strategy. The adopted final configuration is highlighted in yellow.
Variant
Single Phrase QA
Visual QA
Detailed Description
Region-Based QA
Grounding QA
Acc.
F1
BLEU-3
CIDEr
METEOR
ROUGE-L
CIDEr
METEOR
AP50
mIoU
One-pass
0.6343
0.7369
0.3238
0.4692
0.7738
0.3481
0.3098
0.5744
0.9351
0.8049
Baby step
0.8543
0.8655
0.3123
0.4799
0.7648
0.3449
0.3105
0.5932
0.9280
0.8016
Root
0.8874
0.8949
0.3749
0.5106
0.7798
0.3589
0.3280
0.5984
0.9597
0.8360
Geom
0.8853
0.8933
0.3778
0.5095
0.7795
0.3587
0.3232
0.5968
0.9612
0.8396
Linear
0.9004
0.9069
0.3764
0.5135
0.7841
0.3621
0.3311
0.6010
0.9625
0.8471
Table 13: Ablation on the curriculum pacing function . The pacing function is the continuous weighting function in Sec. 3.2 that converts the offline complexity score si into the progress-dependent sample weight wi′(p) through Eqs. (11)-(14). Different variants compare one-pass, baby-step, root, geometric, and linear pacing under the same score signal. The adopted final configuration is highlighted in yellow.
Variant
Single Phrase QA
Visual QA
Detailed Description
Region-Based QA
Grounding QA
Acc.
F1
BLEU-3
CIDEr
METEOR
ROUGE-L
CIDEr
METEOR
AP50
mIoU
Objective schedule only
0.8928
0.9000
0.3783
0.5099
0.7805
0.3543
0.3289
0.5980
0.9619
0.8365
+ static complexity score
0.8978
0.9044
0.3692
0.5096
0.7790
0.3577
0.3295
0.6051
0.9613
0.8399
Full (score + pacing)
0.9004
0.9069
0.3764
0.5135
0.7841
0.3621
0.3311
0.6010
0.9625
0.8471
Table 14: Ablation on the sample-level curriculum additions to the progressive objective schedule, which is retained in every row. Objective schedule only uses unit sample weights; adding the static score introduces fixed weights from the informative complexity estimate; and the full setting further makes these weights progress-dependent through linear pacing. The adopted final configuration is highlighted in yellow.
Variant
Single Phrase QA
Visual QA
Detailed Description
Region-Based QA
Grounding QA
Acc.
F1
BLEU-3
CIDEr
METEOR
ROUGE-L
CIDEr
METEOR
AP50
mIoU
Discrete (3,3,4)
0.8999
0.9065
0.3762
0.5120
0.7826
0.3592
0.3294
0.5966
0.9612
0.8409
Discrete (2,6,2)
0.8963
0.9033
0.3685
0.5103
0.7801
0.3581
0.3284
0.5961
0.9596
0.8374
Continuous
0.9004
0.9069
0.3764
0.5135
0.7841
0.3621
0.3311
0.6010
0.9625
0.8471
Table 15: Ablation on the phase transition design in the progressive objective schedule. In Sec. 3.4 and Eqs. ( 23 )-( 24 ), the transition is defined by the stage coefficient α(p) , which shifts supervision from teacher-dominant to task-dominant optimization through βdist(p) and βce(p) . Here, Discrete denotes fixed stage partitions with hard phase ratios, whereas Continuous denotes the smooth interpolation used in Eq. ( 23 ). The adopted final configuration is highlighted in yellow.
Variant
Single Phrase QA
Visual QA
Detailed Description
Region-Based QA
Grounding QA
Acc.
F1
BLEU-3
CIDEr
METEOR
ROUGE-L
CIDEr
METEOR
AP50
mIoU
w/o latent alignment
0.8909
0.8986
0.3745
0.5102
0.7796
0.3568
0.3284
0.6001
0.9607
0.8361
w/o entropy regularization
0.8843
0.8920
0.3780
0.5097
0.7794
0.3532
0.3231
0.5967
0.9577
0.8301
full alignment
0.9004
0.9069
0.3764
0.5135
0.7841
0.3621
0.3311
0.6010
0.9625
0.8471
Table 16: Ablation on the Hierarchical Representation Alignment objective defined in Sec. 3.3 and used in Eq. ( 28 ). Cross-entropy is kept fixed in all variants, while each row removes one auxiliary component from the alignment objective. Removing the KL or L2 terms causes severe degradation and is omitted.
Model
GPU-hours
MiniLLM
38.481
EfficientVLM
75.777
ULD
37.890
LLaVA-KD
165.514
LLaVA-MoD
81.835
CKD-MDSR
45.956
Table 17: Comparison of reported training budgets and offline difficulty-scoring cost for the seven student methods. The SurgPace total includes both components.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
Target of assessment
Applicability and scale
Instrument identity and count
Whether the instruments are correctly identified and counted.
First five dimensions: applicable=false if the reference contains no assessable information; otherwise 4 = fully correct, 3 = minor omission/imprecision, 2 = material error with partial correctness, 1 = mostly incorrect, 0 = contradictory or absent.
Tissue/target relationship
Whether each instrument is correctly related to the tissue or target.
Same applicability rule and 0–4 scale as above.
Action or motion
Whether the described action, motion, or state is supported by the reference.
Same applicability rule and 0–4 scale as above.
Anatomical/instrument direction
Whether anatomical or instrument direction is correct.
Same applicability rule and 0–4 scale as above.
Spatial grounding or relations
Whether positions and relations among instruments and targets are grounded in the reference.
Same applicability rule and 0–4 scale as above.
Hallucination/unsupported content
Whether the candidate introduces details unsupported by the reference.
Always applicable; reverse severity: 4 = no unsupported content, 3 = minor unsupported detail, 2 = material unsupported content, 1 = major hallucination, 0 = severe or pervasive hallucination.
Appendix
Table S1 : Dimensions used in the blinded clinical audit.
Field
Recorded value
Protocol
Six-dimension blinded clinical audit
Script version
2026-08-18-clinical-audit
Judge model
qwen-max
Endpoint
Alibaba Cloud DashScope compatible-mode route
Temperature
0
Reasoning effort
none
Appendix
Table S2: Recorded configuration for the six-dimension blinded clinical audit.