LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation

Jaeyun Shin; Hangeol Chang; Jong Chul Ye

cs.CVSep 29, 2026

LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation

Authors: Jaeyun Shin, Hangeol Chang, Jong Chul Ye

Organizations: Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Multimodal on-policy distillation (OPD) aims to improve visual grounding while preserving the strong reasoning capabilities of language models. Recent multi-teacher approaches combine LLM and VLM teachers to provide complementary supervision. However, directly using a VLM's full predictive distribution entangles its visual grounding signal with its own language prior, preventing the grounding information from being transferred independently. Conversely, increasing the strength of visual supervision can improve perception but may overemphasize visual evidence and degrade language reasoning. To address this trade-off, we introduce LEGO-OPD, which selectively composes factors from a Language Expert and a Grounding expert into One teacher distribution for multimodal OPD. Under a generalized Bayesian formulation, the language expert provides a prior over candidate tokens, while the grounding expert contributes a visual likelihood that updates this prior, rather than transferring its complete predictive distribution. This factorized composition allows language reasoning and visual grounding to be controlled independently. We further introduce adaptive calibration to determine how strongly the visual likelihood should update the language prior at each decoding prefix. Specifically, LEGO-OPD uses the grounding expert's image-induced prediction shift as a prefix-dependent reference, preventing both insufficient and excessive visual supervision. Experiments with Qwen3 models show that LEGO-OPD consistently outperforms the evaluated single- and multi-teacher OPD baselines on both multimodal and text-only reasoning tasks. Moreover, it improves the initial student's visual perception while preserving text-only reasoning.

Figures & tables

Figure 1: Overview of LEGO-OPD. (a) A grounding expert (VLM) supplies a visual likelihood ratio, while a language expert (LLM) supplies a language prior. (b) Source-referenced calibration adjusts the contribution of the visual likelihood ratio. (c) The calibrated factors are composed into a single teacher distribution for on-policy distillation of the student VLM.

Panel of Figure 2 — Figure 2: Local approximation error in expert composition. Curves show means within source-KL bins at $\rho=1$ . The dotted $y=x$ line denotes source-KL matching. Shaded bands indicate question-bootstrap 95% confidence intervals.

	Teacher	Initial Student	Distillation Baselines				Our Method
Model / Method	Qwen3-VL 4B-Instruct	Qwen3-VL 2B-Instruct	Standard OPD	VA-OPD	M-OPD	H-OPD	LEGO-OPD $\rho=1$	LEGO-OPD $\rho=2$
Teacher	–	–	Qwen3-VL-4B	Qwen3-VL-4B	Qwen3-VL-4B + Qwen3-4B	Qwen3-VL-4B + Qwen3-4B	Qwen3-VL-4B + Qwen3-4B	Qwen3-VL-4B + Qwen3-4B
Multimodal Reasoning
MATH-Vision	39.74	27.96	27.43	25.33	24.41	32.37	33.36	34.28
MathVerse (Total)	66.70	51.51	52.70	51.66	51.28	55.41	56.96	57.19
MathVista	76.26	66.56	68.60	67.36	66.42	67.70	69.62	70.58

Table 1: Results on multimodal reasoning, perception, and text reasoning. Bold marks the best distilled-student score. Evaluation details and SDs are in Appendices C.3 and E.1.1 .

	Fixed coefficient			Adaptive
Benchmark	$\alpha=1$	$\alpha=2$	$\alpha=4$	$\rho=1$	$\rho=2$	$\rho=4$
Multimodal Reasoning
MATH-Vision	31.91	31.25	30.00	33.36	34.28	34.01
MathVerse (Total)	53.46	54.29	53.41	56.96	57.19	57.51
MathVista	64.58	68.98	69.36	69.62	70.58	70.50
MMMU-Pro	39.48	38.87	37.45	40.98	41.38	41.36

Table 2: Adaptive calibration and the KL-budget scale. Generalized Bayesian composition with fixed coefficients or adaptive calibration. Best scores per group are green (fixed) and blue (adaptive). Reasoning standard deviations are in Appendix D.2 .

Teacher composition		Multimodal Reasoning			Text Reasoning
Language prior	Visual residual	MATH-Vision	MathVerse (Total)	Olympiad Bench-MM	MathVerse (Text Only)	Olympiad Bench-Text
Qwen3-4B	Qwen3-VL-4B	33.36	56.96	44.93	59.59	52.31
Qwen3-4B	Student (2B)	33.03 ( $-0.33$ )	54.17 ( $-2.79$ )	43.07 ( $-1.86$ )	59.92 ( $+0.33$ )	52.67 ( $+0.36$ )
Student (2B)	Qwen3-VL-4B	26.58 ( $-6.78$ )	52.69 ( $-4.27$ )	37.20 ( $-7.73$ )	56.98 ( $-2.61$ )	48.07 ( $-4.24$ )

Table 3: Contributions of the language and grounding experts. Student (2B) is the frozen initial Qwen3-VL-2B. All runs use

\rho=1

. Parentheses show pp changes from the full composition.

Teacher composition	Language expert swap		Base	Grounding expert swap
Language expert	LLM 1.7B	Student (2B)	LLM 4B	LLM 4B	LLM 4B
Grounding expert	VLM 4B	VLM 4B	VLM 4B	Student (2B)	VLM 8B
Multimodal Reasoning
MATH-Vision	28.36	26.58	33.36	33.03	34.34
MathVerse (Total)	43.89	52.69	56.96	54.17	57.81
MathVista	64.96	68.20	69.62	69.42	70.24

Table 4: Effect of expert choice on student reasoning. One expert is varied from the shaded Base. All runs use the same 2B student, 200 steps, and

\rho=1

. Student (2B) denotes the frozen initial student. When used as the language expert, it receives no image. Parentheses show average changes from Base in pp (blue increases, red decreases).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Hyperparameter	Value
Models	Student	Qwen3-VL-2B-Instruct
	Language expert	Qwen3-4B-Instruct-2507
	Grounding expert	Qwen3-VL-4B-Instruct
Training	Training data	MMFineReason, 54,840 examples
	Student update	Full-parameter fine-tuning
	Training steps	200

Appendix

Table 5: Training and evaluation settings used in our experiments. The training configuration corresponds to the main LEGO-OPD setup and serves as the reference configuration for the baseline comparisons. Baselines retain these settings whenever compatible with their defining formulations, with method-specific deviations described in Appendix C.5 . The reasoning-evaluation settings apply to sampled decoding and are held fixed across compared methods. Perception settings are given in Appendix C.3.2 .

Benchmark	Questions	Max. tokens	Appended instruction	Scoring
BLINK validation	1,901	128	“Reply with only the letter of the correct choice (A, B, C, …). Do not explain.”	Rule-based letter extraction and exact match, accuracy over all questions
QBench1,2 Group A (A1 perception, dev)	2,495 (1,495 + 1,000)	16	“Reply with only the letter of the correct choice (A, B, C, or D). Do not explain.”	Leading-letter extraction and exact match, accuracy over all pooled questions

Appendix

Table 6: Settings for direct-answer perception evaluation. Both benchmarks use one greedy response per question (temperature

0

, seed 42) and no system prompt. Max. tokens is the maximum number of generated tokens.

Panel of Figure 3 — Figure 3: Common QA prompt used for training and reasoning evaluation. Benchmark-specific input formatting is preserved during evaluation. MMLU-Pro uses its native five-shot prompt without the shared system prompt. Perception settings are given in Appendix C.3.2 .

Panel of Figure 4 — Figure 4: Prompt used by GPT-5.6 Luna for gold-blind semantic option mapping during scoring. The judge receives only the extracted final answer and candidate options, without the original question, image, or reference answer.

Figure 5: Illustrative example: a fixed coefficient cannot fit every prefix. Gray bars show the next-token distribution before the update (

q_{V}^{0}

q_{L}

), and colored bars show it after the update (

q_{V}^{I}

or the composed teacher). Top: at a reasoning prefix, the image barely changes the grounding expert (

D_{\mathrm{KL}}(q_{V}^{I}\|q_{V}^{0})=0.004

). With

\alpha=1

, however, the teacher moves

15.7\times

the source KL, and its argmax changes from

v_{2}

v_{1}

. Bottom: at a grounding prefix, the image determines the answer (

D_{\mathrm{KL}}(q_{V}^{I}\|q_{V}^{0})=0.832

). With

\alpha=1

, the teacher reaches only

0.66\times

the source KL and keeps

v_{2}

. KL calibration with

\rho=1

selects

\alpha^{\star}=0.26

and

1.15

, respectively, which match the source KL in both cases. Multiples in parentheses are relative to the source KL.

	Fixed coefficient			Adaptive calibration
Benchmark	$\alpha=1$	$\alpha=2$	$\alpha=4$	$\rho=1$	$\rho=2$	$\rho=4$
Multimodal Reasoning
MATH-Vision	$31.91\pm 1.09$	$31.25\pm 2.58$	$30.00\pm 1.42$	$33.36\pm 2.56$	$34.28\pm 2.06$	$34.01\pm 0.76$
MathVerse (Total)	$53.46\pm 0.44$	$54.29\pm 0.50$	$53.41\pm 0.69$	$56.96\pm 0.20$	$57.19\pm 0.44$	$57.51\pm 0.51$
MathVista	$64.58\pm 0.72$	$68.98\pm 0.79$	$69.36\pm 0.33$	$69.62\pm 0.72$	$70.58\pm 1.02$	$70.50\pm 0.55$
MMMU-Pro	$39.48\pm 0.77$	$38.87\pm 0.66$	$37.45\pm 0.64$	$40.98\pm 1.14$	$41.38\pm 0.72$	$41.36\pm 0.61$

Appendix

Table 7: Calibration ablations with evaluation variability. All runs use the same 2B student and 4B language and grounding experts for 200 training steps. Fixed composition uses constant

\alpha

, while adaptive calibration uses KL-budget scale

\rho

. Scores are mean

\pm

SD over five evaluation sampling seeds. Averages report means only.

	Base 2B	Distilled 2B students
Training data	–	Multimodal	Text-only	Multimodal	Multimodal
Grounding expert	–	–	–	Student (2B)	VLM 4B
KL-budget scale	–	$\rho=0$	$\rho=0$	$\rho=1$	$\rho=1$
Multimodal Reasoning
MATH-Vision	$27.96\pm 1.68$	$21.05\pm 2.17$	$33.75\pm 1.80$	$33.03\pm 1.35$	$33.36\pm 2.56$
MathVerse (Total)	$51.51\pm 0.48$	$30.25\pm 0.18$	$54.86\pm 0.53$	$54.17\pm 0.65$	$56.96\pm 0.20$

Appendix

Table 8: Language-only supervision and expert composition. Base 2B is the initial student before distillation. Student (2B) denotes its frozen copy used as the grounding expert. Scores are accuracies (%), reported as mean

\pm

SD over five evaluation sampling seeds. Benchmark averages report means only.

Figure 6: Training diagnostics for calibration support masses 0.95 and 0.99. The 4B–4B composition is trained at

\rho=1

with all other settings fixed. Panels compare training loss, top-

k

OPD loss, top-

k

KL estimate, subset entropy gap, student support mass, and teacher support mass over 200 optimizer steps.

Benchmark	$m=0.95$ (default)	$m=0.99$
Multimodal Reasoning
MATH-Vision	$33.36\pm 2.56$	$35.20\pm 1.45$
MathVerse (Total)	$56.96\pm 0.20$	$56.97\pm 0.54$
MathVista	$69.62\pm 0.72$	$69.62\pm 0.51$
MMMU-Pro	$40.98\pm 1.14$	$40.74\pm 0.47$
RBench-M	$29.17\pm 1.20$	$28.39\pm 0.79$

Appendix

Table 9: Effect of calibration support mass on final reasoning performance. Both runs use the same 2B student and 4B language and grounding experts for 200 steps at

\rho=1

. Accuracies (%) are mean

\pm

SD across five evaluation sampling seeds. Averages are unweighted means of the benchmark means.

Panel of Figure 7 — Figure 7: Local approximation error in self-prior recomposition. The 4B VLM supplies both its own image-free prior and visual residual, with $\rho=1$ . Panels compare calibration coefficients (a) and teacher KL relative to the VLM’s image-free prior (b). Curves show means within source-KL bins. The dotted $y=x$ line denotes source-KL matching. Shaded bands indicate question-bootstrap 95% confidence intervals.

Benchmark	Direct VLM supervision	Recomposed teacher	$\Delta$ (pp)
Multimodal Reasoning
MATH-Vision	$27.43\pm 0.89$	$28.62\pm 1.73$	$+1.19$
MathVerse (Total)	$52.70\pm 0.76$	$53.43\pm 0.38$	$+0.73$
MathVista	$68.60\pm 0.39$	$68.34\pm 1.28$	$-0.26$
MMMU-Pro	$36.98\pm 0.89$	$36.72\pm 0.51$	$-0.26$
RBench-M	$16.90\pm 0.52$	$13.86\pm 1.27$	$-3.04$

Appendix

Table 10: Practical recomposition: downstream distillation. Both conditions train the same 2B student for 200 steps using the 4B VLM. Recomposition uses the VLM’s own image-free prior, stabilized visual residual, and approximate calibration at

\rho=1

. Scores are mean

\pm

SD over five evaluation sampling seeds.

\Delta

is recomposed minus direct-supervision accuracy in pp. Averages report means only.

Benchmark	Base 4B	Base 2B	Standard OPD	VA-OPD	M-OPD	H-OPD	LEGO-OPD $\rho=1$
Multimodal Reasoning
MATH-Vision	$39.74\pm 2.25$	$27.96\pm 1.68$	$27.43\pm 0.89$	$25.33\pm 1.12$	$24.41\pm 1.73$	$32.37\pm 1.83$	$33.36\pm 2.56$
MathVerse (Total)	$66.70\pm 0.51$	$51.51\pm 0.48$	$52.70\pm 0.76$	$51.66\pm 0.36$	$51.28\pm 0.53$	$55.41\pm 0.65$	$56.96\pm 0.20$
MathVista	$76.26\pm 0.59$	$66.56\pm 0.71$	$68.60\pm 0.39$	$67.36\pm 0.93$	$66.42\pm 1.00$	$67.70\pm 0.79$	$69.62\pm 0.72$
MMMU-Pro	$50.29\pm 0.85$	$37.40\pm 0.66$	$36.98\pm 0.89$	$35.11\pm 1.16$	$35.06\pm 0.85$	$39.36\pm 0.69$	$40.98\pm 1.14$
RBench-M	$32.81\pm 1.34$	$19.61\pm 0.95$	$16.90\pm 0.52$	$16.81\pm 0.62$	$15.67\pm 1.30$	$26.92\pm 1.59$	$29.17\pm 1.20$

Appendix

Table 11: Main reasoning results with evaluation variability. Scores are accuracies (%, mean

\pm

SD across five sampling seeds). Base 2B is the initial student and Base 4B is the original VLM reference. All distilled models are 2B students evaluated at step 200. Averages report means only.

	4B reference	Initial student	Distillation baselines				LEGO-OPD
Benchmark	Base 4B	Base 2B	Standard OPD	VA-OPD	M-OPD	H-OPD	$\rho=1$	$\rho=2$
Step	–	–	200	200	200	200	200	200
Multimodal Reasoning
MATH-Vision	31.91	8.88	13.82	11.84	11.18	19.74	25.66	28.95
MathVerse (Total)	62.77	38.40	39.80	40.18	39.57	47.11	53.78	53.43
MathVista	75.10	58.70	60.20	60.70	62.50	65.30	68.30	70.60

Appendix

Table 12: Greedy decoding results across reasoning benchmarks. Scores are accuracies (%). Base 2B and Base 4B are the original Qwen3-VL-Instruct models without distillation. All distilled students are evaluated at step 200. Averages are unweighted means over the six multimodal or five text benchmarks. Bold indicates the best score among the 2B students, excluding the 4B reference.

	Language expert swap		Base	Grounding expert swap
Benchmark	LLM 1.7B VLM 4B	VLM 2B VLM 4B	LLM 4B VLM 4B	LLM 4B VLM 2B	LLM 4B VLM 8B
Multimodal Reasoning
MATH-Vision	$28.36\pm 2.84$	$26.58\pm 1.20$	$33.36\pm 2.56$	$33.03\pm 1.35$	$34.34\pm 1.77$
MathVerse (Total)	$43.89\pm 0.27$	$52.69\pm 0.54$	$56.96\pm 0.20$	$54.17\pm 0.65$	$57.81\pm 0.40$
MathVista	$64.96\pm 0.89$	$68.20\pm 0.47$	$69.62\pm 0.72$	$69.42\pm 1.13$	$70.24\pm 0.56$
MMMU-Pro	$35.11\pm 0.63$	$36.36\pm 0.77$	$40.98\pm 1.14$	$41.93\pm 0.69$	$41.70\pm 1.03$

Appendix

Table 13: Expert combinations with evaluation variability. Column headers list the language expert above the grounding expert. The shaded Base column uses the 4B language and grounding experts. VLM 2B is the frozen initial student and receives no image when used as the language expert. Scores are mean

\pm

SD over five evaluation sampling seeds. Averages report means only.

Benchmark	Standard OPD	Decomposed OPD (our implementation)
Multimodal reasoning
MATH-Vision	$27.43\pm 0.89$	$25.59\pm 1.82$
MathVerse (Total)	$52.70\pm 0.76$	$50.80\pm 0.30$
MathVista	$68.60\pm 0.39$	$63.94\pm 1.28$
MMMU-Pro	$36.98\pm 0.89$	$34.52\pm 0.56$
RBench-M	$16.90\pm 0.52$	$10.20\pm 0.91$

Appendix

Table 14: Results of our Decomposed OPD reproduction attempt at step 200. Scores are accuracies (%). Reasoning scores report mean

\pm

SD. Perception scores use greedy decoding. Benchmark averages report means only, computed as unweighted means over six multimodal or five text benchmarks. Decomposed OPD denotes our paper-based implementation.

Explore similar work

Aug 5, 2026cs.CV

OPD-V: Visual On-Policy Self-Distillation with Modality Balance

On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in multimodal large language models (MLLMs). Existing methods draw privileged information from diverse input sources to guide self-distillation. Yet these designs overlook Modality Imbalance, a challenge inherent to MLLM reasoning. When textual information dominates generation, the model cannot fully integrate its multimodal input. Consequently, carefully designed privileged information remains underused, limiting the effectiveness of OPSD. To examine this limitation, we construct a Positive Teacher with the Zoom-In Image and a Negative Teacher with the Mask Image, which exhibit different degrees of Modality Imbalance. Changes in their reasoning correctness and token logits reveal that Modality Balance can itself serve as privileged information. Motivated by this finding, we introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher. Positive Modality-Balance Logits Margins define a Modality-Balance Trust Region that selects the on-policy tokens used for self-distillation. Experiments across 6 benchmarks, 4 MLLM backbones, and 5 post-training methods show that OPD-V consistently improves reasoning performance while reducing training cost.

Aniri, Jinhe Bi, Peng Liao +5

2Ludwig Maximilian University of Munich · 3Munich Center for Machine Learning · 1National University of Singapore +1

Modality Imbalance On-Policy Self-Distillation VLM Distillation Multimodal Large Language Models VLM Reasoning Multimodal Learning

Ask PDF

Jul 1, 2026cs.CV

H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation

On-policy distillation (OPD) has recently emerged as an effective post-training paradigm by providing supervision on student-generated trajectories. However, existing OPD methods for multimodal reasoning usually rely on a static teacher routing, assigning each sample to a single teacher based on modality or task type. This ignores that visual grounding and abstract reasoning may dominate different decoding steps, making a single teacher insufficient for the full trajectory. To this end, H-OPD is proposed as a confidence-aware heterogeneous multi-teacher OPD framework for multimodal reasoning. By verifying the complementarity of heterogeneous teachers in the same reasoning process, H-OPD replaces task or sample level teacher routing with token-level teacher arbitration along the shared student trajectory. H-OPD employs vision-to-language description transfer to enable text-only teachers to access key visual semantics, and uses a confidence-aware arbitration mechanism to dynamically combine vision-language teacher and text-only teachers at each token. Extensive evaluations over 11 widely-used reasoning benchmarks showcase the superior performance of our method.

Qixiang Yin, Huanjin Yao, Yuchen Cai +4

Beijing University of Posts and Telecommunications · Zhongguancun Academy, Beijing, China · HKUST +2

Cross-Modal Knowledge Distillation VLM Distillation Multimodal Reasoning Multi-Teacher Knowledge Distillation On-Policy Distillation Multimodal Understanding

Ask PDF

Sep 15, 2026cs.LG

OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation

Privileged on-policy distillation improves multimodal reasoning by allowing a teacher to evaluate student trajectories using rich, training-only visual evidence. Both models score these trajectories while conditioning on the same student-generated prefix. When a student misinterprets an image early in a response, this accumulating erroneous rationale eventually pulls the teacher away from its visual evidence. The teacher and student converge on the same hallucination, causing standard cross-model supervision to collapse precisely where correction is most needed. We find that the teacher's visual corrective preference is not lost under this misleading agreement. Comparing the predictions of the identical teacher given the real image and a visual null reveals that the privileged evidence still pushes the model toward the correct interpretation. We introduce OPD-Aha, which reconstructs the distillation target directly from this isolated visual preference rather than relying on the fragile teacher-student discrepancy. This reconstructed target aggressively suppresses continuations that contradict the image. Trained with this objective, students learn to naturally interrupt their own flawed reasoning with reflection tokens such as wait and actually. After reflection, subsequent generation relies less on the accumulated erroneous text and more on the visual evidence. Correcting these trajectories mid-generation fundamentally alters the reasoning process, yielding broad and consistent improvements across diverse fine-grained perception and complex multimodal reasoning benchmarks. Our code and models are available at https://github.com/Echochef/OPD-Aha.

Chenhao Qiu, Dawei Li, Yechao Zhang +2

Arizona State University · University of Virginia · Stevens Institute of Technology

Object Hallucination in VLMs Multimodal Reasoning On-Policy Distillation

Ask PDF