M3OPD-Reasoner: An Open Recipe for Post-Training Multimodal Reasoning with Multi-Teacher On-Policy Distillation
Organizations: A*STAR · VinUniversity
Abstract
Multi-Teacher On-Policy Distillation (MOPD) enables a single student to absorb the skills of several domain teachers by learning from its own generated responses. We present Multimodal Model Multi-Teacher On-Policy Distillation Reasoner (M3OPD-Reasoner), a fully open recipe for post-training vision-language models (VLMs) over three reasoning domains: image STEM, image perception, and video. We curate two datasets, M3OPD-SFT-1.35M for supervised fine-tuning (SFT) and M3OPD-RL-193K for reinforcement learning (RL) with verifiable rewards. On these data, we compare post-training recipes that start from Qwen3.5-9B: training one model on the mixed data of all domains (MixSFT and MixRL), and training one teacher per domain with Group Sequence Policy Optimization (GSPO), directly or after an SFT warm-up, and distilling the teachers into one student with MOPD. We evaluate on 21 image STEM, image perception, and video reasoning benchmarks after we decontaminate the training data against them. The comparison establishes MOPD of teachers trained with GSPO directly from Qwen3.5-9B as the best recipe: the student closely matches each teacher on its own domain and is on average above the mixed-training baselines on the same prompts. Since MOPD already merges the domain capabilities that MixRL would otherwise have to learn, its student is a stronger starting point for RL, and we ask whether the same data can carry the model further from there: we add MixRL on top of the MOPD student, on the same prompts and with the same settings, which, to our knowledge, no prior VLM recipe has studied. The same prompts still improve the MOPD student, with no collapse, and give our main model, M3OPD-Reasoner-9B. MOPD and MixRL can therefore work as consecutive stages that get more out of the same data. We release all data, the training code, and the checkpoints of every stage.
Figures & tables
| Domain (avg@8, %) | Qwen3.8-Flash-Next traces | Self traces | |
|---|---|---|---|
| Image STEM | 60.6 | 63.4 | +2.8 |
| Image perception | 77.2 | 77.7 | +0.5 |
| Video | 59.2 | 59.1 | 0.1 |
| Prompts | Size | STEM | Perc. | Video | Overall |
|---|---|---|---|---|---|
| MOPD student (start) | – | 74.8 | 80.6 | 62.8 | 74.2 |
| Full union | 193,027 | 75.3 | 81.2 | 63.7 | 74.8 |
| Drop | 56,713 | 75.4 | 80.6 | 63.6 | 74.6 |
| Keep | 60,863 | 74.8 | 80.6 | 62.6 | 74.1 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | SFT | RL |
|---|---|---|
| Image STEM | ||
| MMFineReason-SFT [ 36 ] | 323,510 | – |
| MMVistaReason-SFT (math, science, logic) [ 37 ] | 30,990 | – |
| FineVision [ 76 ] : TabMWP, ScienceQA, CLEVR-Math | 13,802 | – |
| DeepVision-103K [ 62 ] | – | 23,486 |
| Vero-600K, STEM subsets [ 58 ] | – | 18,093 |
| Answer type | Image STEM | Image perception | Video | All |
|---|---|---|---|---|
| multiple choice | 28,889 | 10,831 | 30,853 | 70,573 |
| math | 30,732 | 23,643 | 2,071 | 56,446 |
| short answer | 4,522 | 6,479 | – | 11,001 |
| counting | 164 | 7,484 | 14,000 | 21,648 |
| OCR | – | 9,979 | – | 9,979 |
| grounding (box/point) | – | 9,499 | – | 9,499 |
| Check | Rule | ||
|---|---|---|---|
| Text, exact | same normalized question | ||
| Text, -gram | 13-gram overlap 0.5 of the item (or 0.9 incl. templates) | ||
| Image, hash | same SHA-1 | ||
| Image, pHash | 64-bit distance 4, confirmed by 256-bit 10 | ||
| Video, id | same video id (YouTube id of clips) | ||
| Video, frames | pHash of frames at 0.25, 0.5, 0.75 of the length, 4 | ||
| Type | Matching rule | Image STEM | Image perc. | Video |
|---|---|---|---|---|
| Math | Math-Verify, units, % = decimal | 30.7K | 23.6K | 2.1K |
| Mult. choice | option letter or letter set | 28.9K | 10.8K | 30.9K |
| Short | normalized text or number | 4.5K | 6.5K | – |
| Count | integer or number word | 164 | 7.5K | 14.0K |
| OCR | ANLS 0.5, digit guard | – | 10.0K | – |
| Box/point | matched IoU, point radius | – | 9.5K | – |
| Answer type | Format instruction (appended to the question) | Matcher | Details |
|---|---|---|---|
| multiple choice | Answer with the option letter only, inside \boxed{}. | option letter or set of letters | letter forms (“B”, “(B)”, “B. text”); multi-answer sets |
| math | Put your final answer within \boxed{}. | Math-Verify + numeric fallback | units must agree, % decimal, relations never match a value; rounding 6 digits |
| short answer | Put your final short answer within \boxed{}. | normalized string + numeric fallback | as math for numbers; multi-part answers part by part |
| counting | Put your final answer as a single integer within \boxed{}. | integer | number words, “about 7”, “7 birds” |
| OCR | Put the exact text of your final answer within \boxed{}. | ANLS | threshold 0.5; different digit sequences score 0 |
| grounding (box/point) | Output the box as [x1, y1, x2, y2] in 0-1000 relative coordinates inside \boxed{}. | Hungarian-matched IoU | coordinates in ; sum IoU / max(#pred, #gold); point gold: within 20 units or inside the box |
| Benchmark | Dataset | Subset | Split | |
|---|---|---|---|---|
| Image STEM | ||||
| DynaMath [ 110 ] | kcz358/DynaMath | – | test | 5,010 |
| EMMA [ 23 ] | lmms-lab/EMMA | All | test | 2,788 |
| LogicVista [ 81 ] | lscpku/LogicVista | – | test | 448 |
| MathVerse [ 104 ] | CaraJ/MathVerse-lmmseval | testmini | testmini | 3,940 |
| MathVista [ 44 ] | AI4Math/MathVista | – | testmini | 1,000 |
| Domain teachers | ||||
| SFT warm-up c | GSPO | MOPD | MixRL a | |
| Init | Qwen3.5-9B | Qwen3.5-9B c | Qwen3.5-9B | final MOPD |
| Data | domain SFT set | domain RL set | union (193K) | union (193K) |
| Learning rate | 1e-5, cosine | 1e-6, const. | 1e-6, const. | 1e-6, const. |
| Warmup | 3% | 20 steps | 20 steps | 20 steps |
| Weight decay / grad. norm | 0.0/1.0 | 0.1/1.0 | 0.1/1.0 | 0.1/1.0 |
| Setting | Image STEM | Image perception | Video | MixSFT |
|---|---|---|---|---|
| Data | ||||
| Initialization | Qwen3.5-9B | |||
| SFT samples | 368,302 | 625,993 | 357,304 | 1,351,599 |
| Tokens (total / target) | 396.8M / 284.2M | 686.8M / 180.8M | 539.3M / 73.3M | – |
| Trace generator | Qwen3.5-9B (self) | Qwen3.5-9B (self) | Qwen3.8-Flash-Next | as the domain runs |
| Chat template / thinking | Qwen3.5 native / on | |||
| Setting | Image STEM teacher | Image perception teacher | Video teacher | MixRL (baseline) | MixRL on top of MOPD |
|---|---|---|---|---|---|
| Per run | |||||
| Initialization | Qwen3.5-9B | Qwen3.5-9B | Qwen3.5-9B | MixSFT | final MOPD |
| RL prompts | 64,307 | 67,915 | 60,805 | 193,027 | 193,027 |
| Prompts per step / mini-batch | 256 / 64 | 256 / 64 | 256 / 64 | 252 / 63 | 256 / 64 |
| Optimizer updates per step | 4 | 4 | 4 | 4 | 4 |
| Max. prompt tokens | 2,560 | 2,048 | 5,632 | 5,632 | 5,632 |
| Setting | MOPD |
|---|---|
| Models and data | |
| Student initialization | Qwen3.5-9B |
| Teachers | image STEM, image perception, video GSPO checkpoints |
| Routing | hard, by the domain label of the prompt |
| Prompts | 193,027 (union of the three RL sets) |
| Objective | |
| Stage | Before | After |
|---|---|---|
| Domain RL Image STEM (DynaMath) | Qwen3.5-9B (wrong): “Wait, let’s look at the reflection again. …” The same check returns until the response limit, and no answer is given. | Domain teacher (correct): tries a rotation of the figure instead and checks it: “This matches the bottom text perfectly!” |
| Domain RL Video (Video-MME) | Qwen3.5-9B (correct): walks through the clip: “Around 00:10, the first dive sequence starts. It shows a solo dive. Wait, let me look closer. At 00:10, we see one diver. Then at 00:13, …” | Domain teacher (correct): lists only the relevant moments: “Around 00:43, the two athletes are diving together again.” |
| MOPD Video (Video-MME) | Domain teacher (correct): “The user wants to identify the location of the final scenes in the video. 1. Observe the video content from around 00:48 onwards.” | MOPD student (correct): “The user wants to identify the location of the final scenes in the video. 1. Look at the video from 00:48 onwards.” |
| MixRL on top of MOPD Video (Video-Holmes) | MOPD student (wrong): builds a timestamped chain on a misperceived event: “At 01:38, a man with a gun is seen firing.” | M 3 OPD-Reasoner-9B (correct): states the event that the question asks about: “The video shows the woman splitting the wall with a sickle, creating an opening.” |
| MixRL from MixSFT Image perception (CV-Bench) | MixSFT (wrong): “I need to count the umbrellas in the image. Looking at the scene, I see a group of colorful umbrellas …” | MixRL baseline (correct): same opening, then one recount: “I need to count the umbrellas in the image. Looking at the scene, … Wait, let me recount carefully.” |
| Solved | Length | |||||
|---|---|---|---|---|---|---|
| Model | ||||||
| Qwen3.5-9B | 0.84 | 0.80 | 0.77 | 0.81 | 0.69 | 0.30 |
| MixSFT | 0.69 | 0.71 | 0.76 | 0.58 | 0.44 | 0.22 |
| + MixRL (MixRL baseline) | 0.80 | 0.77 | 0.81 | 0.71 | 0.58 | 0.32 |
| MOPD, with SFT warm-up | 0.81 | 0.77 | 0.82 | 0.73 | 0.57 | 0.31 |
| MOPD, without SFT warm-up | 0.90 | 0.87 | 0.91 | 0.86 | 0.77 | 0.60 |
VLM Reasoning
Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.
See all 875 papers