BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptation

Anglin Liu; Yanlin Wu; Ruichao Chen; Yuting Zhang; Qingyuan Zeng; Pengxiang Cai; Ziqi Gong; Muchen Li

cs.AISep 27, 2026

BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptation

Authors: Anglin Liu, Yanlin Wu, Ruichao Chen, Yuting Zhang, Qingyuan Zeng, Pengxiang Cai, Ziqi Gong, Muchen Li, +1 more

Organizations: HKUST(GZ) · HKUST

Abstract

Adapting general-purpose multimodal large language models (MLLMs) to specialized domains requires learning domain-specific decision criteria, which often hinge on subtle visual distinctions between otherwise plausible answers. Rationale augmentation aims to expose such evidence through additional observations or inter-sample comparisons, yet a visually valid cue is not necessarily decision-relevant: it may describe how samples differ without changing the model's relative preference between competing answers. We therefore introduce BIRD, a self-improving Boundary-Informed Rationale Distillation framework that uses model-specific confusions to locate unresolved local decision boundaries and distills the evidence that resolves these confusions into rationales. For each sample, BIRD retrieves candidate neighbors from the target MLLM's own representation space and selects the most confusable one according to its answer preferences. It then generates answer-blind candidate evidence from their visual differences and functionally verifies which evidence most effectively strengthens the model's preference for the correct answer while avoiding inappropriate transfer across the pair. The verified evidence is then distilled into a single-sample rationale for standard supervised fine-tuning. Experiments on medical and chart VQA show that BIRD outperforms competing rationale-augmentation methods across two target MLLMs, while further analyses demonstrate clearer separation of confusable answers and stronger gains from model-matched supervision.

Figures & tables

Figure 1: Existing rationale augmentation methods leave confusable answers unresolved. BIRD verifies candidate factors with the target MLLM and distills the selected factor to resolve model-specific confusion.

Figure 2: Pilot study of model-specific boundaries. (a) Model-retrieved neighbors yield lower gold–confuser margins than random pairs, with each model most confused by neighbors retrieved from its own representation space. (b,c) Candidate factors exhibit heterogeneous decision effects, only a small subset falls in the shaded boundary-relevant region.

Figure 3: Overview of BIRD. The target MLLM retrieves model-specific neighbors, selects its most confusable alternative, verifies candidate boundary evidence with energy-based gain and cross-boundary transfer, and distills the validated evidence into a refined rationale for standard SFT.

			Chart / Plot VQA			Medical VQA
Method	Self	Inter	ChartQA	ChartBench	ChartQAPro	SLAKE	PathVQA	MedXpertQA	Avg.
Qwen3.5-9B
Base (no SFT)			85.67	67.42	41.35	73.23	47.12	27.18	57.00
Original SFT			88.36 $\uparrow$ 2.69	68.20 $\uparrow$ 0.78	49.02 $\uparrow$ 7.67	75.49 $\uparrow$ 2.26	49.53 $\uparrow$ 2.41	27.84 $\uparrow$ 0.66	59.74 $\uparrow$ 2.74
Reflective IT			88.48 $\uparrow$ 2.81	68.17 $\uparrow$ 0.75	50.33 $\uparrow$ 8.98	75.81 $\uparrow$ 2.58	49.35 $\uparrow$ 2.23	27.92 $\uparrow$ 0.74	60.01 $\uparrow$ 3.01
STaR	✓		87.64 $\uparrow$ 1.97	67.99 $\uparrow$ 0.57	49.26 $\uparrow$ 7.91	75.19 $\uparrow$ 1.96	49.40 $\uparrow$ 2.28	27.83 $\uparrow$ 0.65	59.55 $\uparrow$ 2.55

Table 1: Main results across two target MLLMs and two domain-adaptation settings. Higher is better. “Self” indicates self-improvement, while “Inter” denotes inter-sample supervision. Arrows indicate absolute changes relative to the corresponding base model without SFT.

Figure 4: Held-out hard-pair distributions of two-sided boundary separation for base, Original SFT, VC-STaR, and BIRD. A consistent rightward shift for BIRD indicates that it more effectively separates the gold answer from the model’s original confuser.

Figure 5: Cross-model boundary utility. Evidence discovered by each target MLLM achieves higher utility when evaluated by the same model, demonstrating model-specific boundary supervision.

		Chart / Plot VQA			Medical VQA
Target Model	BIRD Source	ChartQA	ChartBench	ChartQAPro	SLAKE	PathVQA	MedXpertQA	Avg.
Qwen3.5-9B	Qwen3.5-9B (Own)	89.84	71.55	52.96	77.92	51.49	28.06	61.97
	InternVL3-8B (Cross)	88.05 $\downarrow$ 1.79	69.73 $\downarrow$ 1.82	51.43 $\downarrow$ 1.53	76.42 $\downarrow$ 1.50	49.57 $\downarrow$ 1.92	26.67 $\downarrow$ 1.39	60.31 $\downarrow$ 1.66
InternVL3-8B	InternVL3-8B (Own)	87.06	69.42	51.28	76.31	51.97	23.69	59.96
	Qwen3.5-9B (Cross)	85.45 $\downarrow$ 1.61	67.88 $\downarrow$ 1.54	49.14 $\downarrow$ 2.14	74.56 $\downarrow$ 1.75	49.98 $\downarrow$ 1.99	22.34 $\downarrow$ 1.35	58.23 $\downarrow$ 1.73

Table 2: Cross-model transfer of BIRD supervision. “Own” denotes rationales constructed by the target MLLM itself, while “Cross” uses rationales constructed by the other MLLM.

Figure 6: Functional distribution of candidate evidence across models and domains. BIRD-selected factors concentrate in the high-own-side-gain, low-cross-boundary-transfer region.

	Chart / Plot VQA			Medical VQA
Selection	ChartQA	ChartBench	ChartQAPro	SLAKE	PathVQA	MedXpertQA	Avg.
Qwen3.5-9B
Random	88.71	68.82	50.21	76.12	50.04	27.74	60.27
Max $G_{\mathrm{own}}$	89.31 $\uparrow$ 0.60	69.94 $\uparrow$ 1.12	51.64 $\uparrow$ 1.43	76.91 $\uparrow$ 0.79	50.72 $\uparrow$ 0.68	27.95 $\uparrow$ 0.21	61.08 $\uparrow$ 0.81
Max $U$	89.84 $\uparrow$ 1.13	71.55 $\uparrow$ 2.73	52.96 $\uparrow$ 2.75	77.92 $\uparrow$ 1.80	51.49 $\uparrow$ 1.45	28.06 $\uparrow$ 0.32	61.97 $\uparrow$ 1.70
InternVL3-8B

Table 3: Effect of different boundary-evidence selection strategies. Changes are relative to random selection. All variants use the same candidate pool and update the same number of rationales.

	Chart / Plot VQA			Medical VQA
Variant	ChartQA	ChartBench	ChartQAPro	SLAKE	PathVQA	MedXpertQA	Avg.
Qwen3.5-9B
w/o Confuser Selection	89.50 $\downarrow$ 0.34	69.45 $\downarrow$ 2.10	51.55 $\downarrow$ 1.41	76.18 $\downarrow$ 1.74	50.50 $\downarrow$ 0.99	27.92 $\downarrow$ 0.14	60.88 $\downarrow$ 1.09
w/o Rationale Distillation	89.66 $\downarrow$ 0.18	70.88 $\downarrow$ 0.67	52.43 $\downarrow$ 0.53	77.31 $\downarrow$ 0.61	51.12 $\downarrow$ 0.37	27.97 $\downarrow$ 0.09	61.59 $\downarrow$ 0.38
BIRD	89.84	71.55	52.96	77.92	51.49	28.06	61.97
InternVL3-8B

Table 4: Ablation of confuser selection and rationale distillation. Arrows indicate absolute performance drops relative to full BIRD.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Fine-tuning type	Full-parameter SFT
Training epochs	10
Optimizer	AdamW
Learning rate	$2\times 10^{-5}$
Weight decay	$0.1$
Adam $(\beta_{1},\beta_{2})$	$(0.9,0.95)$

Appendix

Table 5: Training hyperparameters used for all model–domain settings.

\mathcal{B}_{i}=\{b_{i}^{k}=(e_{i}^{k},e_{i^{-}}^{k})\}_{k=1}^{K}

Appendix

Algorithm 1 Boundary-Informed Rationale Distillation (BIRD)

Figure 7: Per-benchmark analysis of decision-boundary sharpening. Each panel shows the distribution of two-sided boundary separation

S_{i}

on 50 fixed held-out hard pairs. Larger values indicate clearer separation between the gold answer and the pre-adaptation confuser.

Model	Domain	Pair	Candidate	Update	$\mathbf{G_{\mathrm{own}}}$	$\mathbf{G_{\mathrm{cross}}}$	$\mathbf{U}$
		Cov. (%)	Cov. (%)	Rate (%)
Qwen3.5-9B	Chart / Plot	93.8	89.7	70.9	1.25	0.17	0.91
	Medical	88.4	84.2	67.8	1.61	0.15	1.31
InternVL3-8B	Chart / Plot	92.9	88.6	68.6	1.14	0.15	0.84
	Medical	87.6	82.9	65.7	1.46	0.15	1.16

Appendix

Table 6: Statistics of BIRD rationale construction. Pair and candidate coverage, and update rate are percentages over the source training set.

G_{\mathrm{own}}

G_{\mathrm{cross}}

, and

U

are averaged over samples whose rationales are updated.

Figure 8: Rationale-length distributions of VC-STaR and BIRD for (a) Qwen3.5-9B and (b) InternVL3-8B. This indicates that BIRD does not obtain richer supervision simply by increasing rationale verbosity.

Figure 9: Sensitivity of BIRD to the retrieval size

P

, cross-boundary penalty

\lambda

, utility threshold

\tau_{U}

, and number of candidate factors

K

. The top and bottom rows report results on the chart and medical domains, respectively. Scores are averaged over the three benchmarks in each domain. Filled markers indicate the default settings (

P=3

\lambda=2.0

\tau_{U}=0.2

, and

K=3

), which consistently achieve the strongest performance across both target MLLMs and domains.

Figure 10: Chart VQA example. BIRD compares a target answered B with a compatible witness answered A and evaluates three answer-blind paired factors. Across-quarter balance (b2) is selected because it provides high own-side gain and little cross-boundary transfer, whereas peak height (b1) and end-point rise (b3) describe patterns shared by both charts.

Figure 11: Medical VQA example. For a target without pneumothorax and a compatible witness with pneumothorax, BIRD selects the pleural-boundary factor (b2), which achieves $G_{\mathrm{own}}=1.10$ , $G_{\mathrm{cross}}=0.07$ , and $U=0.96$ . Regional density and extent are less boundary-specific and are therefore not selected.

Figure 12: Why high-gain evidence can still be unsuitable. The partial-height factor (b1) strongly benefits the target but barely helps the witness and transfers across the pair, resulting in negative boundary utility. The pairwise B–C ordering (b2) supports the appropriate answer on both sides and nearly disappears when swapped, allowing it to identify the local decision boundary.

Figure 13: Model-specific boundary evidence for a shared target sample. Qwen3.5-9B confuses the gold answer $30$ with $40$ and assigns the highest utility to legend-binding evidence, whereas InternVL3-8B confuses it with $70$ and favors segment-span evidence. Even under fixed-pair verification, the two models rank the same factors differently, leading to model-matched rationale supervision.

Explore similar work

Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoning
May 27, 2026Xuanzhao Dong, Wenhui Zhu, Peijie Qiu +11Multimodal Large Language Models Recent Vision-Language Models
Arizona State University, Tempe, AZ, USA · Washington University in St. Louis, St. Louis, MO, USA · Clemson University, Clemson, SC, USA +3
Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Jun 1, 2026Sanchit Sinha, Guangzhi Xiong, Bohan Liu +2Multimodal Large Language Models Chain-of-Thought Reasoning
University of Virginia
Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning
Sep 21, 2026Santi Ram Tiwari, Nihal Naik, Devbrat Pandey +1Fine-Grained Perception Recent Vision-Language Models
KGraph AI Solutions Pvt. Ltd. Bangalore, India – 560016

May 27, 2026, cs.CVJ/K move · Enter open · S save

Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoning

Xuanzhao Dong, Wenhui Zhu, Peijie Qiu, Xiwen Chen+10

Arizona State University, Tempe, AZ, USA · Washington University in St. Louis, St. Louis, MO, USA · Clemson University, Clemson, SC, USA+3

Multimodal Large Language Models Recent Vision-Language Models Scene Reasoning Icr-Rl Agentic Reinforcement Learning Generative Super-Resolution

Open paper AskPDF arXiv

Despite their popularity and success, Multimodal Large Language Models (MLLMs) often struggle to interpret images accurately, which limits their reasoning capability in complex scenarios (e.g., high object density and complex background clutter). Prior work mainly addresses this limitation by incorporating explicit visual cues like bounding boxes that require extra annotations. In addition, the resulting low-resolution crops often miss fine-grained details that MLLMs require for accurate reasoning. Therefore, we propose Mags-RL, an Agentic Reinforcement Learning (RL) framework that equips MLLMs with an external super-resolution "magnifying glass" agent for high-resolution fine-grained inspection. Specifically, the model performs two-round reasoning: in the first round, it generates an initial rationale and autonomously identifies regions of interest without relying on additional annotations; in the second round, it invokes a super-resolution agent to crop and upscale those regions, then revisits and verifies its earlier reasoning to produce the final answer. We also introduce a novel curriculum learning strategy that enables data-efficient RL training, needing as few as only 40 training samples to achieve reasonable performance. Experiments on VSR, TallyQA, and GQA subsets show its superior performance against recent strong competing methods, demonstrating high-quality reasoning with precise visual grounding. Code and weights will be released soon.

Abstract

Explore similar work

Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoning

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning