BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptation

Anglin Liu; Yanlin Wu; Ruichao Chen; Yuting Zhang; Qingyuan Zeng; Pengxiang Cai; Ziqi Gong; Muchen Li

cs.AISep 27, 2026

BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptation

Authors: Anglin Liu, Yanlin Wu, Ruichao Chen, Yuting Zhang, Qingyuan Zeng, Pengxiang Cai, Ziqi Gong, Muchen Li, +1 more

Organizations: HKUST(GZ) · HKUST

Abstract

Adapting general-purpose multimodal large language models (MLLMs) to specialized domains requires learning domain-specific decision criteria, which often hinge on subtle visual distinctions between otherwise plausible answers. Rationale augmentation aims to expose such evidence through additional observations or inter-sample comparisons, yet a visually valid cue is not necessarily decision-relevant: it may describe how samples differ without changing the model's relative preference between competing answers. We therefore introduce BIRD, a self-improving Boundary-Informed Rationale Distillation framework that uses model-specific confusions to locate unresolved local decision boundaries and distills the evidence that resolves these confusions into rationales. For each sample, BIRD retrieves candidate neighbors from the target MLLM's own representation space and selects the most confusable one according to its answer preferences. It then generates answer-blind candidate evidence from their visual differences and functionally verifies which evidence most effectively strengthens the model's preference for the correct answer while avoiding inappropriate transfer across the pair. The verified evidence is then distilled into a single-sample rationale for standard supervised fine-tuning. Experiments on medical and chart VQA show that BIRD outperforms competing rationale-augmentation methods across two target MLLMs, while further analyses demonstrate clearer separation of confusable answers and stronger gains from model-matched supervision.

Figures & tables

Figure 1: Existing rationale augmentation methods leave confusable answers unresolved. BIRD verifies candidate factors with the target MLLM and distills the selected factor to resolve model-specific confusion.

Figure 2: Pilot study of model-specific boundaries. (a) Model-retrieved neighbors yield lower gold–confuser margins than random pairs, with each model most confused by neighbors retrieved from its own representation space. (b,c) Candidate factors exhibit heterogeneous decision effects, only a small subset falls in the shaded boundary-relevant region.

Figure 3: Overview of BIRD. The target MLLM retrieves model-specific neighbors, selects its most confusable alternative, verifies candidate boundary evidence with energy-based gain and cross-boundary transfer, and distills the validated evidence into a refined rationale for standard SFT.

			Chart / Plot VQA			Medical VQA
Method	Self	Inter	ChartQA	ChartBench	ChartQAPro	SLAKE	PathVQA	MedXpertQA	Avg.
Qwen3.5-9B
Base (no SFT)			85.67	67.42	41.35	73.23	47.12	27.18	57.00
Original SFT			88.36 $\uparrow$ 2.69	68.20 $\uparrow$ 0.78	49.02 $\uparrow$ 7.67	75.49 $\uparrow$ 2.26	49.53 $\uparrow$ 2.41	27.84 $\uparrow$ 0.66	59.74 $\uparrow$ 2.74
Reflective IT			88.48 $\uparrow$ 2.81	68.17 $\uparrow$ 0.75	50.33 $\uparrow$ 8.98	75.81 $\uparrow$ 2.58	49.35 $\uparrow$ 2.23	27.92 $\uparrow$ 0.74	60.01 $\uparrow$ 3.01
STaR	✓		87.64 $\uparrow$ 1.97	67.99 $\uparrow$ 0.57	49.26 $\uparrow$ 7.91	75.19 $\uparrow$ 1.96	49.40 $\uparrow$ 2.28	27.83 $\uparrow$ 0.65	59.55 $\uparrow$ 2.55

Table 1: Main results across two target MLLMs and two domain-adaptation settings. Higher is better. “Self” indicates self-improvement, while “Inter” denotes inter-sample supervision. Arrows indicate absolute changes relative to the corresponding base model without SFT.

Figure 4: Held-out hard-pair distributions of two-sided boundary separation for base, Original SFT, VC-STaR, and BIRD. A consistent rightward shift for BIRD indicates that it more effectively separates the gold answer from the model’s original confuser.

Figure 5: Cross-model boundary utility. Evidence discovered by each target MLLM achieves higher utility when evaluated by the same model, demonstrating model-specific boundary supervision.

		Chart / Plot VQA			Medical VQA
Target Model	BIRD Source	ChartQA	ChartBench	ChartQAPro	SLAKE	PathVQA	MedXpertQA	Avg.
Qwen3.5-9B	Qwen3.5-9B (Own)	89.84	71.55	52.96	77.92	51.49	28.06	61.97
	InternVL3-8B (Cross)	88.05 $\downarrow$ 1.79	69.73 $\downarrow$ 1.82	51.43 $\downarrow$ 1.53	76.42 $\downarrow$ 1.50	49.57 $\downarrow$ 1.92	26.67 $\downarrow$ 1.39	60.31 $\downarrow$ 1.66
InternVL3-8B	InternVL3-8B (Own)	87.06	69.42	51.28	76.31	51.97	23.69	59.96
	Qwen3.5-9B (Cross)	85.45 $\downarrow$ 1.61	67.88 $\downarrow$ 1.54	49.14 $\downarrow$ 2.14	74.56 $\downarrow$ 1.75	49.98 $\downarrow$ 1.99	22.34 $\downarrow$ 1.35	58.23 $\downarrow$ 1.73

Table 2: Cross-model transfer of BIRD supervision. “Own” denotes rationales constructed by the target MLLM itself, while “Cross” uses rationales constructed by the other MLLM.

Figure 6: Functional distribution of candidate evidence across models and domains. BIRD-selected factors concentrate in the high-own-side-gain, low-cross-boundary-transfer region.

	Chart / Plot VQA			Medical VQA
Selection	ChartQA	ChartBench	ChartQAPro	SLAKE	PathVQA	MedXpertQA	Avg.
Qwen3.5-9B
Random	88.71	68.82	50.21	76.12	50.04	27.74	60.27
Max $G_{\mathrm{own}}$	89.31 $\uparrow$ 0.60	69.94 $\uparrow$ 1.12	51.64 $\uparrow$ 1.43	76.91 $\uparrow$ 0.79	50.72 $\uparrow$ 0.68	27.95 $\uparrow$ 0.21	61.08 $\uparrow$ 0.81
Max $U$	89.84 $\uparrow$ 1.13	71.55 $\uparrow$ 2.73	52.96 $\uparrow$ 2.75	77.92 $\uparrow$ 1.80	51.49 $\uparrow$ 1.45	28.06 $\uparrow$ 0.32	61.97 $\uparrow$ 1.70
InternVL3-8B

Table 3: Effect of different boundary-evidence selection strategies. Changes are relative to random selection. All variants use the same candidate pool and update the same number of rationales.

	Chart / Plot VQA			Medical VQA
Variant	ChartQA	ChartBench	ChartQAPro	SLAKE	PathVQA	MedXpertQA	Avg.
Qwen3.5-9B
w/o Confuser Selection	89.50 $\downarrow$ 0.34	69.45 $\downarrow$ 2.10	51.55 $\downarrow$ 1.41	76.18 $\downarrow$ 1.74	50.50 $\downarrow$ 0.99	27.92 $\downarrow$ 0.14	60.88 $\downarrow$ 1.09
w/o Rationale Distillation	89.66 $\downarrow$ 0.18	70.88 $\downarrow$ 0.67	52.43 $\downarrow$ 0.53	77.31 $\downarrow$ 0.61	51.12 $\downarrow$ 0.37	27.97 $\downarrow$ 0.09	61.59 $\downarrow$ 0.38
BIRD	89.84	71.55	52.96	77.92	51.49	28.06	61.97
InternVL3-8B

Table 4: Ablation of confuser selection and rationale distillation. Arrows indicate absolute performance drops relative to full BIRD.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Fine-tuning type	Full-parameter SFT
Training epochs	10
Optimizer	AdamW
Learning rate	$2\times 10^{-5}$
Weight decay	$0.1$
Adam $(\beta_{1},\beta_{2})$	$(0.9,0.95)$

Appendix

Table 5: Training hyperparameters used for all model–domain settings.

\mathcal{B}_{i}=\{b_{i}^{k}=(e_{i}^{k},e_{i^{-}}^{k})\}_{k=1}^{K}

Appendix

Algorithm 1 Boundary-Informed Rationale Distillation (BIRD)

Figure 7: Per-benchmark analysis of decision-boundary sharpening. Each panel shows the distribution of two-sided boundary separation

S_{i}

on 50 fixed held-out hard pairs. Larger values indicate clearer separation between the gold answer and the pre-adaptation confuser.

Model	Domain	Pair	Candidate	Update	$\mathbf{G_{\mathrm{own}}}$	$\mathbf{G_{\mathrm{cross}}}$	$\mathbf{U}$
		Cov. (%)	Cov. (%)	Rate (%)
Qwen3.5-9B	Chart / Plot	93.8	89.7	70.9	1.25	0.17	0.91
	Medical	88.4	84.2	67.8	1.61	0.15	1.31
InternVL3-8B	Chart / Plot	92.9	88.6	68.6	1.14	0.15	0.84
	Medical	87.6	82.9	65.7	1.46	0.15	1.16

Appendix

Table 6: Statistics of BIRD rationale construction. Pair and candidate coverage, and update rate are percentages over the source training set.

G_{\mathrm{own}}

G_{\mathrm{cross}}

, and

U

are averaged over samples whose rationales are updated.

Figure 8: Rationale-length distributions of VC-STaR and BIRD for (a) Qwen3.5-9B and (b) InternVL3-8B. This indicates that BIRD does not obtain richer supervision simply by increasing rationale verbosity.

Figure 9: Sensitivity of BIRD to the retrieval size

P

, cross-boundary penalty

\lambda

, utility threshold

\tau_{U}

, and number of candidate factors

K

. The top and bottom rows report results on the chart and medical domains, respectively. Scores are averaged over the three benchmarks in each domain. Filled markers indicate the default settings (

P=3

\lambda=2.0

\tau_{U}=0.2

, and

K=3

), which consistently achieve the strongest performance across both target MLLMs and domains.

Figure 10: Chart VQA example. BIRD compares a target answered B with a compatible witness answered A and evaluates three answer-blind paired factors. Across-quarter balance (b2) is selected because it provides high own-side gain and little cross-boundary transfer, whereas peak height (b1) and end-point rise (b3) describe patterns shared by both charts.

Figure 11: Medical VQA example. For a target without pneumothorax and a compatible witness with pneumothorax, BIRD selects the pleural-boundary factor (b2), which achieves $G_{\mathrm{own}}=1.10$ , $G_{\mathrm{cross}}=0.07$ , and $U=0.96$ . Regional density and extent are less boundary-specific and are therefore not selected.

Figure 12: Why high-gain evidence can still be unsuitable. The partial-height factor (b1) strongly benefits the target but barely helps the witness and transfers across the pair, resulting in negative boundary utility. The pairwise B–C ordering (b2) supports the appropriate answer on both sides and nearly disappears when swapped, allowing it to identify the local decision boundary.

Figure 13: Model-specific boundary evidence for a shared target sample. Qwen3.5-9B confuses the gold answer $30$ with $40$ and assigns the highest utility to legend-binding evidence, whereas InternVL3-8B confuses it with $70$ and favors segment-span evidence. Even under fixed-pair verification, the two models rank the same factors differently, leading to model-matched rationale supervision.

Explore similar work

May 27, 2026cs.CV

Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoning

Despite their popularity and success, Multimodal Large Language Models (MLLMs) often struggle to interpret images accurately, which limits their reasoning capability in complex scenarios (e.g., high object density and complex background clutter). Prior work mainly addresses this limitation by incorporating explicit visual cues like bounding boxes that require extra annotations. In addition, the resulting low-resolution crops often miss fine-grained details that MLLMs require for accurate reasoning. Therefore, we propose Mags-RL, an Agentic Reinforcement Learning (RL) framework that equips MLLMs with an external super-resolution "magnifying glass" agent for high-resolution fine-grained inspection. Specifically, the model performs two-round reasoning: in the first round, it generates an initial rationale and autonomously identifies regions of interest without relying on additional annotations; in the second round, it invokes a super-resolution agent to crop and upscale those regions, then revisits and verifies its earlier reasoning to produce the final answer. We also introduce a novel curriculum learning strategy that enables data-efficient RL training, needing as few as only 40 training samples to achieve reasonable performance. Experiments on VSR, TallyQA, and GQA subsets show its superior performance against recent strong competing methods, demonstrating high-quality reasoning with precise visual grounding. Code and weights will be released soon.

Xuanzhao Dong, Wenhui Zhu, Peijie Qiu +11

Arizona State University, Tempe, AZ, USA · Washington University in St. Louis, St. Louis, MO, USA · Clemson University, Clemson, SC, USA +3

Multimodal Large Language Models Recent Vision-Language Models Scene Reasoning Icr-Rl Agentic Reinforcement Learning Generative Super-Resolution

Ask PDF

Jun 1, 2026cs.CV

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

The effectiveness of Chain-of-Thought (CoT) prompting in Multimodal Large Language Models (MLLMs) remains uncertain: across several visual reasoning benchmarks, CoT prompting often degrades performance compared to direct prompting. In this paper, we provide a systematic analysis of CoT behavior in three modern MLLM families across model scales on datasets requiring step-wise visual evidence. Our analysis identifies two recurring failure modes: premature answer commitment and limited direct visual-token access during rationale generation. We further find that standard CoT-style Supervised Fine-Tuning (CoT-SFT) can mitigate these issues only partially, while often increasing reliance on textual priors and reducing counterfactual visual dependence. Motivated by these findings, we propose Attentive-CoT (Att-CoT), an attention-guided fine-tuning objective that encourages CoT trajectories to delay answer commitment while maintaining sustained visual-token access. Att-CoT can be plugged into any CoT-SFT training run without architectural changes. Experiments on three visual reasoning benchmarks across six MLLMs show that Att-CoT enhances CoT performance over standard fine-tuning.

Sanchit Sinha, Guangzhi Xiong, Bohan Liu +2

University of Virginia

Multimodal Large Language Models Chain-of-Thought Reasoning Recent Vision-Language Models Attention-Guided Training Textual Priors Artificial Intelligence Assistants

Ask PDF

Sep 21, 2026cs.CV

Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning

Multimodal large language models (MLLMs) fail at fine-grained visual questions less because they cannot reason than because they never see the evidence: high-resolution images are downsampled before encoding, so the model answers from linguistic priors. The standard remedies are expensive: annotated answers (SFT), hand-engineered verifiers (RLVR), or a large external teacher (on-policy distillation). We ask whether the visual evidence itself can supply the signal for free. We formalize the contrastive evidence gap, the per-token log-likelihood ratio that a model assigns to its own output when conditioned on a question-relevant region versus an irrelevant one, and study it across Qwen2.5-VL-7B, Qwen3-VL-8B, and Qwen3-VL-30B-A3B on V*Bench. Our main positive result is training-free: selecting the candidate crop under which the model's answer distribution is most peaked, using a single-view, label-free criterion, discovers the answer-bearing region with no bounding boxes, training, or labels. It localizes the target 4.4 to 5.1 times better than chance and raises fine-grained accuracy from 70 percent to 85 percent at inference. We further show that the gap is complementary to the model's own confidence. Combining them predicts correctness better than either alone, with AUC up to 0.99, and flags confidently wrong answers, with AUC ranging from 0.97 to 1.00 within the high-confidence subset. All effects concentrate on perception-bottleneck questions and vanish on a global-context control. Finally, we report an honest negative result: converting the same signal into a training method, gated self-distillation (SEG-Distill), does not outperform the base model at pilot scale across three gate designs, while more aggressive gating degrades accuracy. The signal is real, but converting it into training gains remains an open problem.

Santi Ram Tiwari, Nihal Naik, Devbrat Pandey +1

KGraph AI Solutions Pvt. Ltd. Bangalore, India – 560016

Fine-Grained Perception Recent Vision-Language Models Visual Evidence Multimodal Large Language Models Label-Free Evidence Conflict

Ask PDF