Multimodal Abstention That Preserves Acquired Correct Answers
Abstract
Adding abstention to a task-adapted multimodal model can discard correct answers, including capabilities acquired during adaptation. We formulate abstention repair as maximizing correct abstention on unanswerable questions under a budget on newly lost correct answers. We propose ReFrain (Regression-Free Abstention), which combines a linear probe on first-token logits, threshold calibration on held-out image clusters, and a deterministic veto that replaces rejected outputs with the displayed "none of the above" option while preserving all other responses. "Regression-free" denotes zero loss on the calibration set at the zero-budget target. All abstention methods in our main comparison start from the same answering-SFT checkpoint; we separately track lost correct answers, repaired errors, and retained SFT gains. On Qwen2.5-VL-3B, ReFrain at zero budget retains all 57 SFT-gained answers with no losses and improves correct abstention by 7.30 percentage points. At a 0.5% calibration budget, it reaches 74.05% correct abstention with 8 losses and retains 55 gains; CLIP-UP EmbLoRA reaches 64.65% with 62 losses, 41 repairs, and 40 retained gains. The same probe at its default threshold loses 298 correct answers. With models and thresholds frozen, confirmation on 228 independent question clusters yields 66.05% correct abstention and no losses at the 0.5% point, exceeding EmbLoRA by 32.72 percentage points. Ordinary refusal SFT achieves higher correct abstention at greater regression cost. Code and reproduction materials: https://github.com/ModeLiberaL/ReFrain.
Figures & tables
| Resource | Rule |
|---|---|
| Answering starting point | The same B1 checkpoint |
| Fitting or training | only |
| Selection of operating point or model | only |
| Evaluation | and the confirmation set (Section 3.7 ); no threshold tuning or checkpoint selection on them |
| Parsing and decoding | The same parser, the same NOTA mapping, greedy decoding |
| Readouts | All readouts of Section 3.6 |
| Family | What it does | Modifies | Can rewrite | Representative methods |
|---|---|---|---|---|
| Output-level override | Freezes ; a scalar score decides whether the output is replaced with NOTA | No | No | ReFrain (this paper); the LVLM-LP probe abstaining at its default threshold [ 34 ] , the CLIP-UP correlation-vector classifier [ 22 ] , BICR [ 13 ] , ACR [ 20 ] |
| Parameter update | Continues training on abstention examples | Yes | Yes | Plain abstention SFT, CLIP-UP EmbLoRA (which also injects CLIP correlation vectors) [ 22 ] , MAD-MED [ 10 ] , SAVOR [ 4 ] , VL-Calibration [ 27 ] , Reliable-V-RL [ 11 ] |
| Input or representation injection | Adds a signal to the input embeddings or to intermediate activations | Only the added projection or steering parameters | Yes | CLIP-UP Emb [ 22 ] , Dr. No [ 7 ] |
| Inference-time sampling | Perturbs the input several times and decides by the consistency of the outputs | No | Depends on the execution | VL-Uncertainty [ 33 ] |
| Learned routing | Trains the model to output an answer-or-abstain decision before answering | Yes | Yes | Rep2Act [ 8 ] |
| Prompting | Changes the instruction only | No | Yes | The additional-instruction setting of MM-UPD [ 16 ] |
| Original base model | Scheme | Standard (%) | UPD (%) | Joint criterion |
| Qwen2.5-VL-3B | No abstention stage | 86.72 | 56.43 | reference |
| ReFrain | 86.69 | 67.85 | passed | |
| CLIP-UP EmbLoRA | 86.76 | 65.82 | passed | |
| CLIP-UP Emb | 86.21 | 68.01 | not passed | |
| VL-Uncertainty (native execution) | 80.16 | 75.36 | not passed | |
| VL-Uncertainty score, common executor | 86.70 | 56.63 | passed |
| Branch | Standard (%) | UPD (%) | Circular Dual (%) | Lost / corrected | Retained |
|---|---|---|---|---|---|
| B0 | 86.72 | 56.43 | 40.26 | — | — |
| B1 | 87.22 | 54.63 | 38.86 | 0 / 0 | 57 / 57 |
| ReFrain | 87.22 | 61.94 | 46.22 | 0 / 0 | 57 / 57 |
| ReFrain ( ) | 87.08 | 74.05 | 55.73 | 8 / 0 | 55 / 57 |
| CLIP-UP EmbLoRA | 86.85 | 64.65 | 47.82 | 62 / 41 | 40 / 57 |
| Plain abstention SFT | 85.32 | 87.47 | 66.41 | 140 / 33 | 23 / 57 |
| Scorer | (%) | Standard (%) | UPD (%) | Lost | Retained |
|---|---|---|---|---|---|
| Cross-validated probe (default) | 0 | 87.22 | 61.94 | 0 | 57 / 57 |
| Probe with fixed | 0 | 87.22 | 59.88 | 0 | 57 / 57 |
| Cross-validated probe (default) | 0.5 | 87.08 | 74.05 | 8 | 55 / 57 |
| Probe with fixed | 0.5 | 86.99 | 75.40 | 13 | 55 / 57 |
| MSP | 0.5 | 86.97 | 59.34 | 14 | 54 / 57 |
| Negative entropy | 0.5 | 87.17 | 57.30 | 3 | 57 / 57 |
| Classifier fit | Operating point | Standard (%) | UPD (%) | Lost | Retained | Joint criterion |
|---|---|---|---|---|---|---|
| Fixed | Default threshold ( ) | 48.99 | 92.40 | 2151 | 30 / 57 | not passed |
| Calibrated, | 87.22 | 55.09 | 0 | 57 / 57 | passed | |
| Calibrated, | 86.70 | 57.46 | 29 | 55 / 57 | not passed | |
| Cross-validated | Default threshold ( ) | 66.57 | 85.25 | 1162 | 51 / 57 | not passed |
| Calibrated, | 87.22 | 54.75 | 0 | 57 / 57 | not passed | |
| Calibrated, | 87.08 | 56.59 | 8 | 57 / 57 | passed |
| Feature | Decision rule | Standard (%) | UPD (%) | Lost | Retained | Joint criterion |
|---|---|---|---|---|---|---|
| CLIP correlation vector | Default threshold ( ) | 66.57 | 85.25 | 1162 | 51 / 57 | not passed |
| Calibrated, | 87.08 | 56.59 | 8 | 57 / 57 | passed | |
| First-token logits | Default threshold ( ) | 81.92 | 92.16 | 298 | 45 / 57 | not passed |
| Calibrated, | 87.08 | 74.05 | 8 | 55 / 57 | passed |
| Features | Operating point | Standard (%) | UPD (%) | Lost | Retained |
|---|---|---|---|---|---|
| Raw | Default threshold ( ) | 85.76 | 67.00 | 82 | 25 / 57 |
| Calibrated, | 87.22 | 55.56 | 0 | 57 / 57 | |
| Calibrated, | 86.97 | 59.32 | 14 | 54 / 57 | |
| Standardized | Default threshold ( ) | 82.28 | 82.21 | 278 | 23 / 57 |
| Calibrated, | 87.22 | 57.98 | 0 | 57 / 57 | |
| Calibrated, | 86.58 | 67.20 | 36 | 53 / 57 |
| Operating point | Standard (%) | UPD (%) | Lost | Retained | Joint criterion |
|---|---|---|---|---|---|
| Default threshold ( ) | 66.81 | 95.19 | 1148 | 22 / 57 | not passed |
| Calibrated, | 87.22 | 55.48 | 0 | 57 / 57 | passed |
| Calibrated, | 87.04 | 58.23 | 10 | 57 / 57 | passed |
| Task | Scheme | Standard (%) | UPD (%) | Lost | Retained |
|---|---|---|---|---|---|
| AAD | ReFrain ( ) | 88.32 | 60.47 | 3 | 20 / 21 |
| CLIP-UP EmbLoRA | 88.01 | 38.80 | 25 | 14 / 21 | |
| Plain abstention SFT | 86.39 | 80.39 | 58 | 7 / 21 | |
| IASD | ReFrain ( ) | 86.31 | 85.38 | 4 | 22 / 23 |
| CLIP-UP EmbLoRA | 85.87 | 77.35 | 32 | 15 / 23 | |
| Plain abstention SFT | 84.24 | 94.09 | 70 | 8 / 23 |
| Branch | Standard (%) | UPD, micro (%) | UPD, macro (%) | Lost / corrected | Retained | Joint criterion |
|---|---|---|---|---|---|---|
| B1 | 91.67 | 21.69 | 21.29 | 0 / 0 | 8 / 8 | reference |
| ReFrain | 91.67 | 41.79 | 37.95 | 0 / 0 | 8 / 8 | passed |
| ReFrain ( ) | 91.67 | 66.05 | 60.10 | 0 / 0 | 8 / 8 | passed |
| CLIP-UP EmbLoRA | 92.32 | 33.33 | 33.33 | 7 / 13 | 6 / 8 | passed |
| Plain abstention SFT | 91.78 | 70.96 | 66.75 | 4 / 5 | 6 / 8 | passed |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Scorer | Losses on | Loss rate on (%) | Losses on | Loss rate on (%) |
|---|---|---|---|---|
| Cross-validated probe (default) | 11 | 0.46 | 8 | 0.14 |
| Probe with fixed | 12 | 0.50 | 13 | 0.23 |
| MSP | 7 | 0.29 | 14 | 0.25 |
| Negative entropy | 8 | 0.33 | 3 | 0.05 |
| CLIP classifier, fixed | 12 | 0.50 | 29 | 0.52 |
| CLIP classifier, cross-validated | 12 | 0.50 | 8 | 0.14 |
| Nominal budget on (%) | Correct-abstention rate on (%) | Losses on | Loss rate on (%) |
| 0 | 61.94 | 0 | 0.00 |
| 0.1 | 66.83 | 0 | 0.00 |
| 0.25 | 71.81 | 5 | 0.09 |
| 0.5 | 74.05 | 8 | 0.14 |
| 1 | 82.90 | 70 | 1.24 |
| Scheme | SFT-gained records lost (%) | Persistently correct records lost (%) | Difference [95% interval] (pp) |
|---|---|---|---|
| ReFrain ( ) | 2 / 57 (3.51) | 6 / 4,850 (0.12) | 3.39 [ 0.22, 12.25] |
| CLIP-UP EmbLoRA | 17 / 57 (29.82) | 45 / 4,850 (0.93) | 28.90 [12.82, 49.88] |
| Plain abstention SFT | 34 / 57 (59.65) | 106 / 4,850 (2.19) | 57.46 [37.68, 79.47] |
| Type | Reviewed | Excluded | Final | Reasons for exclusion |
|---|---|---|---|---|
| AAD | 100 | 4 | 96 | Options overlap in meaning, a meta-option cannot be judged, the stem reveals the answer, or the original question is unclear |
| IASD | 100 | 11 | 89 | Nested option labels (3); an “NA” meta-option (3); still answerable or ambiguous after replacement (3); a synonymous correct answer (1); confused concepts in the original question (1) |
| IVQD | 100 | 57 | 43 | Insufficient dependence on the image or an ambiguous question (55); the original answer is not unique on review of the image (2) |
| Branch | Standard [95% interval] (pp) | UPD [95% interval] (pp) | Lost (%) | Circular Dual (%) | Invalid |
|---|---|---|---|---|---|
| B1 | reference | reference | 0 (0.00) | 10.08 | 0 |
| ReFrain | 0.00 [0.00, 0.00] | 20.10 [16.11, 24.64] | 0 (0.00) | 23.93 | — |
| ReFrain ( ) | 0.00 [0.00, 0.00] | 44.36 [39.30, 50.00] | 0 (0.00) | 44.94 | — |
| CLIP-UP EmbLoRA | 0.66 [ 0.44, 1.87] | 11.64 [8.30, 15.17] | 7 (0.77) | 20.05 | 1 |
| Plain abstention SFT | 0.11 [ 0.55, 0.88] | 49.26 [44.00, 54.63] | 4 (0.44) | 52.82 | 0 |
| Comparison | Standard (pp) | UPD (pp) | Reduction in original-correct loss (pp) |
|---|---|---|---|
| ReFrain CLIP-UP EmbLoRA | 0.66 [ 1.87, 0.44] | 8.46 [3.58, 13.73] | 0.77 [0.22, 1.43] |
| ReFrain plain abstention SFT | 0.11 [ 0.88, 0.55] | 29.17 [ 34.10, 24.26] | 0.44 [0.11, 0.88] |
| ReFrain ( ) CLIP-UP EmbLoRA | 0.66 [ 1.87, 0.44] | 32.72 [27.13, 38.53] | 0.77 [0.22, 1.43] |
| ReFrain ( ) plain abstention SFT | 0.11 [ 0.88, 0.55] | 4.90 [ 8.49, 1.34] | 0.44 [0.11, 0.88] |
| Task | Branch | Standard (%) | UPD (%) | Lost / corrected |
|---|---|---|---|---|
| AAD | B1 | 92.19 | 11.11 | 0 / 0 |
| ReFrain | 92.19 | 28.47 | 0 / 0 | |
| ReFrain ( ) | 92.19 | 60.07 | 0 / 0 | |
| CLIP-UP EmbLoRA | 94.01 | 8.68 | 1 / 8 | |
| Plain abstention SFT | 92.97 | 63.19 | 1 / 4 | |
| IASD | B1 | 95.79 | 29.49 | 0 / 0 |