cs.CVOct 17, 2026
SaveInvestigating Length Bias and Robustness in LVLMs for Multiple-Choice Question Answering
Organizations: Department of Computer Science, Virginia Tech
Abstract
Large Vision-Language Models (LVLMs) demonstrate strong performance on multiple-choice question answering (MCQA), yet their reliability under biased conditions remains underexplored. We investigate length bias, the systematic preference for longer answer options, through controlled benchmarks spanning fine-grained visual domains. LVLMs consistently favor longer options even when visually inconsistent, with accuracy dropping 2--38% when correct answers are shorter and increasing 1--70% when longer. This bias persists across model scales. Through layer-wise mechanistic analysis, we reveal that length bias emerges from late-layer suppression of visual evidence in favor of language-side length priors. Blurry-image experiments show that bias intensifies when visual evidence weakens, while abstention tests reveal that LVLMs struggle to refuse incorrect predictions. We propose and evaluate multiple inference-time mitigation strategies; while these provide partial improvements, none achieves deployable bias mitigation without relying on oracle information unavailable at inference time. These results underscore that inference-time interventions are insufficient, pointing to the need for architectural solutions and bias-aware training.
Figures & tables
Figure 1: Overview of our framework for investigating length bias in LVLMs. We create length-controlled MCQA variants to systematically expose length bias, conduct layer-wise analysis revealing late-layer suppression of visual evidence, perform diagnostic experiments using abstention and blurry images to localize bias sources, and evaluate mitigation strategies like ELED, caption-guided MCQA and Yes/No QA.
Figure 2: Accuracy (%) of LVLMs across five datasets under different option-length conditions , with and without class names. The vertical line at "Equal Long" indicates baseline performance for each LVLM-dataset pair.
| Dataset | Model | Equal Long | Equal Short | Shorter Correct | Longer Correct |
| CUB (W/o) | GPT-4o | 71.17 | 62.83 | 45.17 | 92.83 |
| o4-mini | 71.67 | 66.00 | 44.33 | 91.00 | |
| C-Sonnet-4.5 | 65.33 | 54.50 | 54.50 | 80.00 | |
| Dogs (W/o) | GPT-4o | 86.11 | 75.56 | 55.83 | 97.78 |
| o4-mini | 86.11 | 73.61 | 46.94 | 96.67 | |
| C-Sonnet-4.5 | 68.06 | 69.17 | 57.50 | 86.94 |
Table 1: Accuracy (%) of advanced proprietary LVLMs on CUB and Dogs datasets with (W.) and without (W/o) class names in the answer options.
Figure 3: Layer-wise preference for long distractors in the Shorter Correct setting using Qwen2.5-VL-7B-Instruct . Mean preference across 200 CUB examples via logit-lens decoding of hidden states (95% CI shaded). Early layers show near-zero systematic preference for long options; mid layers exhibit a positive shift, indicating emergence of length bias in internal representations prior to decoding. Dashed line: stable inversion layer ( ), the first layer at which the mean exceeds zero for three consecutive layers. Dotted line: median commitment layer ( ), beyond which the predicted answer remains unchanged.
Figure 4: Vision–language conflict resolution in late layers. Layer-wise probabilities for the short correct answer (green) versus average probability of each long distractor (red), both starting at unbiased baseline 0.25 (dotted line). (a) In length-biased errors (N=144, 72%), late layers (20–28, shaded) amplify long distractors to 0.31 while suppressing the correct answer to 0.10—each long option becomes more likely than correct. (b) When predictions align with ground truth (visual wins) (N=56, 28%), the correct answer amplifies to 0.73 while each long distractor suppresses to 0.06 ( preference ratio). Shaded regions: 95% confidence intervals. Despite stronger amplification capacity for correct answers (+0.48 vs +0.06), the model utilizes this in only 28% of cases.
| Dataset | Setting | Model | Equal Long | Equal Short | Shorter Correct | Longer Correct |
| CUB (W/o) | w/o NoA | Qwen2.5-VL-7B | 35.70 | 34.80 | 11.70 | 83.70 |
| InternVL_2.5-8B | 32.10 | 31.90 | 21.90 | 74.30 | ||
| NoA w/C | Qwen2.5-VL-7B | 25.20 | 18.60 | 11.60 | 66.40 | |
| InternVL_2.5-8B | 2.30 | 4.40 | 1.80 | 10.80 | ||
| Dogs (W/o) | w/o NoA | Qwen2.5-VL-7B | 34.83 | 36.83 | 14.50 | 82.67 |
| InternVL_2.5-8B | 32.50 | 40.00 | 26.17 | 76.67 |
Table 2: Accuracy (%) of LVLMs on blurry image evaluation across CUB and Dogs datasets with (W.) and without (W/o) class names in options. "w/o NoA" denotes standard MCQ format without "None of the Above" options. "NoA w/C" includes "E. None of the above" while the correct answer remains present among A–D, testing over-abstention. For each setting, bold marks the lowest accuracy between the two models.
| Dataset | Method | Shorter | Longer | Avg. |
| Correct | Correct | Gain | ||
| CUB (W/o) | Baseline | 26.6 | 93.4 | – |
| PMI | 46.5 | 74.5 | 0.5 | |
| Len. Norm ( ) | 41.9 | 85.8 | 3.85 | |
| Len. Norm ( ) | 30.5 | 92.2 | 1.35 | |
| Dogs (W/o) | Baseline | 33.2 | 90.8 | – |
Table 3: Comparison of output-level calibration baselines on Qwen2.5-VL-7B across CUB and Dogs, with (W.) and without (W/o) class names. Avg. Gain is the mean change over the Baseline across the Shorter Correct and Longer Correct splits. Both PMI and length normalization improve Shorter Correct at the cost of degrading Longer Correct, yielding small or negative average gains.
| Dataset | Model | Shorter Correct | Longer Correct | |
| CUB (W/o) | Qwen2.5-VL-7B | 48.40 (+21.80) | 95.00 (+1.60) | 46.60 (-20.20) |
| InternVL_2.5-8B | 52.10 (+14.30) | 90.40 (+2.10) | 38.30 (-12.20) | |
| Dogs (W/o) | Qwen2.5-VL-7B | 46.49 (+13.32) | 91.47 (+0.64) | 44.98 (-12.68) |
| InternVL_2.5-8B | 61.17 (+23.00) | 81.67 (+2.00) | 20.50 (-21.00) | |
| CUB (W.) | Qwen2.5-VL-7B | 61.50 (+10.80) | 94.70 (+0.80) | 33.20 (-10.00) |
| InternVL_2.5-8B | 52.20 (+13.00) | 90.40 (+2.10) | 38.20 (-10.90) |
Table 4: Split-aware ELED performance for length bias mitigation, using oracle split information (Algorithm 1 ; see Section 9.2 for details). Values in parentheses show changes over baseline.
Figure 5: Accuracy (%) of LVLMs on "None of the Above" (NoA) evaluation settings across CUB datasets, reported both with and without class names. "w/o NoA" denotes the original MCQ format without a NoA option. "NoA w/C" appends a fifth option, "E. None of the above", while keeping the correct answer present among A–D (over-abstention test). "NoA wo/C" removes the correct answer and uses "D. None of the above." as the only valid choice (forced-abstention test).
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Example of normal and blurred image of German Shepherd.
Figure 7: Example multiple-choice questions from the Dogs dataset illustrating four length-controlled variants: Equal Long , Equal Short , Shorter Correct , and Longer Correct . The correct answer (D: German Shepherd) remains constant while option lengths vary systematically to isolate the effect of textual length on model predictions.
Figure 8: Effect of visual degradation on length-bias conflict resolution. Layer-wise probabilities for the short correct answer (green) and the average long distractor (red) under clear images (left) and Gaussian blur ( , right), evaluated on the same set of length-biased error cases. Dashed lines mark the median override layer under each condition. Clear images exhibit greater dispersion in probability trajectories due to heterogeneity in visual evidence strength across examples, while blurring homogenizes visual inputs and produces earlier, more uniform dominance of length bias. Shaded regions denote 95% confidence intervals.
Figure 9: Text-to-vision attention correlation across transformer layers (N=200 examples). We measure the correlation between option length and visual attention at each layer. Late layers (20–28, shaded region) show statistically significant positive correlation (mean , ), indicating that longer options query vision more than shorter options. Despite this increased visual grounding, long incorrect options are preferentially selected, demonstrating that length bias operates through override of visual evidence in the language decoder rather than insufficient visual information retrieval.
Figure 10: Prompt template used for caption-guided MCQA.
| Dataset | Captioner | Model | Equal Long | Equal Short | Shorter Correct | Longer Correct |
| CUB | Qwen2.5-VL-7B | Qwen2.5-VL-7B | 59.30 | 52.10 | 41.30 | 70.50 |
| Qwen3-VL-30B | Qwen2.5-VL-7B | 66.25 | 55.50 | 44.25 | 70.00 | |
| GPT-4o | Qwen2.5-VL-7B | 67.00 | 58.00 | 48.50 | 70.75 | |
| InternVL-2.5-8B | InternVL-2.5-8B | 64.50 | 55.10 | 41.80 | 74.40 | |
| Qwen3-VL-30B | InternVL-2.5-8B | 66.75 | 57.50 | 46.00 | 78.25 | |
| GPT-4o | InternVL-2.5-8B | 65.25 | 57.25 | 43.50 | 76.75 |
Table 5: Accuracy (%) of LVLMs on Caption-Guided MCQA across CUB and Dogs without class names in the answer options.
| Dataset | Model | Equal Long | Equal Short | Shorter Correct | Longer Correct |
| CUB (W/o) | Qwen2.5-VL-7B | 58.40 | 51.30 | 62.80 | 45.70 |
| InternVL_2.5-8B | 59.00 | 53.50 | 67.10 | 44.00 | |
| Dogs (W/o) | Qwen2.5-VL-7B | 57.17 | 53.17 | 53.00 | 61.33 |
| InternVL_2.5-8B | 52.00 | 52.00 | 73.33 | 32.50 | |
| CUB (W.) | Qwen2.5-VL-7B | 71.20 | 68.60 | 69.10 | 70.40 |
| InternVL_2.5-8B | 63.60 | 62.00 | 53.20 | 70.10 |
Table 6: Accuracy (%) of LVLMs on " Yes/No " evaluation across CUB and Dogs, reported with (W.) and without (W/o) class names in the answer options.
| Dataset | Model | Equal Long | Equal Short | Shorter Correct | Longer Correct |
| CUB (W/o) | Qwen2.5-VL-7B | 52.80 | 52.40 | 31.40 | 81.80 |
| InternVL_2.5-8B | 56.80 | 49.80 | 33.10 | 82.90 | |
| Dogs (W/o) | Qwen2.5-VL-7B | 54.17 | 53.83 | 41.50 | 73.50 |
| InternVL_2.5-8B | 54.00 | 56.33 | 32.67 | 76.67 | |
| CUB (W.) | Qwen2.5-VL-7B | 66.70 | 63.40 | 50.00 | 84.50 |
| InternVL_2.5-8B | 57.90 | 56.00 | 33.40 | 88.10 |
Table 7: Accuracy (%) of LVLMs on " selection by rejection " across CUB and Dogs, with (W.) and without (W/o) class names in the answer options.
| Class Name | Dataset | Model | Equal Long | Equal Short | Shorter Correct | Longer Correct | |
| Without | Aircraft | LLaVA-v1.5-7B | 32.86 | 34.00 (+1.14) | 18.29 (-14.57) | 84.57 (+51.71) | 66.28 |
| LLaVA-v1.5-13B | 39.71 | 35.71 (-4.00) | 18.86 (-20.85) | 87.43 (+47.72) | 68.57 | ||
| Qwen2.5-VL-7B | 54.86 | 43.71 (-11.15) | 21.43 (-33.43) | 84.00 (+29.14) | 62.57 | ||
| InternVL_2.5-8B | 39.71 | 36.29 (-3.42) | 32.29 (-7.42) | 57.43 (+17.72) | 25.14 | ||
| Cars | LLaVA-v1.5-7B | 26.73 | 32.65 (+5.92) | 12.35 (-14.38) | 94.29 (+67.56) | 81.94 | |
| LLaVA-v1.5-13B | 40.71 | 44.08 (+3.37) | 16.94 (-23.77) | 93.16 (+52.45) | 76.22 |
Table 8: Accuracy (%) of LVLMs across five datasets under different option-length conditions, with and without class names in answer choices. Results show LVLM performance on "Equal Long," "Equal Short," "Shorter Correct," and "Longer Correct" variants. Values in parentheses indicate differences from the "Equal Long" baseline for each model-dataset pair. (Longer Correct - Shorter Correct) quantifies length bias magnitude. The best result for each setting and dataset is highlighted in bold.
Figure 11: Accuracy (%) of LVLMs on "None of the Above" (NoA) evaluation settings across Dogs datasets, reported both with and without class names in the answer options. "w/o NoA" denotes the original MCQ format without a NoA option. "NoA w/C" appends a fifth option, "E. None of the above", while keeping the correct answer present among A–D (over-abstention test). "NoA wo/C" removes the correct answer and uses "D. None of the above." as the only valid choice (forced-abstention test).
| Class Name | Dataset | Setting | Model | Equal Long | Equal Short | Shorter Correct | Longer Correct |
| Without | CUB | w/o NoA | Qwen2.5-VL-7B | 64.90 | 61.00 | 26.60 | 93.40 |
| InternVL_2.5-8B | 62.50 | 57.10 | 37.80 | 88.30 | |||
| NoA w/C | Qwen2.5-VL-7B | 64.10 | 60.20 | 28.10 | 93.30 | ||
| InternVL_2.5-8B | 59.30 | 54.40 | 34.70 | 87.40 | |||
| NoA wo/C | Qwen2.5-VL-7B | 3.20 | 5.80 | 3.20 | 5.40 | ||
| InternVL_2.5-8B | 17.70 | 21.00 | 18.00 | 20.00 |
Table 9: Accuracy (%) of LVLMs on "None of the Above" (NoA) evaluation settings across CUB and Dogs datasets, reported both with and without class names in the answer options. "w/o NoA" denotes the original MCQ format without a NoA option. "NoA w/C" appends a fifth option, "E. None of the above", while keeping the correct answer present among A–D (over-abstention test). "NoA wo/C" removes the correct answer and uses "D. None of the above." as the only valid choice (forced-abstention test). In the "NoA wo/C" setting, removing the correct answer from "Shorter Correct" leaves only long distractors, making its option pool structurally similar to "Equal Long"; likewise, removing the correct answer from "Longer Correct" leaves only short distractors, similar to "Equal Short". Accordingly, LVLM performance under "NoA wo/C" for these two conditions closely aligns with, though does not exactly match, "Equal Long" and "Equal Short", respectively.
| Dataset | A | B | C | D |
| Cars | 245 | 245 | 240 | 250 |
| CUB | 260 | 215 | 250 | 275 |
| Dogs | 170 | 145 | 160 | 125 |
| Food | 140 | 125 | 100 | 140 |
Table 10: Distribution of correct answers across positions A, B, C, and D for each dataset. Distributions are inherited from base benchmarks and remain unchanged across all length-controlled variants.
| Model | Dataset | Class name only | Shorter Correct | Longer Correct |
| InternVL-2.5-8B | Dogs | 54.17 | 45.33 | 79.17 |
| InternVL-2.5-8B | CUB | 47.10 | 48.60 | 89.60 |
| Qwen2.5-VL-7B | CUB | 67.00 | 53.30 | 93.20 |
| Qwen2.5-VL-7B | Dogs | 75.83 | 60.17 | 93.33 |
Table 11: Accuracy (%) under class-name-only ablations. All conditions use identical answer positions, varying only the length of the correct option versus distractors.
More in this topic
VLM Robustness
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.
See all 443 papers