cs.CVSep 5, 2026

GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them

Authors: Deep Dessai

Organizations: The University of Texas at Austin · Department of Computer Science The University of Texas at Austin

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As vision-language models (VLMs) become increasingly capable and are deployed in consequential real-world settings, they must evaluate evidence independently rather than defer uncritically to human authority. We introduce GradeTrap, a controlled evaluation that places two social cues in direct conflict: a student answer, which should attract sycophantic agreement, and a conflicting answer attributed to a peer, teacher, or official answer key, which should attract authority-based deference. Models produce free-form answers while being explicitly instructed to solve independently and ignore all student answers, feedback, and grading marks. We test the models on 60 synthetic real-world trade-off scenarios. Five neutral trials establish a stable model-relative preference, followed by three repetitions of six experimental cues including controls. On the 45-item common intersection across Gemini 3.5 Flash-Lite, GPT-5.6 Luna, and Claude Haiku 4.5, a generic second-answer control yields 5.4% conflicting-answer selection. Relative to that control, pooled within-item changes show no reliable peer-review effect, a 6.9-point teacher-review effect, and a 19.5-point official-key effect. In contrast, a displayed conflicting student answer alone compared to a displayed student reference answer alone only raises selection from 2.2% to 5.2%. Official-key provenance therefore redirects judgements more than a student answer or the generic second-answer control, despite an explicit ignore instruction and an opposing student answer given along with the official key. Effects vary in magnitude across the three models.

Figures & tables

Figure 1: Example neutral rendering used to establish the model-relative baseline. The question is shown without a filled student answer or a second-answer card. Each model receives five independently generated neutral trials for every item.
Figure 2: Exact renderings for one item. Panels (a)–(b) contain no second-answer card. Panels (c)–(f) are identical outside the card heading: the student answer, conflicting answer, position, and styling are fixed. In (c)–(f), the student answer field shows the reference answer, and the conflicting answer is shown under an alternative answer, peer-review, teacher-review, or official answer key heading.
 Full stable setCommon 45-item intersection
ConditionGeminiGPTClaudeGeminiGPTClaudePooled
Stable nn53565245454545/model
Student reference5.71.87.10.72.23.72.2
Student-only alt.7.52.415.44.43.08.15.2
Alternative answer8.83.016.74.43.78.15.4
Peer review6.34.221.22.25.214.17.2
Table 1: Conservative conflicting answer selection percentages across three repetitions per stable model–item pair. Columns under Full stable set use each model’s own eligible items. Columns under Common 45-item intersection restrict every model to the same 45 items; Pooled is the mean of those 45×3×3=40545\times 3\times 3=405 trials per condition and is not an average of the full-set model columns.
Figure 3: Conflicting-answer selection on each model’s full stable set (53 items for Gemini, 56 for GPT, and 52 for Claude). Points show the percentage of trials in which the model selected the conflicting answer under the conservative scoring policy; error bars show 95% item-cluster bootstrap intervals based on 5,000 resamples. Lines join conditions within each model, and points are offset horizontally to keep the error bars visually distinguishable. The shaded band marks the two student-only conditions, which contain no second-answer card. The four conditions to its right are identical outside the card heading, and the dashed line separates the Alternative answer control from the three provenance conditions. The official-key condition has the highest conflicting-answer selection rate for all three models, with the largest increases appearing for Gemini 3.5 Flash-Lite and Claude Haiku 4.5.
More in this topic

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Explore similar work

CardsList
Oct 1, 2026

When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts

Vision-language models (VLMs) increasingly act as judges that pick the best of several generated images, so their choices decide what users see. Such judges are usually validated by score agreement with human ratings, not by the images they return. We audit VLM judges as decision-makers: on 300 culturally situated prompts, we compare the returned image with human ratings the judge never sees and with random choice from the same candidates, and repeat every decision with the candidates reordered. A 4B-parameter judge barely beats random and falls short of a CLIP similarity baseline. It picks the first image shown in 49% of calls (chance: 28%), and reordering changes its choice on 60% of prompts. For this judge, agreement across orders is informative: decisions that survive reordering are much better than random, whereas agreement with a weaker second judge keeps the wrong ones. An 8B judge shows almost no position bias and outperforms CLIP, yet for it the same filter mostly discards good decisions. Agreement helps only when it targets the judge's failure mode, so filters must be re-audited whenever the judge changes. The 4B judge's slight rise in stereotype ratings is no longer detectable after aggregating across orders or with the larger judge.

Huichan Seo

Independent Researcher

VLM EvaluationPosition Bias in VLMsLLM-as-a-JudgeCultural Bias in VLMsVision-Language Model-as-a-Judge
AskPDF
Sep 24, 2026

Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

One long-form exam in a large course costs hundreds of grader-hours, and qualified graders are scarce; LLM graders are a tempting alternative. To show its pitfalls we grade a practical Computer Vision exam (570570 dual-graded students) under 171171 configurations spanning closed and open-weights models; the best reaches mean absolute error 1.64/351.64/35, below the 2.61/352.61/35 two human graders achieve against each other. The catch is the prompt: a short ''strict grader'' preamble drives 1414 of 1717 open-weights models out of the graded band (MAE≥8\text{MAE} \ge 8), three stopping grading altogether. The damage traces to the preamble's two credit-withholding sentences, not to tone or model scale; one of them, ''never give partial credit'', alone makes two of three probed models stop grading. The closed flagships of three vendors shift calibration under it but stay in the band. In 162162 further configurations on a second, independent Machine Learning exam from another course (1,0381{,}038 dual-graded students), the preamble worsens ten models, moving three out of the band into collapse and one into refusal, yet improves seven whose neutral prompts over-mark: the vulnerability replicates, but its direction is exam-specific. Light LoRA fine-tuning repairs it: one adapter on the two exams' pooled ∼3,900\sim 3{,}900 graded examples brings five small open models to parity or better with a human grader in agreement with the grader pair, and sensitivity to the three harsh personas nearly vanishes (≤0.32\le 0.32 MAE). We release the anonymised dataset, full ablation grid, and grading, fine-tuning and analysis pipelines.

Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh +2

KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · Visual Artificial Intelligence Laboratory Oxford Brookes University · KAUST Academy & CEMSE, KAUST +1

Prompt SensitivityLLM-as-a-JudgeLLM AuditingLanguage Model CalibrationAutomated Grading
AskPDF
Mar 19, 2026

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

When VLMs answer correctly, do they genuinely rely on visual information? We introduce a Tri-Layer Diagnostic Framework with three per-sample metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score, which disentangle perception, dependency, and alignment failures. Across 9 VLMs and 9,000 model-sample pairs under counterfactual blind, noise, and conflict interventions, 72.9% of samples exhibit Visual Sycophancy, a Split Beliefs pattern in which internal evidence is preserved yet a hallucinated answer is decoded, while zero samples show Robust Refusal, indicating that current alignment training has eliminated refusal as a decoding outcome. Scaling within the Qwen-VL family, both within- and across-generation, monotonically reduces Language Shortcuts but amplifies Visual Sycophancy, showing that scale and newer post-training alone cannot resolve the grounding problem. Diagnostic scores further enable a training-free selective-prediction strategy yielding up to +9.5 percentage points accuracy at 50% coverage.

Rui Hong, Shuxue Quan

George Mason University, Fairfax, VA, USA · Independent Researcher

Selective Prediction for VLMsSycophancy in VLMsVision-Language ModelsVision-Language GroundingVisual Evidence Sensitivity in VLMsHallucination Detection in VLMs+1
AskPDF