Gaze patterns predict preference and confidence in pairwise AI image evaluation

Nikolas Papadopoulos; Shreenithi Navaneethan; Sheng Bai; Ankur Samanta; Paul Sajda

cs.HCMar 25, 2026

Gaze patterns predict preference and confidence in pairwise AI image evaluation

Authors: Nikolas Papadopoulos, Shreenithi Navaneethan, Sheng Bai, Ankur Samanta, Paul Sajda

Organizations: Columbia University, New York, NY

Abstract

Preference learning methods, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on pairwise human judgments, yet little is known about the cognitive processes underlying these judgments. We investigate whether eye-tracking can reveal preference formation during pairwise AI-generated image evaluation. Thirty participants completed 1,800 trials while their gaze was recorded. We replicated the gaze cascade effect, with gaze shifting toward chosen images approximately one second before the decision. Cascade dynamics were consistent across confidence levels. Gaze features predicted binary choice (68% accuracy), with chosen images receiving more dwell time, fixations, and revisits. Gaze transitions distinguished high-confidence from uncertain decisions (66% accuracy), with low-confidence trials showing more image switches per second. These results show that gaze patterns predict both choice and confidence in pairwise image evaluations, suggesting that eye-tracking provides implicit signals relevant to the quality of preference annotations.

Figures & tables

Category	Features
Per-AOI metrics	Dwell time, fixation count, mean fixation
	duration, revisit count
Image comparisons	Dwell time ratio, fixation ratio, absolute
	differences between images
Temporal markers	First/last fixation location, time to
	first image fixation

Table 1: Gaze feature categories.

Figure 2: Gaze cascade effect in AI image evaluation. Likelihood of fixating on the eventually chosen image as a function of time before decision (key press). The blue line shows grand average across participants (N=30), with individual participants shown in gray lines.

Figure 3: Gaze cascade dynamics by confidence level.

Metric	Chosen	Unchosen	$p$	Cohen $d$
Dwell time (s)	2.30	2.13	$<.001$	0.99
Fixations	10.7	9.7	$<.001$	1.16
Revisits	3.3	2.9	$<.001$	1.77

Table 2: Gaze metrics between chosen and unchosen images. Paired t-tests on participant-level means (N=30)

Figure 4: Permutation importance for preference prediction (top 5 features). Error bars show SD across 5 folds.

Features	Accuracy	ROC-AUC
Image transitions	$66.1\pm 5.1$ %	0.70
All gaze features	$62.4\pm 4.3$ %	0.67
Response time	$58.1\pm 1.2$ %	0.66
Baseline (majority)	55.0%	—

Table 3: Confidence prediction performance. Image transitions refer to gaze shifts between the two images. All models use logistic regression with group 5-fold cross-validation (

N=1{,}740

trials, 29 participants).

Human Preference Evaluation

Human preference evaluation measures how people compare, rate, and explain AI outputs to assess and guide model behavior.

See all 56 papers

Aligning to Illusions: Choice Blindness in Human and AI Feedback
Mar 9, 2026Wenbin WuRL from Human Feedback Bias in LLM-as-a-Judge
Cambridge Judge Business School, University of Cambridge, UK
Attention Limited Reward Learning
Jul 6, 2026Wenqian XingPairwise Preference Learning Reward Modeling
Management Science and Engineering Department, Stanford University
Same Words, Different Judgments: How Preferences Vary Across Modalities
Feb 26, 2026Aaron Broukhim, Nadir Weibel, Eshin JollyHuman Preference Evaluation Inter-Rater Reliability

Mar 9, 2026J/K move · Enter open · S save

Aligning to Illusions: Choice Blindness in Human and AI Feedback

Wenbin Wu

Cambridge Judge Business School, University of Cambridge, UK

RL from Human Feedback Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Reinforcement Learning from Human Feedback (RLHF) assumes annotator preferences reflect stable internal states. We challenge this through three experiments spanning the preference pipeline. In a human choice blindness study, 91% of surreptitiously swapped preferences go undetected, extending choice blindness to third-person evaluative comparison of unfamiliar text. Testing fifteen LLM judges as potential replacements, we find detection relies on shallow text matching rather than genuine self-monitoring: removing prior reasoning from context causes blindness to surge from near-zero to over 50%, while explicit social pressure induces near-universal compliance. In a dose-response experiment across two architectures from 86M to 2B parameters, one-sixth to one-third of labels must be corrupted before the reward signal halves, yet standard pairwise accuracy remains virtually unchanged. A Best-of-N evaluation confirms this translates to downstream policy degradation: at 50% corruption, reward-guided selection produces no improvement over random sampling, while the proxy model reports monotonically increasing scores. Together, these results reveal a preference construction problem: the signal entering RLHF is shaped by elicitation context in ways that neither human metacognition, LLM self-monitoring, nor standard evaluation metrics can detect.

Abstract

Human Preference Evaluation

Explore similar work

Aligning to Illusions: Choice Blindness in Human and AI Feedback

Attention Limited Reward Learning

Same Words, Different Judgments: How Preferences Vary Across Modalities