Leveraging Psychophysical Attentional Distribution for Gaze-Augmented Reward Modeling
Authors: Chaeho Lee, Junyup Kim, Wansoo Kim, Young Min Jung, Sang Ho Lee
Organizations: School of Digital Humanities and Computational Social Sciences, KAIST · Department of Brain & Cognitive Sciences, KAIST · Graduate School of Data Science, KAIST
Integrating human feedback into language models has gained increasing attention as a means to align model outputs with human preferences, with Reinforcement Learning from Human Feedback (RLHF) serving as a prominent framework for preference-based alignment. Recent studies have incorporated eye-tracking (ET) data as an additional supervisory signal for reward model training, grounding reward learning in human reading behavior. Building on this approach, we extend prior fixation-centric models by leveraging established findings from cognitive psychology on human letter recognition. Specifically, we model visual attention as a spatially graded distribution centered on the current gaze location and incorporate this representation into reward model training. Our results show that the reward model trained with the proposed gaze distribution achieves higher preference prediction accuracy than a baseline model. Moreover, some results from our experiments are qualitatively consistent with established findings on human attention during reading, suggesting that they potentially capture cognitively meaningful aspects of human attention allocation.
Figures & tables
Figure 1: A schematic illustration of visual attention allocation during each fixation while reading.
Figure 2: An overview of the GazeReward framework with our redistribution module. First, we obtain original TRT values for a given sentence and redistribute them using an asymmetric Gaussian kernel with learnable parameters. The TRT values are then transformed into eye-tracking embeddings through the ET feature projector.
Figure 3: Asymmetric Gaussian redistribution of total reading time (TRT). Example of redistributing per-token TRT values for the sentence “The quick brown fox jumps over the lazy dog.” (Top) Original TRT values predicted per token. (Middle) Each cell indicates how much of source token’s TRT is redistributed to neighboring tokens weighted by an asymmetric Gaussian kernel with σleft=1.0 and σright=4.0 . (Bottom) Redistributed TRT values obtained by summing each column of the contribution matrix.
OASST1
HelpSteer2
UltraFeedback
Input Length Cap (characters)
5,000
10,000
10,000
Total Data Samples
8,161
7,592
63,966
Data Loss
307 (3.76%)
241 (3.17%)
1,632 (2.55%)
Table 1: Data loss under the input length cap across the three datasets.
Llama-3-8B-Instruct
Llama-3-8B
Accuracy (%)
Diff. (%)
Accuracy (%)
Diff. (%)
GazeConcat
Baseline
68.38 ± 3.88
–
69.23 ± 4.92
–
Our Approach
74.73 ± 1.55
9.29
74.81 ± 1.32
8.06
GazeAdd
Baseline
70.30 ± 1.25
–
69.27 ± 0.81
–
Our Approach
71.89 ± 1.22
2.26
71.17 ± 1.21
2.74
Table 2: Mean reward modeling accuracy on OASST1 across six random seeds. Diff. denotes the relative improvement of our approach over the corresponding baseline (mean ± standard deviation).
Llama-3-8B-Instruct
Llama-3-8B
σleft
σright
σleft
σright
GazeConcat
1.62 ± 1.18
2.25 ± 1.71
0.80 ± 0.56
3.47 ± 2.13
GazeAdd
0.58 ± 0.21
1.99 ± 1.17
0.64 ± 0.23
0.80 ± 0.31
Table 3: Asymmetric Gaussian parameters from the best checkpoints on OASST1 across six random seeds (mean ± standard deviation).
Figure 4: An illustration of the human perceptual span and the proposed token-level gaze redistribution. (a) The asymmetric human perceptual span in left-to-right reading. (b) TRT values redistributed to the tokens adjacent to the token fox . The red dotted lines indicate the hypothetical boundaries of the human perceptual span mapped onto the tokenized sentence.
Dataset
Method
Accuracy (%)
Diff. (%)
σleft
σright
HelpSteer2
Baseline
59.48±3.66
–
–
–
Our Approach
62.64±1.68
5.31
3.18±2.34
1.85±2.01
UltraFeedback
Baseline
65.65±2.12
–
–
–
Our Approach
74.20±0.46
13.02
1.48±1.69
3.79±1.94
Table 4: Reward modeling performance on additional preference datasets. Results are reported as mean ± standard deviation across six random seeds. Diff. denotes the relative improvement of our approach over the corresponding baseline.
Method
Description
Accuracy (%)
Diff. (%)
Learned σ
Baseline
No redistribution; each predicted TRT value remains assigned to its original token
69.23±4.92
–
N/A
Uniform 1:4
TRT is uniformly redistributed over one token to the left and four tokens to the right
70.18±1.15
1.37
N/A
Fixed 1:4
Predetermined asymmetric Gaussian with a fixed 1:4 left-to-right spread
71.46±2.28
3.22
N/A
Symmetric
A single learnable σ parameter shared by the left and right sides
74.19±1.86
7.16
σ=3.28±0.84
Asymmetric
Independently learned left- and right-side σ parameters
74.81±1.32
8.06
σleft=0.80±0.56σright=3.47±2.13
Table 5: Reward modeling accuracy across five conditions: Pinpoint baseline, Uniform 1:4, Fixed 1:4, Symmetric, and Asymmetric. A description for each condition and its accuracy (mean ± standard deviation) across the six random seeds are provided. Diff. denotes the relative improvement over the pinpoint baseline. Where applicable, the learned values of the σ parameters are shown.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Redistribution Condition
Spearman Correlation ( ↑ )
JS Divergence ( ↓ )
No redistribution (Pinpoint)
0.4645
0.2756
Symmetric Gaussian
0.8958
0.1584
Learned asymmetric Gaussian (Ours)
0.9095
0.1473
Appendix
Table 6: Quantitative comparison between redistributed token-level TRT under different conditions and attention simulated by OB1-reader on the Provo Corpus under the 1:4 left-to-right attentional asymmetry ( asymleft=0.25 , asymright=1.0 ). Values represent the mean across six random seeds.
Department of Computer Science, University of Potsdam, Germany. · Department of Computational Linguistics, University of Zurich, Switzerland. · Department of Informatics, University of Zurich, Switzerland.