Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation
Authors: Lingfeng Huang, Huizhong Guo, Tianjun Wei, Yingpeng Du, Zhu Sun
Organizations: Singapore University of Technology and Design, Singapore · Zhejiang University, Hangzhou, China · Nanyang Technological University, Singapore
Large language model (LLM) agents are increasingly deployed as scalable user simulators for recommender system evaluation. Yet existing simulators perceive recommendations through text or structured metadata rather than the visual interfaces real users browse-a critical gap, since attention over recommendation layouts is both visually driven and highly personalized. We investigate whether aligning a vision-language model's (VLM's) visual attention with user-specific gaze patterns can improve simulation fidelity. Analysis of a real-world eye-tracking dataset collected in a carousel-based recommendation setting reveals that users exhibit stable individual gaze patterns strongly predictive of click behavior. Building on this finding, we propose Fixation-Aligned Tuning for user Emulation (FixATE). Our approach first probes the VLM's internal visual attention via interpretability operators to obtain a slot-level relevance distribution comparable with human fixation, and then learns personalized soft prompts to steer the model's attention toward each user's characteristic fixation pattern. Experiments across three interpretability-based probing operators and two architecturally distinct VLM backbones demonstrate consistent improvements in both attention alignment and click prediction accuracy. These results suggest that making the model "see like the user" is a viable path toward simulators that more faithfully reproduce how users perceive and act in recommendation interfaces.
Figures & tables
Figure 1. The difference of the perception of recommendations between real users and LLM-based user simulators. This figure shows the difference of the perception of recommendations between real users and LLM-based user simulators. The right panel shows the real users' attention distribution when browsing the movie poster carousel. The left panel shows the LLM-based user simulator's perception: movie posters are converted into text descriptions and fed into the LLM user simulators.
Figure 2. Empirical analyses on the RecGaze dataset. Two subfigures. (a) Four heatmaps showing click position distributions on a 3x5 grid for Ground Truth, Text w/o Pos., Image, and Text w/ Pos. conditions. The Image condition best matches the ground truth F-shaped pattern. (b) A grouped bar chart showing that the clicked movie ranks first in cumulative fixation duration 71% of the time and first in cursor duration 67% of the time.
Figure 3. Overview of the Fixation-aligned Tuning for User Emulation ( FixATE ) framework.
Figure 4. Multimodal prompt template used in FixATE .
Model
Method
Setting
Attention alignment metrics
Prediction-level metrics
CSH@1
CSH@3
CSH@5
TGO@1
TGO@3
TGO@5
Cosine
JS ↓
KL ↓
Accuracy
LogLoss ↓
AUC
Qwen3-VL- 4B-Instruct
GLIMPSE
Backbone
0.1447
0.3947
0.5789
0.1447
0.3684
0.5211
0.5270
0.2552
1.1033
0.1053
2.7860
0.5893
FixATE
0.2368
0.4211
0.6184
0.1974
0.4035
0.5579
0.5559
0.2354
0.9986
0.1974
2.6042
0.6607
Improv.
63.64%
6.67%
6.82%
36.36%
9.52%
7.07%
5.49%
7.76%
9.49%
87.50%
6.53%
12.12%
AttnLRP
Backbone
0.0395
0.3026
0.4474
0.0658
0.3158
0.4974
0.4828
0.2704
1.2187
0.1053
2.7860
0.5893
FixATE
0.1316
0.4605
0.5921
0.1184
0.3860
0.5289
0.5350
0.2559
1.0954
0.2105
2.5759
0.6701
Table 1. Evaluation results across probing operators and VLM backbones. Backbone denotes the baseline without personalized soft prompts, while FixATE uses the learned user-specific prompt Pu . Improv. denotes the relative improvement (%).
Model
Setting
Attention alignment metrics
Prediction-level metrics
CSH@1
CSH@3
CSH@5
TGO@1
TGO@3
TGO@5
Cosine
JS ↓
KL ↓
Accuracy
LogLoss ↓
AUC
Qwen3-VL- 4B-Instruct
Backbone
0.0395
0.3026
0.4474
0.0658
0.3158
0.4974
0.4828
0.2704
1.2187
0.1053
2.7860
0.5893
Rand. SP
0.0658
0.1711
0.3684
0.0526
0.1263
0.4263
0.4250
0.3055
1.3509
0.0789
2.8992
0.5620
zu=1
0.0263
0.3289
0.5263
0.0789
0.3070
0.4816
0.4964
0.2727
1.1892
0.0921
2.8945
0.5883
w/o ωn
0.0789
0.3158
0.4737
0.0921
0.3246
0.5026
0.4890
0.2690
1.1926
0.1184
2.7594
0.6118
w/o LAttn
0.0395
0.2500
0.4605
0.0789
0.2675
0.4211
0.4336
0.2983
1.3272
0.0921
2.8066
0.5780
Table 2. Ablation study results using AttnLRP as the probing operator. Best values per model are highlighted with a light red background and boldface; values better than Backbone are highlighted with a light blue background.
Figure 5. Sensitivity analysis of key hyperparameters on two backbones with AttnLRP . Each curve reports a representative metric as the hyperparameter varies.
Figure 6. Case study of slot-level attention on Qwen3-VL . (a) Original backbone attention; (b) Human gaze; (c) Attention from FixATE . Posters are blurred for copyright reasons.
Model
Setting
CSH@1
TGO@1
KL ↓
Accuracy
LogLoss ↓
AUC
Qwen3-VL- 4B-Instruct
Backbone
0.2727
0.3939
0.5797
0.2727
1.4450
0.6760
FixATE
0.3636
0.5758
0.5547
0.3636
1.3516
0.6851
Improv.
33.33%
46.15%
4.31%
33.33%
6.46%
1.35%
InternVL3.5- 4B-Instruct
Backbone
0.2121
0.4242
0.8482
0.2121
1.8929
0.5048
FixATE
0.2727
0.4848
0.7355
0.2727
1.7947
0.5547
Improv.
28.57%
14.29%
13.29%
28.57%
5.19%
9.89%
Table 3. Evaluation of AttnLRP on the AdSERP dataset.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Method
Setting
Attention alignment metrics
Prediction-level metrics
CSH@1
TGO@1
KL ↓
AUC
LogLoss ↓
Accuracy
Qwen3-VL- 4B-Instruct
GLIMPSE
Backbone
0.1447
0.1447
1.1033
0.5893
2.7860
0.1053
FixATE
0.2368 ± 0.053 ∗
0.1974 ± 0.029 ∗
0.9986 ± 0.060 ∗
0.6607 ± 0.008 ∗∗
2.6042 ± 0.082 ∗∗
0.1974 ± 0.045 ∗∗
AttnLRP
Backbone
0.0395
0.0658
1.2187
0.5893
2.7860
0.1053
FixATE
0.1316 ± 0.038 ∗∗
0.1184 ± 0.029 ∗
1.0954 ± 0.056 ∗∗
0.6701 ± 0.009 ∗∗
2.5759 ± 0.081 ∗∗
0.2105 ± 0.047 ∗∗
Attention Rollout
Backbone
0.0921
0.0658
1.7114
0.5893
2.7860
0.1053
Appendix
Table 4. Extended evaluation on RecGaze with standard deviations and significance levels (cf. Table 1 in the main paper). ∗ p<0.05 , ∗∗ p<0.01 ( two-sided paired t -test, df=4 ). Values are reported as mean ± std. Backbone values are deterministic (frozen model, greedy decoding) and thus reported without variance. Bold indicates improvement over Backbone .
Model
Setting
Attention alignment metrics
Prediction-level metrics
CSH@1
TGO@1
KL ↓
AUC
LogLoss ↓
Accuracy
Qwen3-VL- 4B-Instruct
Backbone
0.2727
0.3939
0.5797
0.6760
1.4450
0.2727
FixATE
0.3636 ± 0.048 ∗
0.5758 ± 0.057 ∗∗
0.5547 ± 0.023
0.6851 ± 0.006 ∗
1.3516 ± 0.025 ∗∗
0.3636 ± 0.030 ∗∗
InternVL3.5- 4B-Instruct
Backbone
0.2121
0.4242
0.8482
0.5048
1.8929
0.2121
FixATE
0.2727 ± 0.043 ∗
0.4848 ± 0.048 ∗
0.7355 ± 0.035 ∗∗
0.5547 ± 0.006 ∗∗
1.7947 ± 0.040 ∗∗
0.2727 ± 0.037 ∗
Appendix
Table 5. Extended evaluation on AdSERP (AttnLRP) with standard deviations and significance levels (cf. Table 3 in the main paper). ∗ p<0.05 , ∗∗ p<0.01 ( two-sided paired t -test, df=4 ).
Aspect
Attn Rollout
AttnLRP
GLIMPSE
Gradient required
No
Yes
Yes
Head aggregation
Uniform mean
Uniform mean
Adaptive softmax
Per-layer normalization
Row-norm with 21I
Signed-sum
Row-norm after ReLU
Cross-layer propagation
Sequential product
Depth-weighted additive
Iterative multiplicative
Differentiable
No (post-hoc)
Yes
Yes
Relative cost
Lowest
Moderate
Highest
Appendix
Table 6. Comparison of the three probing operators.
Figure 7. Supplementary qualitative comparison of slot-level attention. Each row is one session (user × task); columns are frozen Qwen3-VL ( Backbone ), human fixation, and FixATE after personalized tuning. Posters are blurred where required for copyright.
As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full τ-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.
Xuhui Zhou, Weiwei Sun, Qianou Ma +8
Carnegie Mellon University, Language Technologies Institute
LLM-based user simulators are increasingly used to evaluate autonomous agents at scale, in place of costly human evaluations. Despite this promise, these simulators exhibit "assistant bias," a tendency to cooperate and pursue task goals. They rarely reproduce the frustration or disengagement that real users exhibit, compromising evaluation validity. Prior work outlines that this bias is baked in during model training, which role-playing prompts fail to override. We analyze this bias from model activations, extracting a user role vector by contrasting how the model represents user versus assistant perspectives on the same dialogue. We observe two findings: (i) the user direction is identifiable in activations, elicits user-like behaviors, and captures characteristics distinct from assistant traits; and (ii) although user-role activation associates with simulation realism and steering strengthens it, it can exaggerate user behaviors and override individual user profiles. Together, our findings provide a representation-level analysis of LLM user simulators, confirming that assistant bias is structurally identifiable and that user behavior can be directionally analyzed.
Daeheon Jeong, Yoonjoo Lee, Eugene Choi +2
1KAIST · University of Michigan · 3Seoul National University +2
LLM-based user simulators aim to bridge the offline-online gap in recommender evaluation by emulating users through injected traits, where preference attributes determine what a user engages with and a behavioral activity trait governs how long they browse. However, we show this intended trait independence collapses during simulation, causing two failures: (i) Trait Interference, where amplified activity distorts preference boundaries and forces interactions with mismatched items to sustain browsing, and (ii) Evaluation Invalidity, where satisfaction scores inflate with activity-driven page counts despite taste mismatches, biasing evaluation toward trait distributions rather than recommender performance. To resolve this, we propose PQA, a page-level quality anchoring method that guides simulators using a personalized anchor reflecting each user's intrinsic preference standard. By assessing whether a page meets this standard before further browsing, PQA enables proactive exits from low-quality pages, letting the activity trait retain its intended role of modulating browsing depth within preference-conforming pages. Experiments show PQA mitigates trait interference and improves the reliability of LLM-based simulator evaluation under activity shifts. Our code is available at https://github.com/chaehyun1/PQA