Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation

Lingfeng Huang; Huizhong Guo; Tianjun Wei; Yingpeng Du; Zhu Sun

cs.MMApr 10, 2026

Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation

Authors: Lingfeng Huang, Huizhong Guo, Tianjun Wei, Yingpeng Du, Zhu Sun

Organizations: Singapore University of Technology and Design, Singapore · Zhejiang University, Hangzhou, China · Nanyang Technological University, Singapore

Abstract

Large language model (LLM) agents are increasingly deployed as scalable user simulators for recommender system evaluation. Yet existing simulators perceive recommendations through text or structured metadata rather than the visual interfaces real users browse-a critical gap, since attention over recommendation layouts is both visually driven and highly personalized. We investigate whether aligning a vision-language model's (VLM's) visual attention with user-specific gaze patterns can improve simulation fidelity. Analysis of a real-world eye-tracking dataset collected in a carousel-based recommendation setting reveals that users exhibit stable individual gaze patterns strongly predictive of click behavior. Building on this finding, we propose Fixation-Aligned Tuning for user Emulation (FixATE). Our approach first probes the VLM's internal visual attention via interpretability operators to obtain a slot-level relevance distribution comparable with human fixation, and then learns personalized soft prompts to steer the model's attention toward each user's characteristic fixation pattern. Experiments across three interpretability-based probing operators and two architecturally distinct VLM backbones demonstrate consistent improvements in both attention alignment and click prediction accuracy. These results suggest that making the model "see like the user" is a viable path toward simulators that more faithfully reproduce how users perceive and act in recommendation interfaces.

Figures & tables

Figure 1. The difference of the perception of recommendations between real users and LLM-based user simulators. This figure shows the difference of the perception of recommendations between real users and LLM-based user simulators. The right panel shows the real users' attention distribution when browsing the movie poster carousel. The left panel shows the LLM-based user simulator's perception: movie posters are converted into text descriptions and fed into the LLM user simulators.

Panel of Figure 2 — Figure 2. Empirical analyses on the RecGaze dataset. Two subfigures. (a) Four heatmaps showing click position distributions on a 3x5 grid for Ground Truth, Text w/o Pos., Image, and Text w/ Pos. conditions. The Image condition best matches the ground truth F-shaped pattern. (b) A grouped bar chart showing that the clicked movie ranks first in cumulative fixation duration 71% of the time and first in cursor duration 67% of the time.

Figure 3. Overview of the Fixation-aligned Tuning for User Emulation ( FixATE ) framework.

Panel of Figure 4 — Figure 4. Multimodal prompt template used in FixATE .

Model	Method	Setting	Attention alignment metrics									Prediction-level metrics
Model	Method	Setting	CSH@1	CSH@3	CSH@5	TGO@1	TGO@3	TGO@5	Cosine	JS $\downarrow$	KL $\downarrow$	Accuracy	LogLoss $\downarrow$	AUC
Qwen3-VL- 4B-Instruct	GLIMPSE	Backbone	0.1447	0.3947	0.5789	0.1447	0.3684	0.5211	0.5270	0.2552	1.1033	0.1053	2.7860	0.5893
		FixATE	0.2368	0.4211	0.6184	0.1974	0.4035	0.5579	0.5559	0.2354	0.9986	0.1974	2.6042	0.6607
		Improv.	63.64%	6.67%	6.82%	36.36%	9.52%	7.07%	5.49%	7.76%	9.49%	87.50%	6.53%	12.12%
	AttnLRP	Backbone	0.0395	0.3026	0.4474	0.0658	0.3158	0.4974	0.4828	0.2704	1.2187	0.1053	2.7860	0.5893
		FixATE	0.1316	0.4605	0.5921	0.1184	0.3860	0.5289	0.5350	0.2559	1.0954	0.2105	2.5759	0.6701

Table 1. Evaluation results across probing operators and VLM backbones. Backbone denotes the baseline without personalized soft prompts, while FixATE uses the learned user-specific prompt

\mathbf{P}_{u}

. Improv. denotes the relative improvement (%).

Model	Setting	Attention alignment metrics									Prediction-level metrics
Model	Setting	CSH@1	CSH@3	CSH@5	TGO@1	TGO@3	TGO@5	Cosine	JS $\downarrow$	KL $\downarrow$	Accuracy	LogLoss $\downarrow$	AUC
Qwen3-VL- 4B-Instruct	Backbone	0.0395	0.3026	0.4474	0.0658	0.3158	0.4974	0.4828	0.2704	1.2187	0.1053	2.7860	0.5893
	Rand. SP	0.0658	0.1711	0.3684	0.0526	0.1263	0.4263	0.4250	0.3055	1.3509	0.0789	2.8992	0.5620
	$\boldsymbol{z}_{u}=\mathbf{1}$	0.0263	0.3289	0.5263	0.0789	0.3070	0.4816	0.4964	0.2727	1.1892	0.0921	2.8945	0.5883
	w/o $\omega_{n}$	0.0789	0.3158	0.4737	0.0921	0.3246	0.5026	0.4890	0.2690	1.1926	0.1184	2.7594	0.6118
	w/o $\mathcal{L}_{\text{Attn}}$	0.0395	0.2500	0.4605	0.0789	0.2675	0.4211	0.4336	0.2983	1.3272	0.0921	2.8066	0.5780

Table 2. Ablation study results using AttnLRP as the probing operator. Best values per model are highlighted with a light red background and boldface; values better than Backbone are highlighted with a light blue background.

Figure 5. Sensitivity analysis of key hyperparameters on two backbones with AttnLRP . Each curve reports a representative metric as the hyperparameter varies.

Panel of Figure 6 — Figure 6. Case study of slot-level attention on Qwen3-VL . (a) Original backbone attention; (b) Human gaze; (c) Attention from FixATE . Posters are blurred for copyright reasons.

Model	Setting	CSH@1	TGO@1	KL $\downarrow$	Accuracy	LogLoss $\downarrow$	AUC
Qwen3-VL- 4B-Instruct	Backbone	0.2727	0.3939	0.5797	0.2727	1.4450	0.6760
	FixATE	0.3636	0.5758	0.5547	0.3636	1.3516	0.6851
	Improv.	33.33%	46.15%	4.31%	33.33%	6.46%	1.35%
InternVL3.5- 4B-Instruct	Backbone	0.2121	0.4242	0.8482	0.2121	1.8929	0.5048
	FixATE	0.2727	0.4848	0.7355	0.2727	1.7947	0.5547
	Improv.	28.57%	14.29%	13.29%	28.57%	5.19%	9.89%

Table 3. Evaluation of AttnLRP on the AdSERP dataset.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	Setting	Attention alignment metrics			Prediction-level metrics
Model	Method	Setting	CSH@1	TGO@1	KL $\downarrow$	AUC	LogLoss $\downarrow$	Accuracy
Qwen3-VL- 4B-Instruct	GLIMPSE	Backbone	0.1447	0.1447	1.1033	0.5893	2.7860	0.1053
	GLIMPSE	FixATE	0.2368 $\pm$ 0.053 ∗	0.1974 $\pm$ 0.029 ∗	0.9986 $\pm$ 0.060 ∗	0.6607 $\pm$ 0.008 ∗∗	2.6042 $\pm$ 0.082 ∗∗	0.1974 $\pm$ 0.045 ∗∗
	AttnLRP	Backbone	0.0395	0.0658	1.2187	0.5893	2.7860	0.1053
	AttnLRP	FixATE	0.1316 $\pm$ 0.038 ∗∗	0.1184 $\pm$ 0.029 ∗	1.0954 $\pm$ 0.056 ∗∗	0.6701 $\pm$ 0.009 ∗∗	2.5759 $\pm$ 0.081 ∗∗	0.2105 $\pm$ 0.047 ∗∗
	Attention Rollout	Backbone	0.0921	0.0658	1.7114	0.5893	2.7860	0.1053

Appendix

Table 4. Extended evaluation on RecGaze with standard deviations and significance levels (cf. Table 1 in the main paper). ∗

p<0.05

, ∗∗

p<0.01

( two-sided paired

t

-test,

\mathrm{df}=4

). Values are reported as mean

\pm

std. Backbone values are deterministic (frozen model, greedy decoding) and thus reported without variance. Bold indicates improvement over Backbone .

Model	Setting	Attention alignment metrics			Prediction-level metrics
Model	Setting	CSH@1	TGO@1	KL $\downarrow$	AUC	LogLoss $\downarrow$	Accuracy
Qwen3-VL- 4B-Instruct	Backbone	0.2727	0.3939	0.5797	0.6760	1.4450	0.2727
Qwen3-VL- 4B-Instruct	FixATE	0.3636 $\pm$ 0.048 ∗	0.5758 $\pm$ 0.057 ∗∗	0.5547 $\pm$ 0.023	0.6851 $\pm$ 0.006 ∗	1.3516 $\pm$ 0.025 ∗∗	0.3636 $\pm$ 0.030 ∗∗
InternVL3.5- 4B-Instruct	Backbone	0.2121	0.4242	0.8482	0.5048	1.8929	0.2121
InternVL3.5- 4B-Instruct	FixATE	0.2727 $\pm$ 0.043 ∗	0.4848 $\pm$ 0.048 ∗	0.7355 $\pm$ 0.035 ∗∗	0.5547 $\pm$ 0.006 ∗∗	1.7947 $\pm$ 0.040 ∗∗	0.2727 $\pm$ 0.037 ∗

Appendix

Table 5. Extended evaluation on AdSERP (AttnLRP) with standard deviations and significance levels (cf. Table 3 in the main paper). ∗

p<0.05

, ∗∗

p<0.01

( two-sided paired

t

-test,

\mathrm{df}=4

Aspect	Attn Rollout	AttnLRP	GLIMPSE
Gradient required	No	Yes	Yes
Head aggregation	Uniform mean	Uniform mean	Adaptive softmax
Per-layer normalization	Row-norm with $\tfrac{1}{2}\mathbf{I}$	Signed-sum	Row-norm after ReLU
Cross-layer propagation	Sequential product	Depth-weighted additive	Iterative multiplicative
Differentiable	No (post-hoc)	Yes	Yes
Relative cost	Lowest	Moderate	Highest

Appendix

Table 6. Comparison of the three probing operators.

Panel of Figure 7 — Figure 7. Supplementary qualitative comparison of slot-level attention. Each row is one session (user $\times$ task); columns are frozen Qwen3-VL ( Backbone ), human fixation, and FixATE after personalized tuning. Posters are blurred where required for copyright.

Explore similar work

Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
Investigating Assistant Bias in LLM User Simulators Using a Role Vector
Sep 1, 2026Daeheon Jeong, Yoonjoo Lee, Eugene Choi +2Large Language Model-Based Role-Play Simulation User Simulation
1KAIST · University of Michigan · 3Seoul National University +2
A Behavioral Trait Leaks into Preferences: Diagnosing Trait Interference in LLM User Simulators
Sep 22, 2026Chaehyun Kim, Sein Kim, Hongseok Kang +1Recommender Systems User Simulation
KAIST Daejeon, Republic of Korea

Mar 11, 2026J/K move · Enter open · S save

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie+7

Carnegie Mellon University, Language Technologies Institute

User Simulation Sim-to-Real Transfer LLM Agent Evaluation Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full

τ

-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Abstract

Explore similar work

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Investigating Assistant Bias in LLM User Simulators Using a Role Vector

A Behavioral Trait Leaks into Preferences: Diagnosing Trait Interference in LLM User Simulators