Alfa: Attentive Low-Rank Filter Adaptation for Structure-Aware Cross-Domain Personalized Gaze Estimation

He-Yen Hsieh; Wei-Te Mark Ting; H. T. Kung

cs.CVMar 9, 2026

Alfa: Attentive Low-Rank Filter Adaptation for Structure-Aware Cross-Domain Personalized Gaze Estimation

Authors: He-Yen Hsieh, Wei-Te Mark Ting, H. T. Kung

Organizations: Harvard University

Abstract

Pre-trained gaze models learn to identify useful patterns commonly found across users, but subtle user-specific variations (i.e., eyelid shape or facial structure) can degrade model performance. Test-time personalization (TTP) adapts pre-trained models to these user-specific domain shifts using only a few unlabeled samples. Efficient fine-tuning is critical in performing this domain adaptation: data and computation resources can be limited-especially for on-device customization. While popular parameter-efficient fine-tuning (PEFT) methods address adaptation costs by updating only a small set of weights, they may not be taking full advantage of structures encoded in pre-trained filters. To more effectively leverage existing structures learned during pre-training, we reframe personalization as a process to reweight existing features rather than learning entirely new ones. We present Attentive Low-Rank Filter Adaptation (Alfa) to adapt gaze models by reweighting semantic patterns in pre-trained filters. With Alfa, singular value decomposition (SVD) extracts dominant spatial components that capture eye and facial characteristics across users. Via an attention mechanism, we need only a few unlabeled samples to adjust and reweight pre-trained structures, selectively amplifying those relevant to a target user. Alfa achieves the lowest average gaze errors across four cross-dataset gaze benchmarks, outperforming existing TTP methods and low-rank adaptation (LoRA)-based variants. We also show that Alfa's attentive low-rank methods can be applied to applications beyond vision, such as diffusion-based language models.

Figures & tables

Figure 1: Alfa achieves the lowest average gaze error, with the smallest model size, across four cross-dataset benchmarks: from ETH-XGaze to MPIIGaze, ETH-XGaze to EyeDiap, Gaze360 to MPIIGaze, and Gaze360 to EyeDiap. Top: Comparison with other test-time personalization (TTP) methods. Baseline refers to a ResNet-18 without fine-tuning. Bottom: Comparison with low-rank adaptation (LoRA)-based variants.

Figure 2: Overview of Attentive Low-Rank Filter Adaptation (Alfa). (a) The pre-trained weight matrix is approximated using truncated SVD: $W_{d}=U_{d}S_{d}V_{d}^{\top}$ . Then, a tunable low-rank update $\Delta W$ is added for adaptation. (b) Alfa adapts gaze models by reweighting spatial structures encoded in pre-trained filters. Alfa extracts dominant spatial patterns ( $V_{\text{base}}=S_{d}V_{d}^{\top}$ ) via singular value decomposition (SVD). For personalization, multi-head low-rank modules $A^{\mathcal{Q}}$ and $B^{\mathcal{Q}}$ generate query weights, and $V_{\text{base}}$ and $V_{\text{base}}^{\top}$ are reused as key and value matrices. Using multi-head scaled dot-product attention, Alfa identifies the spatial structures most relevant to a target user. Alfa aggregates this into a personalized update using additional low-rank modules $A^{\mathcal{P}}$ and $B^{\mathcal{P}}$ , forming $V_{\text{Alfa}}$ , which encodes gaze-specific adaptations informed by the pre-trained spatial structure.

Figure 3: Comparison with other LoRA-based variants, including LoRA, MiLoRA, Spectral Adapter A , DoRA, MoSLoRA, MELoRA, and FLoRA. Alfa selectively reuses semantic patterns encoded in pre-trained weights and activates the most relevant ones during adaptation. Only blocks with red backgrounds are tunable. Best viewed in color.

Method	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Baseline (ResNet-18)	11.18	0	11.18	8.02	7.30	7.79	8.19	7.83
PnP-GA † ( Liu et al. 2021 )	116.9	116.9	116.9	6.91	7.18	7.36	8.17	7.41
RUDA † ( Bao et al. 2022 )	12.20	12.20	12.20	6.86	6.84	6.96	5.32	6.50
TPGaze ( Liu et al. 2024a )	11.18	0.13	11.18	6.30	5.89	6.62	5.04	5.96
Alfa	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72

Table 1: Comparison with state-of-the-art TTP methods across four cross-domain benchmarks. Datasets are denoted as follows:

D_{E}

= ETH-XGaze,

D_{M}

= MPIIGaze,

D_{G}

= Gaze360, and

D_{D}

= EyeDiap. Baseline is a ResNet-18 model without any fine-tuning. Results are reported in angular gaze error ( ∘ ). Bold indicates the best result. The symbol

\dagger

indicates results obtained from the re-implementation of TPGaze ( Liu et al. 2024a ) .

Method	Rank		# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	SVD	LoRA	Train	Tuned	Test	$D_{E}\mkern-4.0mu\rightarrow\mkern-4.0muD_{M}$	$D_{E}\mkern-4.0mu\rightarrow\mkern-4.0muD_{D}$	$D_{G}\mkern-4.0mu\rightarrow\mkern-4.0muD_{M}$	$D_{G}\mkern-4.0mu\rightarrow\mkern-4.0muD_{D}$	Avg
Baseline (No Adaptation)	64	-	2.31	0	2.31	6.60	8.84	6.86	6.83	7.29
LoRA ( Hu et al. 2022 )	64	8	2.84	0.53	2.84	5.66	6.17	6.23	5.72	5.95
MiLoRA ( Wang et al. 2025 )	64	8	2.84	0.29	2.84	5.67	6.25	6.23	6.00	6.04
DoRA ( Liu et al. 2024b )	64	8	2.87	0.56	2.87	5.51	5.83	6.30	5.98	5.91
MoSLoRA ( Wu et al. 2024 )	64	8	2.85	0.54	2.85	5.55	6.13	6.31	5.70	5.92

Table 2: Comparison with other LoRA-based variant methods. The baseline is not fine-tuned on the target domain ( i.e. no adaptation). All methods use the same truncated pre-trained weight matrix

W_{d}=U_{d}S_{d}V_{d}^{\top}

. While some variants introduce additional components that cannot be merged into

W_{d}

, Alfa supports full mergeability, enabling efficient inference without extra computational overhead. Bold indicates the best result.

Backbone	Method	LoRA Rank	Tuned Params (Usage %)	GSM8K (0-shot)		MATH500 (0-shot)		Countdown (0-shot)		Sudoku (0-shot)
Backbone	Method	LoRA Rank		128	256	128	256	128	256	128	256
LLaDA-8B-Instruct	LoRA	128	100.7M (1.24 $\%$ )	66.5	78.8	26.2	32.6	20.3	14.5	16.5	8.5
	DoRA †	128	101.1M (1.25 $\%$ )	68.1	76.8	26.2	33.4	21.5	16.0	17.2	8.0
	LoRA †	64	50.3M (0.62 $\%$ )	67.9	77.9	25.0	33.0	16.4	12.5	14.5	7.9
	Alfa	64	69.2M (0.85 $\%$ )	68.4	77.1	26.6	33.8	27.3	17.2	9.7	8.3

Table 3: Comparison of Alfa, LoRA, and DoRA on LLaDA-8B-Instruct across four zero-shot reasoning tasks. Alfa achieves competitive or better performance while using only 0.85% of tunable parameters. Bold indicates the best result, and italics indicate the second best. The symbol

\dagger

indicates re-implementation.

Figure 4: Spatial patterns captured during pre-training. Visualizations use rank slices from SVD-decomposed weights of ResNet-18 (pre-trained on ETH-XGaze) from conv1 and the first block of layer3. Left column: visualization of encoded pattern. Middle and right columns: activations for Subject 0 and 16 from ETH-XGaze using $U_{d}[:,s]S_{d}[s]V_{d}^{\top}[s]$ for slice $s$ . Red regions indicate higher activations.

Figure 5: Visualization of low-rank updates $\Delta W$ on the MPIIGaze test set for LoRA and Alfa using filters from conv1 and the first block of layer3 in the ResNet-18 model (pre-trained on ETH-XGaze). Red regions indicate higher activation values. When using LoRA updates, model is highly inconsistent with respect to the significant regions of focus across users. In contrast, Alfa captures localized regions consistently across users. This shows Alfa identifies useful components that translate well between source and target domains from the semantic base dictionary. Reweighting these components allows for effective adaptation.

Heads	# Parameters (M)			Source $\mkern-2.0mu\rightarrow\mkern-2.0mu$ Target Domain (5-shot)				Avg
Heads	Train	Tuned	Test	$D_{E}\mkern-4.0mu\rightarrow\mkern-4.0muD_{M}$	$D_{E}\mkern-4.0mu\rightarrow\mkern-4.0muD_{D}$	$D_{G}\mkern-4.0mu\rightarrow\mkern-4.0muD_{M}$	$D_{G}\mkern-4.0mu\rightarrow\mkern-4.0muD_{D}$	Avg
1	2.35	0.04	2.31	5.74	6.84	6.36	5.87	6.20
2	2.40	0.10	2.31	5.91	6.65	6.32	5.75	6.16
4	2.58	0.27	2.31	5.47	6.12	6.32	5.82	5.93
8	3.16	0.86	2.31	5.67	6.19	6.24	5.76	5.97
16	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72

Table 4: Ablation study on attention head count in Alfa. Personalization performance generally improves with head count, with the lowest average gaze error at 16 heads. Heads share the left basis

U_{d}

, so adapted weights can be merged into the base model without extra computation at inference.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Augmentation	From $D_{E}$		From $D_{G}$
Augmentation	$\rightarrow D_{M}$	$\rightarrow D_{E}$	$\rightarrow D_{M}$	$\rightarrow D_{E}$
Repeat	1	1	1	1
ColorJitter	brightness=0.2	brightness=0.2	brightness=0.2	brightness=0.2
	contrast=0.2	contrast=0.2	contrast=0.2	contrast=0.2
	saturation=0.1	saturation=0.1	saturation=0.1	saturation=0.1
	hue=0.05	hue=0.05	hue=0.05	hue=0.05

Appendix

Table 5: Gaze data augmentation settings used for TTP. We use PyTorch torchvision to apply augmentations to each target user image. Each image is repeated once to generate 10 personalized samples per subject.

Method	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Baseline (ResNet-18)	11.18	0	11.18	8.02	7.30	7.79	8.19	7.83
Source-available UDA Methods
DAGEN ( Guo et al. 2020 )	-	-	-	7.53	8.46	9.31	12.05	9.34
GazeAdv ( Wang et al. 2019 )	-	-	-	8.48	7.70	9.15	11.15	9.12
Gaze360 ( Kellnhofer et al. 2019 )	-	-	-	7.86	9.64	7.71	9.54	8.69

Appendix

Table 6: Comparison with state-of-the-art source-available UDA and TTP methods across four cross-domain benchmarks. Results are reported in angular gaze error ( ∘ ). Datasets are denoted:

D_{E}

= ETH-XGaze,

D_{M}

= MPIIGaze,

D_{G}

= Gaze360, and

D_{D}

= EyeDiap. Baseline is a ResNet-18 model without any fine-tuning. Bold indicates the best result. The symbol

\dagger

indicates results obtained from a re-implementation of TPGaze ( Liu et al. 2024a ) . We also include source-available in-domain adaptation results with supervision ( e.g. , DFT Gaze ( Hsieh et al. 2024 ) ). Despite DFT Gaze having access to target-domain labels, Alfa achieves better performance when adapting to

D_{M}

without labels.

Method	LoRA Rank	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	LoRA Rank	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Alfa	4	3.79	1.48	2.31	5.39	6.09	6.08	5.78	5.84
Alfa	8	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72
Alfa	16	8.22	5.92	2.31	5.44	6.12	6.13	5.91	5.90
Alfa	32	14.14	11.83	2.31	5.36	6.10	6.21	5.82	5.87

Appendix

Table 7: Ablation study on LoRA rank in Alfa. Rank

r=8

achieves the best trade-off between adaptation capacity and stability, resulting in the lowest average gaze error.

Method	SVD Rank	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	SVD Rank	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
LoRA	32	1.70	0.53	1.70	6.98	6.93	7.19	7.41	7.13
Alfa	32	2.64	1.48	1.16	6.45	6.64	7.46	6.22	6.69
LoRA	64	2.84	0.53	2.84	5.66	6.17	6.23	5.72	5.95
Alfa	64	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72
LoRA	128	4.95	0.53	4.95	5.90	6.83	6.18	5.94	6.21

Appendix

Table 8: Effect of SVD rank on Alfa. A rank of

d=64

provides the best balance between adaptation capacity and stability, achieving the lowest average gaze error across domains.

Method	Num Samples	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	Num Samples	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Alfa	1	5.26	2.98	2.31	5.52	6.96	6.37	5.65	6.13
Alfa	5	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72
Alfa	10	5.26	2.98	2.31	5.45	5.88	5.64	5.84	5.70

Appendix

Table 9: Effect of the number of personal samples. Performance improves as more samples are provided.

Method	With Attention	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	With Attention	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Alfa	✓	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72
Alfa	✗	5.26	2.98	2.31	6.60	8.84	6.86	6.84	7.29

Appendix

Table 10: Effect of using attention in Alfa. Removing attention degrades performance, showing its role in selecting and reweighting semantic basis slices for personalization.

Method	# Parameters (M)			$D_{E}\rightarrow D_{E}$					Avg
Method	Train	Tuned	Test	subject0003	subject0007	subject0027	subject0075	subject0102	Avg
Baseline (No Adaptation)	2.84	0	2.31	7.40	3.85	5.75	5.14	4.43	5.31
LoRA	2.84	0.53	2.84	5.47	3.87	5.79	5.18	4.42	4.95
Alfa	5.26	2.98	2.31	4.44	3.78	5.73	5.15	4.05	4.63

Appendix

Table 11: Personalization using only 5 frontal images. Although the pre-trained model already covers diverse head poses, Alfa adapts effectively from limited frontal views and achieves the lowest gaze error across all subjects.

Figure 6: Visualization of low-rank updates $\Delta W$ across all target users on the MPIIGaze test set for Alfa , using filters from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation.

Figure 7: Visualization of low-rank updates $\Delta W$ across all target users on the MPIIGaze test set for LoRA , using filters from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation.

Figure 8: Visualization of selected rank slices from Alfa with head index $h=0$ for subject p02 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p02 . Head index starting from 0.

Figure 9: Visualization of selected rank slices from Alfa with head index $h=7$ for subject p02 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p02 . Head index starting from 0.

Figure 10: Visualization of selected rank slices from Alfa with head index $h=12$ for subject p02 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p02 . Head index starting from 0.

Figure 11: Visualization of selected rank slices from Alfa with head index $h=1$ for subject p04 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p04 . Head index starting from 0.

Figure 12: Visualization of selected rank slices from Alfa with head index $h=4$ for subject p04 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p04 . Head index starting from 0.

Figure 13: Visualization of selected rank slices from Alfa with head index $h=10$ for subject p04 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p04 . Head index starting from 0.

Figure 14: Visualization of selected rank slices from Alfa with head index $h=5$ for subject p13 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p13 . Head index starting from 0.

Figure 15: Visualization of selected rank slices from Alfa with head index $h=6$ for subject p13 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p13 . Head index starting from 0.

Figure 16: Visualization of selected rank slices from Alfa with head index $h=9$ for subject p13 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p13 . Head index starting from 0.

Gaze Estimation

Gaze estimation predicts where a person is looking from eye, face, head-pose, and scene cues.

See all 66 papers

PAGE: Towards Practical Human-level Gaze Target Estimation
Jul 6, 2026Zhoutong Ye, Chengwen Zhang, Zhaibin Cui +10Gaze Estimation
1Tsinghua University · 2Jinan University
DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation
Oct 13, 2026Yangming Zhang, Ling Yi, Datong Li +4Low-Rank Adaptation Parameter-Efficient Fine-Tuning
Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China
Investigating Bias and Fairness in Appearance-based Gaze Estimation
Apr 12, 2026Burak Akgül, Erol Şahin, Sinan KalkanGaze Estimation Algorithmic Fairness
Dept. of Computer Eng. and ROMER Robotics Center, Middle East Technical University, Ankara, Turkey

Jul 6, 2026J/K move · Enter open · S save

PAGE: Towards Practical Human-level Gaze Target Estimation

Zhoutong Ye, Chengwen Zhang, Zhaibin Cui, Mingze Sun+9

1Tsinghua University · 2Jinan University

Gaze Estimation

Open paper AskPDF arXiv

Gaze target estimation, the task of predicting where a person is looking in a scene, is crucial to understanding human attention and intent. It is a challenging task that combines high-level understanding of global scene semantics and precise spatial reasoning using human appearance (e.g. pose, eye orientation). As a result, human-level performance remains elusive for existing models, limiting their practical application. To this end, we propose PaGE (Practical Gaze Estimator), a gaze estimation model that explicitly models the complex interaction between scene and head features. Using a PaGE model with a large ViT-H+ backbone as the teacher, we further distill student models with lighter backbones on a much larger and more diverse unlabeled dataset. The architectural improvements and novel training recipe allow PaGE to achieve state-of-the-art performance on several gaze estimation tasks, outperforming humans in 7 out of 9 metrics while reducing the human-AI gap by at least 60% in the remaining 2. The distilled student models retain most of the teacher's performance while being lightweight enough for practical deployment on robots and consumer devices. The code and model checkpoints are available at our project page.

Abstract

Gaze Estimation

Explore similar work

PAGE: Towards Practical Human-level Gaze Target Estimation

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Investigating Bias and Fairness in Appearance-based Gaze Estimation