Alfa: Attentive Low-Rank Filter Adaptation for Structure-Aware Cross-Domain Personalized Gaze Estimation

He-Yen Hsieh; Wei-Te Mark Ting; H. T. Kung

cs.CVMar 9, 2026

Alfa: Attentive Low-Rank Filter Adaptation for Structure-Aware Cross-Domain Personalized Gaze Estimation

Authors: He-Yen Hsieh, Wei-Te Mark Ting, H. T. Kung

Organizations: Harvard University

Abstract

Pre-trained gaze models learn to identify useful patterns commonly found across users, but subtle user-specific variations (i.e., eyelid shape or facial structure) can degrade model performance. Test-time personalization (TTP) adapts pre-trained models to these user-specific domain shifts using only a few unlabeled samples. Efficient fine-tuning is critical in performing this domain adaptation: data and computation resources can be limited-especially for on-device customization. While popular parameter-efficient fine-tuning (PEFT) methods address adaptation costs by updating only a small set of weights, they may not be taking full advantage of structures encoded in pre-trained filters. To more effectively leverage existing structures learned during pre-training, we reframe personalization as a process to reweight existing features rather than learning entirely new ones. We present Attentive Low-Rank Filter Adaptation (Alfa) to adapt gaze models by reweighting semantic patterns in pre-trained filters. With Alfa, singular value decomposition (SVD) extracts dominant spatial components that capture eye and facial characteristics across users. Via an attention mechanism, we need only a few unlabeled samples to adjust and reweight pre-trained structures, selectively amplifying those relevant to a target user. Alfa achieves the lowest average gaze errors across four cross-dataset gaze benchmarks, outperforming existing TTP methods and low-rank adaptation (LoRA)-based variants. We also show that Alfa's attentive low-rank methods can be applied to applications beyond vision, such as diffusion-based language models.

Figures & tables

Figure 1: Alfa achieves the lowest average gaze error, with the smallest model size, across four cross-dataset benchmarks: from ETH-XGaze to MPIIGaze, ETH-XGaze to EyeDiap, Gaze360 to MPIIGaze, and Gaze360 to EyeDiap. Top: Comparison with other test-time personalization (TTP) methods. Baseline refers to a ResNet-18 without fine-tuning. Bottom: Comparison with low-rank adaptation (LoRA)-based variants.

Figure 2: Overview of Attentive Low-Rank Filter Adaptation (Alfa). (a) The pre-trained weight matrix is approximated using truncated SVD: $W_{d}=U_{d}S_{d}V_{d}^{\top}$ . Then, a tunable low-rank update $\Delta W$ is added for adaptation. (b) Alfa adapts gaze models by reweighting spatial structures encoded in pre-trained filters. Alfa extracts dominant spatial patterns ( $V_{\text{base}}=S_{d}V_{d}^{\top}$ ) via singular value decomposition (SVD). For personalization, multi-head low-rank modules $A^{\mathcal{Q}}$ and $B^{\mathcal{Q}}$ generate query weights, and $V_{\text{base}}$ and $V_{\text{base}}^{\top}$ are reused as key and value matrices. Using multi-head scaled dot-product attention, Alfa identifies the spatial structures most relevant to a target user. Alfa aggregates this into a personalized update using additional low-rank modules $A^{\mathcal{P}}$ and $B^{\mathcal{P}}$ , forming $V_{\text{Alfa}}$ , which encodes gaze-specific adaptations informed by the pre-trained spatial structure.

Figure 3: Comparison with other LoRA-based variants, including LoRA, MiLoRA, Spectral Adapter A , DoRA, MoSLoRA, MELoRA, and FLoRA. Alfa selectively reuses semantic patterns encoded in pre-trained weights and activates the most relevant ones during adaptation. Only blocks with red backgrounds are tunable. Best viewed in color.

Method	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Baseline (ResNet-18)	11.18	0	11.18	8.02	7.30	7.79	8.19	7.83
PnP-GA † ( Liu et al. 2021 )	116.9	116.9	116.9	6.91	7.18	7.36	8.17	7.41
RUDA † ( Bao et al. 2022 )	12.20	12.20	12.20	6.86	6.84	6.96	5.32	6.50
TPGaze ( Liu et al. 2024a )	11.18	0.13	11.18	6.30	5.89	6.62	5.04	5.96
Alfa	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72

Table 1: Comparison with state-of-the-art TTP methods across four cross-domain benchmarks. Datasets are denoted as follows:

D_{E}

= ETH-XGaze,

D_{M}

= MPIIGaze,

D_{G}

= Gaze360, and

D_{D}

= EyeDiap. Baseline is a ResNet-18 model without any fine-tuning. Results are reported in angular gaze error ( ∘ ). Bold indicates the best result. The symbol

\dagger

indicates results obtained from the re-implementation of TPGaze ( Liu et al. 2024a ) .

Method	Rank		# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	SVD	LoRA	Train	Tuned	Test	$D_{E}\mkern-4.0mu\rightarrow\mkern-4.0muD_{M}$	$D_{E}\mkern-4.0mu\rightarrow\mkern-4.0muD_{D}$	$D_{G}\mkern-4.0mu\rightarrow\mkern-4.0muD_{M}$	$D_{G}\mkern-4.0mu\rightarrow\mkern-4.0muD_{D}$	Avg
Baseline (No Adaptation)	64	-	2.31	0	2.31	6.60	8.84	6.86	6.83	7.29
LoRA ( Hu et al. 2022 )	64	8	2.84	0.53	2.84	5.66	6.17	6.23	5.72	5.95
MiLoRA ( Wang et al. 2025 )	64	8	2.84	0.29	2.84	5.67	6.25	6.23	6.00	6.04
DoRA ( Liu et al. 2024b )	64	8	2.87	0.56	2.87	5.51	5.83	6.30	5.98	5.91
MoSLoRA ( Wu et al. 2024 )	64	8	2.85	0.54	2.85	5.55	6.13	6.31	5.70	5.92

Table 2: Comparison with other LoRA-based variant methods. The baseline is not fine-tuned on the target domain ( i.e. no adaptation). All methods use the same truncated pre-trained weight matrix

W_{d}=U_{d}S_{d}V_{d}^{\top}

. While some variants introduce additional components that cannot be merged into

W_{d}

, Alfa supports full mergeability, enabling efficient inference without extra computational overhead. Bold indicates the best result.

Backbone	Method	LoRA Rank	Tuned Params (Usage %)	GSM8K (0-shot)		MATH500 (0-shot)		Countdown (0-shot)		Sudoku (0-shot)
Backbone	Method	LoRA Rank		128	256	128	256	128	256	128	256
LLaDA-8B-Instruct	LoRA	128	100.7M (1.24 $\%$ )	66.5	78.8	26.2	32.6	20.3	14.5	16.5	8.5
	DoRA †	128	101.1M (1.25 $\%$ )	68.1	76.8	26.2	33.4	21.5	16.0	17.2	8.0
	LoRA †	64	50.3M (0.62 $\%$ )	67.9	77.9	25.0	33.0	16.4	12.5	14.5	7.9
	Alfa	64	69.2M (0.85 $\%$ )	68.4	77.1	26.6	33.8	27.3	17.2	9.7	8.3

Table 3: Comparison of Alfa, LoRA, and DoRA on LLaDA-8B-Instruct across four zero-shot reasoning tasks. Alfa achieves competitive or better performance while using only 0.85% of tunable parameters. Bold indicates the best result, and italics indicate the second best. The symbol

\dagger

indicates re-implementation.

Figure 4: Spatial patterns captured during pre-training. Visualizations use rank slices from SVD-decomposed weights of ResNet-18 (pre-trained on ETH-XGaze) from conv1 and the first block of layer3. Left column: visualization of encoded pattern. Middle and right columns: activations for Subject 0 and 16 from ETH-XGaze using $U_{d}[:,s]S_{d}[s]V_{d}^{\top}[s]$ for slice $s$ . Red regions indicate higher activations.

Figure 5: Visualization of low-rank updates $\Delta W$ on the MPIIGaze test set for LoRA and Alfa using filters from conv1 and the first block of layer3 in the ResNet-18 model (pre-trained on ETH-XGaze). Red regions indicate higher activation values. When using LoRA updates, model is highly inconsistent with respect to the significant regions of focus across users. In contrast, Alfa captures localized regions consistently across users. This shows Alfa identifies useful components that translate well between source and target domains from the semantic base dictionary. Reweighting these components allows for effective adaptation.

Heads	# Parameters (M)			Source $\mkern-2.0mu\rightarrow\mkern-2.0mu$ Target Domain (5-shot)				Avg
Heads	Train	Tuned	Test	$D_{E}\mkern-4.0mu\rightarrow\mkern-4.0muD_{M}$	$D_{E}\mkern-4.0mu\rightarrow\mkern-4.0muD_{D}$	$D_{G}\mkern-4.0mu\rightarrow\mkern-4.0muD_{M}$	$D_{G}\mkern-4.0mu\rightarrow\mkern-4.0muD_{D}$	Avg
1	2.35	0.04	2.31	5.74	6.84	6.36	5.87	6.20
2	2.40	0.10	2.31	5.91	6.65	6.32	5.75	6.16
4	2.58	0.27	2.31	5.47	6.12	6.32	5.82	5.93
8	3.16	0.86	2.31	5.67	6.19	6.24	5.76	5.97
16	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72

Table 4: Ablation study on attention head count in Alfa. Personalization performance generally improves with head count, with the lowest average gaze error at 16 heads. Heads share the left basis

U_{d}

, so adapted weights can be merged into the base model without extra computation at inference.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Augmentation	From $D_{E}$		From $D_{G}$
Augmentation	$\rightarrow D_{M}$	$\rightarrow D_{E}$	$\rightarrow D_{M}$	$\rightarrow D_{E}$
Repeat	1	1	1	1
ColorJitter	brightness=0.2	brightness=0.2	brightness=0.2	brightness=0.2
	contrast=0.2	contrast=0.2	contrast=0.2	contrast=0.2
	saturation=0.1	saturation=0.1	saturation=0.1	saturation=0.1
	hue=0.05	hue=0.05	hue=0.05	hue=0.05

Appendix

Table 5: Gaze data augmentation settings used for TTP. We use PyTorch torchvision to apply augmentations to each target user image. Each image is repeated once to generate 10 personalized samples per subject.

Method	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Baseline (ResNet-18)	11.18	0	11.18	8.02	7.30	7.79	8.19	7.83
Source-available UDA Methods
DAGEN ( Guo et al. 2020 )	-	-	-	7.53	8.46	9.31	12.05	9.34
GazeAdv ( Wang et al. 2019 )	-	-	-	8.48	7.70	9.15	11.15	9.12
Gaze360 ( Kellnhofer et al. 2019 )	-	-	-	7.86	9.64	7.71	9.54	8.69

Appendix

Table 6: Comparison with state-of-the-art source-available UDA and TTP methods across four cross-domain benchmarks. Results are reported in angular gaze error ( ∘ ). Datasets are denoted:

D_{E}

= ETH-XGaze,

D_{M}

= MPIIGaze,

D_{G}

= Gaze360, and

D_{D}

= EyeDiap. Baseline is a ResNet-18 model without any fine-tuning. Bold indicates the best result. The symbol

\dagger

indicates results obtained from a re-implementation of TPGaze ( Liu et al. 2024a ) . We also include source-available in-domain adaptation results with supervision ( e.g. , DFT Gaze ( Hsieh et al. 2024 ) ). Despite DFT Gaze having access to target-domain labels, Alfa achieves better performance when adapting to

D_{M}

without labels.

Method	LoRA Rank	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	LoRA Rank	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Alfa	4	3.79	1.48	2.31	5.39	6.09	6.08	5.78	5.84
Alfa	8	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72
Alfa	16	8.22	5.92	2.31	5.44	6.12	6.13	5.91	5.90
Alfa	32	14.14	11.83	2.31	5.36	6.10	6.21	5.82	5.87

Appendix

Table 7: Ablation study on LoRA rank in Alfa. Rank

r=8

achieves the best trade-off between adaptation capacity and stability, resulting in the lowest average gaze error.

Method	SVD Rank	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	SVD Rank	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
LoRA	32	1.70	0.53	1.70	6.98	6.93	7.19	7.41	7.13
Alfa	32	2.64	1.48	1.16	6.45	6.64	7.46	6.22	6.69
LoRA	64	2.84	0.53	2.84	5.66	6.17	6.23	5.72	5.95
Alfa	64	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72
LoRA	128	4.95	0.53	4.95	5.90	6.83	6.18	5.94	6.21

Appendix

Table 8: Effect of SVD rank on Alfa. A rank of

d=64

provides the best balance between adaptation capacity and stability, achieving the lowest average gaze error across domains.

Method	Num Samples	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	Num Samples	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Alfa	1	5.26	2.98	2.31	5.52	6.96	6.37	5.65	6.13
Alfa	5	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72
Alfa	10	5.26	2.98	2.31	5.45	5.88	5.64	5.84	5.70

Appendix

Table 9: Effect of the number of personal samples. Performance improves as more samples are provided.

Method	With Attention	# Parameters (M)			Source $\rightarrow$ Target Domain (5-shot)				Avg
Method	With Attention	Train	Tuned	Test	$D_{E}\rightarrow D_{M}$	$D_{E}\rightarrow D_{D}$	$D_{G}\rightarrow D_{M}$	$D_{G}\rightarrow D_{D}$	Avg
Alfa	✓	5.26	2.98	2.31	5.30	5.82	5.91	5.86	5.72
Alfa	✗	5.26	2.98	2.31	6.60	8.84	6.86	6.84	7.29

Appendix

Table 10: Effect of using attention in Alfa. Removing attention degrades performance, showing its role in selecting and reweighting semantic basis slices for personalization.

Method	# Parameters (M)			$D_{E}\rightarrow D_{E}$					Avg
Method	Train	Tuned	Test	subject0003	subject0007	subject0027	subject0075	subject0102	Avg
Baseline (No Adaptation)	2.84	0	2.31	7.40	3.85	5.75	5.14	4.43	5.31
LoRA	2.84	0.53	2.84	5.47	3.87	5.79	5.18	4.42	4.95
Alfa	5.26	2.98	2.31	4.44	3.78	5.73	5.15	4.05	4.63

Appendix

Table 11: Personalization using only 5 frontal images. Although the pre-trained model already covers diverse head poses, Alfa adapts effectively from limited frontal views and achieves the lowest gaze error across all subjects.

Figure 6: Visualization of low-rank updates $\Delta W$ across all target users on the MPIIGaze test set for Alfa , using filters from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation.

Figure 7: Visualization of low-rank updates $\Delta W$ across all target users on the MPIIGaze test set for LoRA , using filters from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation.

Figure 8: Visualization of selected rank slices from Alfa with head index $h=0$ for subject p02 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p02 . Head index starting from 0.

Figure 9: Visualization of selected rank slices from Alfa with head index $h=7$ for subject p02 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p02 . Head index starting from 0.

Figure 10: Visualization of selected rank slices from Alfa with head index $h=12$ for subject p02 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p02 . Head index starting from 0.

Figure 11: Visualization of selected rank slices from Alfa with head index $h=1$ for subject p04 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p04 . Head index starting from 0.

Figure 12: Visualization of selected rank slices from Alfa with head index $h=4$ for subject p04 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p04 . Head index starting from 0.

Figure 13: Visualization of selected rank slices from Alfa with head index $h=10$ for subject p04 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p04 . Head index starting from 0.

Figure 14: Visualization of selected rank slices from Alfa with head index $h=5$ for subject p13 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p13 . Head index starting from 0.

Figure 15: Visualization of selected rank slices from Alfa with head index $h=6$ for subject p13 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p13 . Head index starting from 0.

Figure 16: Visualization of selected rank slices from Alfa with head index $h=9$ for subject p13 on the MPIIGaze test set. We show 64 rank slices from conv1 and the first block of layer3 in a ResNet-18 model pre-trained on ETH-XGaze. Red regions indicate stronger activation. Black borders highlight the top-10 personalized rank slices selected for subject p13 . Head index starting from 0.

Gaze Estimation

Gaze estimation predicts where a person is looking from eye, face, head-pose, and scene cues.

See all 66 papers

Jul 6, 2026

PAGE: Towards Practical Human-level Gaze Target Estimation

Gaze target estimation, the task of predicting where a person is looking in a scene, is crucial to understanding human attention and intent. It is a challenging task that combines high-level understanding of global scene semantics and precise spatial reasoning using human appearance (e.g. pose, eye orientation). As a result, human-level performance remains elusive for existing models, limiting their practical application. To this end, we propose PaGE (Practical Gaze Estimator), a gaze estimation model that explicitly models the complex interaction between scene and head features. Using a PaGE model with a large ViT-H+ backbone as the teacher, we further distill student models with lighter backbones on a much larger and more diverse unlabeled dataset. The architectural improvements and novel training recipe allow PaGE to achieve state-of-the-art performance on several gaze estimation tasks, outperforming humans in 7 out of 9 metrics while reducing the human-AI gap by at least 60% in the remaining 2. The distilled student models retain most of the teacher's performance while being lightweight enough for practical deployment on robots and consumer devices. The code and model checkpoints are available at our project page.

Zhoutong Ye, Chengwen Zhang, Zhaibin Cui +10

1Tsinghua University · 2Jinan University

Gaze Estimation

Ask PDF

Oct 13, 2026

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA variants reveals that these residual components account for most of the gradient energy and exhibit strong spectral concentration. Motivated by this finding, we propose Dynamic Tangent LoRA (DT-LoRA), which partitions a fixed rank budget into a stable branch and a dynamic branch to capture the principal residual gradient directions. We show that these directions define the optimal update of a local quadratic model under orthogonal-complement and rank constraints. Matrix-free randomized SVD recovers them without materializing the residual gradient, while controlled write-back preserves accumulated updates and stabilizes training within the allocated rank. DT-LoRA adds no trainable parameters or inference cost over LoRA. Experiments across computer vision and natural language processing tasks demonstrate consistent improvements over LoRA and its variants. Moreover, despite being a general-purpose adaptation method, DT-LoRA outperforms task-specific approaches to open-vocabulary domain generalization for semantic segmentation, achieving state-of-the-art results. Code is available at https://github.com/Snipine/DT-LoRA.

Yangming Zhang, Ling Yi, Datong Li +4

Chongqing University of Posts and Telecommunications, China · Anhui University of Science and Technology · Northeastern University, China

Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Apr 12, 2026

Investigating Bias and Fairness in Appearance-based Gaze Estimation

While appearance-based gaze estimation has achieved significant improvements in accuracy and domain adaptation, the fairness of these systems across different demographic groups remains largely unexplored. To date, there is no comprehensive benchmark quantifying algorithmic bias in gaze estimation. This paper presents the first extensive evaluation of fairness in appearance-based gaze estimation, focusing on ethnicity and gender attributes. We establish a fairness baseline by analyzing state-of-the-art models using standard fairness metrics, revealing significant performance disparities. Furthermore, we evaluate the effectiveness of existing bias mitigation strategies when applied to the gaze domain and show that their fairness contributions are limited. We summarize key insights and open issues. Overall, our work calls for research into developing robust, equitable gaze estimators. To support future research and reproducibility, we publicly release our annotations, code, and trained models at: github.com/akgulburak/gaze-estimation-fairness

Burak Akgül, Erol Şahin, Sinan Kalkan

Dept. of Computer Eng. and ROMER Robotics Center, Middle East Technical University, Ankara, Turkey

Gaze Estimation Algorithmic Fairness

Ask PDF

Abstract

Gaze Estimation

Explore similar work

PAGE: Towards Practical Human-level Gaze Target Estimation

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Investigating Bias and Fairness in Appearance-based Gaze Estimation