FaceLiVTv2: An Improved Hybrid Architecture for Efficient Mobile Face Recognition

Novendra Setyawan; Chi-Chia Sun; Mao-Hsiu Hsu; Wen-Kai Kuo; Jun-Wei Hsieh

cs.CVApr 10, 2026

FaceLiVTv2: An Improved Hybrid Architecture for Efficient Mobile Face Recognition

Authors: Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu, Wen-Kai Kuo, Jun-Wei Hsieh

Organizations: Department of Electro-Optics, National Formosa University, Taiwan · Department of Electrical Engineering University of Muhammadiyah Malang, Indonesia · Department of Electrical Engineering, National Taipei University, Taiwan · College of Artificial Intelligence and Green Energy, National Yang Ming Chiao Tung University, Taiwan

Abstract

Lightweight face recognition is increasingly important for deployment on edge and mobile devices, where strict constraints on latency, memory, and energy consumption must be met alongside reliable accuracy. Although recent hybrid CNN-Transformer architectures have advanced global context modeling, striking an effective balance between recognition performance and computational efficiency remains an open challenge. In this work, we present FaceLiVTv2, an improved version of our FaceLiVT hybrid architecture designed for efficient global--local feature interaction in mobile face recognition. At its core is Lite MHLA, a lightweight global token interaction module that replaces the original multi-layer attention design with multi-head linear token projections and affine rescale transformations, reducing redundancy while preserving representational diversity across heads. We further integrate Lite MHLA into a unified RepMix block that coordinates local and global feature interactions and adopts global depthwise convolution for adaptive spatial aggregation in the embedding stage. Under our experimental setup, results on LFW, CA-LFW, CP-LFW, CFP-FP, AgeDB-30, and IJB show that FaceLiVTv2 consistently improves the accuracy-efficiency trade-off over existing lightweight methods. Notably, FaceLiVTv2 reduces mobile inference latency by 22% relative to FaceLiVTv1, achieves speedups of up to 30.8% over GhostFaceNets on mobile devices, and delivers 20-41% latency improvements over EdgeFace and KANFace across platforms while maintaining higher recognition accuracy. These results demonstrate that FaceLiVTv2 offers a practical and deployable solution for real-time face recognition. Code is available at https://github.com/novendrastywn/FaceLiVT.

Figures & tables

Fig. 1: Comparison of our proposed FaceLiVTv2 with other lightweight FR models, including EdgeFace [ 3 ] , SwiftFaceFormer [ 4 ] , and KANFace [ 5 ] . Mean accuracy of Face Benchmark v.s. Mobile Latency (iPhone 15 Pro). The upper-left region of the plot represents the most desirable operating point, where high accuracy is achieved with minimal latency.

Methods	Context		Low	High	Fast Inference		Approach
	Global	Local	Comp.	Acc.	Edge	Mobile
MobileFaceNet [ 6 ]	✗	✓	✓	✗	✓	✓	Lightweight CNN optimized for mobile face recognition using depthwise separable convolutions [ 7 ] .
GhostFaceNet [ 8 ]	✗	✓	✓	✗	✓	✓	CNN-based face recognition model leveraging ghost modules [ 9 ] to reduce redundancy.
TransFace [ 10 ]	✓	✗	✗	✓	✗	✗	Vanilla Vision Transformer [ 11 ] for face recognition with patch-level data augmentation.
EdgeFace [ 3 ]	✓	✓	✓	✓	✓	✗	EdgeNext [ 12 ] -based architecture with low-rank linear layer.
KANFace [ 5 ]	✓	✓	✓	✓	✓	✗	Hybrid EdgeFace-based [ 3 ] architecture incorporating Kolmogorov–Arnold Networks [ 13 ] for enhanced representation.

TABLE I: Key architectural properties of representative lightweight face recognition models

Fig. 2: FaceLiVTv2 architecture with Lite MHLA and structural reparameterization. Stages 1 and 2 use RepMix and the last stage combines RepMix with Lite MHLA as spatial mixer.

CNN :	Convolutional Neural Network.
FR :	Face Recognition
FFN :	Feed Forward Network
GDConv :	Global Depthwise Convolution
MHSA :	Multi-Head Self-Attention
MHLA :	Multi-Head Linear Attention
MLP :	Multi Layer Perceptron

Table 4

Fig. 3: Architecture comparison of (a) LiteMHLA in FaceLiVTv2 and (b) MHLA in FaceLiVTv1, (c) FaceLiVTv2 Block and (d) FaceLiVT Block. C, H and W denotes as the number of channel, height and width of feature map, h denotes as the number of head on the MHLA.

Fig. 4: Structural Reparameterization on RepMix Block. This visualization illustrate the process of merging $DW_{3\times 3}$ , $DW_{1\times 1}$ , residual connection, and BN into single $DW_{3\times 3}$ . The residual connection is substituted by the identity weight with the dimension similar like $DW_{1\times 1}$ weight.

Component	MHSA	MHLA (v1)	Lite MHLA (v2)
Q,K,V + Out Proj.	$4NC^{2}$	–	–
Token Interaction	$2N^{2}C$	$2(NrN)C$	$N^{2}C$
	( $QK^{T}$ and $Attn\cdot V$ )	(MLP)	(Linear Proj.)
Nonlinearity	softmax	GELU	-
Normalization	LayerNorm	LayerNorm	Affine Transf.
Total	$4NC^{2}+2N^{2}C$	$2(NrN)C$	$\approx N^{2}C+\epsilon$

TABLE II: Complexity comparison of MHSA, MHLA, and Lite MHLA

Stage	Layer	Res.	Kernel	$n$	FaceLiVTv2 Channels
Stage	Layer	Res.	Size	$n$	XS	S	M	L
Input	Image	$112^{2}$	-	1	-	-	-	-
Stem	Conv	$56^{2}$	3 $\times$ 3	1	16	24	28	32
Stem	Conv	$28^{2}$	3 $\times$ 3	1	32	48	56	64
Stage-1	RepMix Enc.	$28^{2}$	3 $\times$ 3	3	32	48	56	64
Stage-1	Downsampling	$14^{2}$	3 $\times$ 3	1	64	96	112	128

TABLE III: Architecture Details of FaceLiVTv2 Variants.

Model	Year	Param	FLOPs	Training	LFW $\uparrow$	CA-	CP-	CFP	Age	Mean	IJB (FAR@ $10^{-4}$ ) $\uparrow$		Latency
Model	Year	(M) $\downarrow$	(M) $\downarrow$	Dataset	LFW $\uparrow$	LFW $\uparrow$	LFW $\uparrow$	-FP $\uparrow$	DB30 $\uparrow$	Acc(%) $\uparrow$	B	C	( $ms$ ) $\downarrow$
Large-Scale FR Models (FLOPs $\geq$ 1G)
ResNet200-TopoFR [ 27 ]	’24	118.8	23.5G	Glint360K	99.87	-	-	99.45	98.82	-	97.84	96.56	12.24
TransFace-L [ 34 , 10 ]	’25	271.6	25.4G	Glint360K	99.85	-	-	99.37	98.66	-	96.73	97.85	OOM
TransFace-B [ 34 , 10 ]	’25	124.5	11.5G	Glint360K	99.85	-	-	99.24	98.62	-	96.46	97.59	18.20
TransFace-S [ 34 , 10 ]	’25	86.7	5.8G	Glint360K	99.85	-	-	98.96	98.56	-	96.12	97.45	14.31

TABLE IV: Comparison of FaceLiVTv2 variant with SOTA on FR benchmarks. Mobile latency are measured on iPhone 15 Pro.

Fig. 6: Model in CoreML runtime breakdown. Operations highlighted in the redbox represent memory-bound operations, where memory access dominates execution time over computation.

Model	Mean	ONNX Latency (ms) $\downarrow$
Model	Acc(%) $\uparrow$	GPU	CPU	Jetson-(GPU/CPU)
KANFace-0.5 [ 5 ]	96.63	1.96	6.19	7.12 / 20.47
FaceLiVTv1-M [ 15 , 16 ]	96.25	1.83	3.61	7.04 / 13.65
EdgeFace-S [ 3 ]	96.16	1.84	2.90	6.85 / 11.16
SwiftFaceFormer-L1 [ 4 ]	95.83	1.36	4.51	5.53 / 10.85
SwiftFaceFormer-S [ 4 ]	95.74	1.28	2.53	5.28 / 8.18

TABLE V: Cross-Platform Latency Comparison of FaceLiVTv2 with SOTA

Model	FLOP	Rank
Model	(M) $\downarrow$	1	5	20	50
VGGFace [ 30 ]	-	30.4	-	40.4	42.7
CenterFace [ 30 ]	-	32.1	-	44.5	48.4
MobileFaceNet [ 19 ] from [ 18 ]	440	42.3	44.6	47.1	50.2
ShuffleFaceNet-1.5 [ 33 , 19 ] from [ 18 ]	577	50.2	52.4	54.5	55.3
VarGFaceNet [ 20 ] from [ 18 ]	1022	52.8	54.1	56.3	58.2

TABLE VI: Comparison of FaceLiVTv2 Variant with SOTA on TinyFace

Architectural	Param	FLOPs	LFW	CA-LFW	CP-LFW	CFP-FP	AgeDB-30	Mean Acc	Latency
Ablation	(M) $\downarrow$	(M) $\downarrow$	(%) $\uparrow$	(%) $\uparrow$	(%) $\uparrow$	(%) $\uparrow$	(%) $\uparrow$	(%) $\uparrow$	(ms) $\downarrow$
FaceLiVTv1-S (Baseline)	5.9	237	99.70	95.63	90.70	95.10	96.60	95.55	0.47
Adjust Dim $\uparrow$ , Depth $\uparrow$ , & MLP $\downarrow$	6.5 (10%) $\uparrow$	203 (14%) $\downarrow$	99.70 (+0.00)	95.63 (+0.00)	91.53 (+0.83)	95.73 (+0.63)	97.05 (+0.45)	95.92 (+0.37)	0.67 (42%) $\uparrow$
GAP $\rightarrow$ GDConv Recog. Head	7.4 (25%) $\uparrow$	210 (11%) $\downarrow$	99.72 (+0.02)	95.75 (+0.12)	91.20 (+0.50)	96.12 (+1.02)	97.03 (+0.43)	95.96 (+0.40)	0.73 (55%) $\uparrow$
Separate $\rightarrow$ Combined RepMix	7.4 (25%) $\uparrow$	211 (11%) $\downarrow$	99.73 (+0.03)	95.87 (+0.24)	91.22 (+0.52)	96.15 (+1.05)	97.23 (+0.63)	96.00 (+0.45)	0.74 (57%) $\uparrow$
MHLA $\rightarrow$ LiteMHLA	4.6 (22%) $\downarrow$	179 (24%) $\downarrow$	99.78 (+0.08)	95.93 (+0.30)	92.45 (+1.75)	97.47 (+2.37)	97.82 (+1.22)	96.69 (+1.14)	0.52 (10%) $\uparrow$

TABLE VII: Ablation On The Overall Impact Of Components On FaceLiVTv2-S Variant with Glint360K as a Training Dataset

Block Config.	Param	FLOPs	LFW	CA	CP	CFP	Age	Lat
[4-Stages]	(M)	(M)	LFW	LFW	LFW	FP	DB30	(ms)
[R, R, R, R]	4.52	175	99.73	95.88	92.07	97.19	97.67	0.50
[R, R, RL, RL]	4.62	179	99.78	95.93	92.45	97.47	97.82	0.54
[RL, RL, RL, RL]	12.47	279	99.80	95.99	92.54	97.57	97.83	1.36

TABLE VIII: Block Configurations Ablation of FaceLiVTv2-S

Fig. 7: Comparison of our proposed FaceLiVTv2 with other lightweight FR models on mean accuracy of FR benchmark dataset v.s. FLOPs and mean accuracy v.s i5-12500 CPU Latency.

Loss Function	LFW	CA	CP	CFP	Age
Loss Function	LFW	LFW	LFW	FP	DB30
ArcFace [ 17 ]	99.75	95.75	91.63	96.63	97.65
CosFace [ 23 ]	99.78	95.93	92.45	97.47	97.82

TABLE IX: Loss Function Ablation of FaceLiVTv2-S

Ablations	Param	FLOPs	LFW	CA	CP	CFP	Age	Lat
Ablations	(M)	(M)	LFW	LFW	LFW	FP	DB30	(ms)
Baseline	4.62	179	99.78	95.93	92.45	97.47	97.82	0.54
w/o Res Rep	4.62	179	99.78	95.93	92.45	97.47	97.82	0.63
w/o fused BN	4.65	183	99.78	95.93	92.45	97.47	97.82	0.62
w/o $DW_{1\times 1}$	4.62	179	99.72	95.65	92.23	97.40	97.68	0.54

TABLE X: RepMix Block Ablations of FaceLiVTv2-S

Normalization
Ablations	Param	FLOPs	LFW	CA	CP	CFP	Age	Lat
Ablations	(M)	(M)	LFW	LFW	LFW	FP	DB30	(ms)
Layer Norm	4.58	179	99.68	95.70	90.90	95.75	97.05	0.55
Affine	4.62	179	99.78	95.93	92.45	97.47	97.82	0.54
Activation Function
GELU	4.62	179	99.76	95.90	92.38	97.44	97.79	0.55

TABLE XI: Lite MHLA Ablations of FaceLiVTv2-S

Num	Param	FLOPs	LFW	CA	CP	CFP	Age	Lat
Head	(M)	(M)	LFW	LFW	LFW	FP	DB30	(ms)
1	4.55	179	99.70	95.87	91.88	97.20	97.75	0.49
2	4.58	179	99.75	95.77	92.07	97.36	97.72	0.52
2	4.58	179	(+0.05)	(-0.10)	(+0.19)	(+0.16)	(-0.03)	(6% $\uparrow$ )
3	4.59	179	99.75	95.80	92.77	97.36	97.97	0.53
3	4.59	179	(+0.05)	(-0.07)	(+0.89)	(+0.16)	(+0.22)	(8% $\uparrow$ )

TABLE XII: Number of Head Ablations of Lite MHLA On FaceLiVTv2-S

Fig. 8: Performance evolution for FaceLiVTv2-S across different number of head configuration on IJBB and IJBC Dataset

Num	IJB-B		IJB-C
Head	FAR@ $10^{-6}$	FAR@ $10^{-4}$	FAR@ $10^{-6}$	FAR@ $10^{-4}$
1	38.38	93.62	59.31	94.89
2	38.10 (-0.28)	93.89 (+0.27)	70.96 (+11.65)	95.50 (+0.61)
3	41.76 (+3.38)	93.96 (+0.34)	75.45 (+16.14)	95.56 (+0.67)
4	41.97 (+3.59)	94.51 (+0.89)	80.22 (+20.91)	95.99 (+1.10)
5	39.53 (+1.15)	94.26 (+0.64)	62.97 (+3.66)	95.67 (+0.78)

TABLE XIII: Performance evolution of FaceLiVTv2-S with different numbers of Lite MHLA heads on the IJB-B and IJB-C Test

Table 27

Hyperparameters	Config
optimizer	AdamW
learning rate	0.006
image resolution	112 $\times$ 112
batch size	384
LR schedule	polynomial
warmup epochs	0

TABLE XIV: FaceLiVT Training Hyperparameters and Augmentation on Glint360K datasets

Model	IJB-B			IJB-C
Model	$1e^{-5}$	$1e^{-4}$	$1e^{-3}$	$1e^{-5}$	$1e^{-4}$	$1e^{-3}$
MobileFaceNet [ 19 ]	87.9	92.8	95.6	92.2	94.7	96.6
ShuffleFaceNet-1.5 [ 19 ]	86.5	92.3	95.2	91.3	94.3	96.3
EdgeFace-S(0.5) [ 3 ]	-	93.6	-	-	95.6	-
ResNet50-ArcFace [ 17 ]	80.5	89.9	94.5	86.1	92.1	96.0
EdgeFace-XS(0.6) [ 3 ]	-	92.7	-	-	94.8	-

TABLE XV: Comparison of FaceLiVTv2 Variant with State-Of-The-Art on IJB-B and IJB-C Benchmark Dataset.

Fig. 9: Receiver Operating Characteristic of FaceLiVTv2 on IJBB and IJBC Dataset

Panel of Figure 31 — Fig. 10: Latency test of (1) FaceLiVTv2-XS, (2) FaceLiVTv2-S, (3) FaceLiVTv2-M, (4). FaceLiVTv2-L with XCode using iPhone 15 pro. All model is cverted from pytorch framework to coremltols.

Compute Operations
Category	Operation Type	Description and Runtime Characteristic
Tensor MatMul	Linear projections and matrix multiplications	Dominated by arithmetic FLOPs and executed on Tensor Cores or ALUs. These operations are highly compute-bound and critical in attention and fully connected layers.
convolution	fully, depthwise and pointwise convolutions	Performs spatial feature extraction using high multiply–accumulate (MAC) operations. Generally compute-intensive but cache-friendly and efficient on GPUs or NPUs.
Pooling	Global or average pooling operations	Aggregates spatial information into compact representations. Has moderate compute cost with minimal memory overhead.
Activation	Non-linear activation or attention weighting (GELU, ReLU, Softmax)	Introduces non-linearity or attention weighting in neural modules. Arithmetic-dominated and often fused into preceding compute kernels for efficiency.
Memory Operations

TABLE XVI: Operations Category by Runtime Type

Fig. 11: Layer distribution provided by XCode.

Model	Year	Param	FLOPs	Training	LFW $\uparrow$	CFP	Age	Mean	IJB (FAR@ $10^{-4}$ ) $\uparrow$		Latency
Model	Year	(M)	(M)	Dataset	LFW $\uparrow$	-FP $\uparrow$	DB30 $\uparrow$	Acc(%) $\uparrow$	B	C	( $ms$ ) $\downarrow$
Large-Scale FR Models (FLOPs $\geq$ 1G)
ResNet200-TopoFR [ 27 ]	’24	118.8	23.5G	Glint360K	99.87	99.45	98.82	99.38	97.84	96.56	12.24
TransFace-L [ 34 ]	’25	271.6	25.4G	Glint360K	99.85	99.37	98.66	99.29	96.73	97.85	OOM
TransFace-B [ 34 ]	’25	124.5	11.5G	Glint360K	99.85	99.24	98.62	99.24	96.46	97.59	18.20
TransFace-S [ 10 ]	’23	86.7	5.8G	Glint360K	99.85	98.91	98.50	99.09	96.05	97.33	14.31

TABLE XVII: Performance gap between lightweight FaceLiVTv2 variants and large-scale SOTA FR models. Mean Acc denotes the mean accuracy over LFW, CFP-FP, and AgeDB-30. FLOPs and mobile latency are measured at

112\times 112

input on iPhone 15 Pro.

Face Recognition

Face recognition matches or identifies people from facial images, videos, and biometric representations across changing conditions.

See all 108 papers

ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation
Jun 10, 2026Tahar Chettaoui, Guray Ozgur, Eduarda Caldeira +2Inference-Time Optimization Efficient ViTs
Fraunhofer Institute for Computer Graphics Research IGD, Germany · Department of Computer Science, TU Darmstadt, Germany
Lightweight Cross-Spectral Face Recognition via Contrastive Alignment and Distillation
May 6, 2026Anjith George, Sebastien MarcelFace Recognition
Idiap Research Institute, Martigny, Switzerland · Université de Lausanne (UNIL), Lausanne, Switzerland
Sustainable Face Recognition on Low-Power Devices with VQ-VAE Embeddings
Jun 13, 2026Christos Chronis, Georgios Th. Papadopoulos, Iraklis VarlamisEnergy-Efficient ML Edge Inference
Dep. of Informatics and Telematics Harokopio University of Athens Athens, Greece

Jun 10, 2026J/K move · Enter open · S save

ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation

Tahar Chettaoui, Guray Ozgur, Eduarda Caldeira, Naser Damer+1

Fraunhofer Institute for Computer Graphics Research IGD, Germany · Department of Computer Science, TU Darmstadt, Germany

Inference-Time Optimization Efficient ViTs Face Recognition Early-Exit Neural Networks

Open paper AskPDF arXiv

Vision Transformers (ViTs) have gained significant attention in computer vision and shown strong potential for face recognition (FR). However, their high computational cost makes deployment on resource-constrained devices challenging, motivating the need for methods that balance efficiency and accuracy. In this work, we investigate early exiting in pretrained ViTs as a simple yet effective training-free strategy for efficient FR inference. Leveraging the uniform feature dimensionality across transformer encoder blocks, we introduce ViT-FREE, a multi-exit framework that enables face verification directly from intermediate representations without modifying or retraining the backbone model, and thus, reducing inference cost. Empirically, we show that patch embeddings and attention maps evolve progressively across depth, exhibiting high similarity between consecutive ViT blocks and increasing alignment with the final representation. This indicates gradual feature refinement and attention convergence, suggesting that intermediate layers already provide stable and discriminative representations suitable for early exiting. Through extensive experiments on multiple FR benchmarks, we systematically analyze the accuracy-efficiency trade-off across exit depths. Our results demonstrate that later exits achieve a highly favorable balance, with exiting at layer 10 yielding up to a 20% speedup while incurring only a 1.5 drop in verification performance on benchmarks such as IJB-C. Also, we propose ViT-FREE_FT, a lightweight exit-specific fine-tuning strategy that adapts only the projection layers using a small synthetic dataset while keeping the transformer backbone frozen. This approach improves the performance of shallow exits while preserving the efficiency benefits and leaving deeper exits largely unaffected.

Abstract

Face Recognition

Explore similar work

ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation

Lightweight Cross-Spectral Face Recognition via Contrastive Alignment and Distillation

Sustainable Face Recognition on Low-Power Devices with VQ-VAE Embeddings