RCFormer: Reliability-aware Context Transformer for Occlusion-Robust Facial Landmark Detection

Ashish Kumar; Prashant Kumar; Mallika Garg

cs.CVNov 1, 2026

RCFormer: Reliability-aware Context Transformer for Occlusion-Robust Facial Landmark Detection

Authors: Ashish Kumar, Prashant Kumar, Mallika Garg

Organizations: Indian Institute of Technology Roorkee, Roorkee, INDIA · Afinit Co. Ltd, Seoul, Republic of Korea · Indian Institute of Technology Kharagpur, Kharagpur, INDIA

Abstract

Facial landmark detection has advanced substantially under regular imaging conditions, yet occlusion, poor illumination, and large pose changes remain major sources of error because local facial evidence can become incomplete or misleading. Context-guided latent patch recovery offers a promising way to address this issue by inferring unreliable patch representations from surrounding facial context before heatmap prediction. However, the recovered context can itself be unreliable when it is weakly adapted to the input face, influenced by corrupted neighboring patches, or blended without preserving local spatial coherence. We propose RCFormer, a Reliability-aware Context Transformer for occlusion-robust facial landmark detection. RCFormer strengthens latent patch recovery with three reliability-oriented components: visible-context conditioning (VCC), which adapts the context representation using a visibility-weighted global facial summary; visibility-guided context aggregation (VGCA), which reduces the influence of unreliable source patches during attention; and spatial residual correction (SRC), which refines blended latent features using neighboring patch information. The recovered heatmaps are then used as reliability-aware structural cues for existing landmark detection networks. Experiments show that RCFormer improves heatmap recovery and achieves strong landmark localization results on the WFLW, COFW, and 300W datasets.

Figures & tables

Figure 1: Overview of the quantized heatmap prior learning stage. A clean face image is encoded into latent features, quantized through a learned codebook, and decoded into an edge heatmap representation.

Figure 2: Architecture of RCFormer for reliable context-guided latent recovery. Given latent patches from an occluded input, RCFormer maintains regular and context token streams within an $L$ -layer reliability-aware recovery transformer. Visible-context conditioning (VCC) conditions context queries using reliable global evidence, while visibility-guided context aggregation (VGCA) suppresses unreliable source patches during context aggregation. The final regular and context tokens predict code sequences $\mathbf{S}_{I}$ and $\mathbf{S}_{C}$ , which are mapped through the fixed codebook to obtain $\mathbf{Z}_{I}$ and $\mathbf{Z}_{C}$ . The final unreliability map $\boldsymbol{\alpha}$ guides patch-wise blending and gates spatial residual correction (SRC) to produce $\mathbf{Z}_{rec}$ . The fixed decoder maps $\mathbf{Z}_{rec}$ to the recovered heatmap $\mathbf{H}_{rec}$ . For downstream FLD, $\mathbf{H}_{rec}$ is concatenated with an early-stage detector feature map and fused using a lightweight $1\times 1$ convolutional block before landmark prediction.

Original WFLW bounding boxes
Method	Backbone	WFLW-Full			COFW	300W (NME $\downarrow$ )
Method	Backbone	NME $\downarrow$	FR $\downarrow$	AUC $\uparrow$	NME $\downarrow$	Full	Comm.	Chal.
LAB [ 28 ]	Hourglass	5.27	7.56	0.532	-	3.49	2.98	5.19
Wing [ 4 ]	ResNet-50	4.99	6.00	0.550	5.44	-	-	-
AWing [ 27 ]	Hourglass	4.36	2.84	0.572	4.94	3.07	2.72	4.52
PIPNet [ 8 ]	ResNet-101	4.31	-	-	-	3.19	2.78	4.89

Table 1: Comparison with baseline FLD methods. NME is reported for all datasets; FR and AUC at the 10% threshold are reported for WFLW. WFLW methods are grouped by bounding-box protocol, with direct comparisons and highlighting restricted to the HRNet-based protocol. Best results are shown in bold and second-best results are underlined.

\dagger

denotes baseline results sourced from ORFormer [ 3 ] .

Figure 3: Qualitative landmark comparison on challenging WFLW samples. Blue points denote ground-truth landmarks, red points denote predicted landmarks, and green line segments show the displacement between matched landmark pairs. Yellow ellipses highlight difficult regions where RCFormer produces shorter landmark-error lines than STAR.

Original WFLW bounding boxes
Method	Backbone	Full	Pose	Exp.	Ill.	Mak.	Occ.	Blur
Method	Backbone	NME $\downarrow$
LAB [ 28 ]	Hourglass	5.27	10.24	5.51	5.23	5.15	6.79	6.12
Wing [ 4 ]	ResNet-50	4.99	8.43	5.21	4.88	5.26	6.21	5.81
AWing [ 27 ]	Hourglass	4.36	7.38	4.58	4.32	4.27	5.19	4.96
PIPNet [ 8 ]	ResNet-101	4.31	7.51	4.44	4.19	4.02	5.36	5.02

Table 2: WFLW subset evaluation. NME is reported on the full test set and six challenging subsets. Methods are grouped by bounding-box protocol, with direct comparisons and highlighting restricted to the HRNet-based protocol. Best results are shown in bold and second-best results are underlined.

\dagger

denotes baseline results sourced from ORFormer [ 3 ] .

Architecture	WFLW							COFW	300W
Architecture	Full	Pose	Exp.	Ill.	Mak.	Occ.	Blur	Full	Full
Heatmap L2 Loss $\downarrow$
VQVAE	29.38	59.37	32.64	28.45	33.25	40.12	34.23	16.57	25.97
ORFormer	22.29	43.25	26.49	22.07	23.31	28.48	27.05	12.80	16.58
RCFormer	15.22	25.41	18.46	16.56	16.75	19.12	17.20	8.39	12.59
Rel. gain vs. ORFormer	31.7%	41.2%	30.3%	25.0%	28.1%	32.9%	36.4%	34.5%	24.1%

Table 3: Heatmap generation comparison using L2 loss. All architectures are implemented under the same setting. Results are reported on WFLW and its six subsets, with full-set results on COFW and 300W. Bold percentages indicate relative L2 reduction of RCFormer over ORFormer.

Figure 4: Predicted unreliability maps from RCFormer. Red regions indicate higher unreliability, highlighting occlusions, low-visibility areas, and ambiguous facial boundaries.

Figure 5: Edge heatmap outputs on WFLW samples. Ground truth (GT) heatmaps are derived from landmark-based facial edge annotations, with brighter regions indicating stronger responses. Values denote per-sample L2 errors against GT.

Method	Components			WFLW
Method	VCC	VGCA	SRC	L2 Loss $\downarrow$	NME $\downarrow$
Base	–	–	–	22.05	3.96
Base + VCC	✓	–	–	21.08	3.90
Base + VCC + VGCA	✓	✓	–	20.14	3.86
Base + SRC	–	–	✓	18.46	3.89
RCFormer	✓	✓	✓	15.22	3.73

Table 4: Component ablation on WFLW. VCC, VGCA, and SRC denote visible-context conditioning, visibility-guided context aggregation, and spatial residual correction, respectively.

Method	Params (M)	FLOPs (G)	Latency (ms)	NME $\downarrow$
STAR	17.177	34.796	34.154	4.03
ORFormer	22.038	36.260	37.033	3.86
RCFormer	22.699	36.264	37.931	3.73

Table 5: Inference efficiency on WFLW measured on an NVIDIA RTX 4090 with batch size 1 and FP32 precision.

Facial Landmark Localization

See all 8 papers

FAHCD-Net: Frequency-Adaptive Heatmap-Conditional Diffusion Networks for Robust Facial Landmark Detection
Sep 15, 2026Jun Wan, Jiwei Hu, Shengkai Hu +1Facial Landmark Localization Frequency-Domain Feature Learning
School of Information Engineering, Zhongnan University of Economics and Law, Wuhan 430073, China
Towards Unified Dynamic Face Landmark Detection
Aug 11, 2026Sebastian Regalado, Varshanth R. Rao, Ruowei Jiang +2Facial Landmark Localization
University of Toronto · 2ModiFace · †Work completed while employed at ModiFace +1
PLGSA-Transformer: Periocular Landmark-Guided Attention with Occlusion-Adaptive Cosine Thresholding for Cross-Modal Masked and Unmasked Face Recognition
Jul 3, 2026Dana A AbdullahCross-Modal Feature Matching Face Recognition
Faculty of Engineering and Computer Science, Qaiwan International University, Sulaymaniyah, KRG, Iraq. · Department of Information and Communication Technology Center (ICTC)-System Information, Ministry of Higher Education and Scientific Research.

Sep 15, 2026J/K move · Enter open · S save

FAHCD-Net: Frequency-Adaptive Heatmap-Conditional Diffusion Networks for Robust Facial Landmark Detection

Jun Wan, Jiwei Hu, Shengkai Hu, Qilu Zhu

School of Information Engineering, Zhongnan University of Economics and Law, Wuhan 430073, China

Facial Landmark Localization Frequency-Domain Feature Learning Conditional Diffusion Models

Open paper AskPDF arXiv

Facial Landmark Detection(FLD) is a crucial task in various applications and has achieved significant advancements in recent years. However, current FLD methods still struggle under challenging conditions, where facial structural variations, information loss, and noise interference severely compromise the integrity and accuracy of learned facial features. To address these issues, we propose Frequency-Adaptive Heatmap-Conditional Diffusion Network (FAHCD-Net), which integrates a Frequency-Adaptive Heatmap-Conditional Diffusion (FAHCD) model with a Smoothness Regularization (SR) loss in a cascaded framework. Specifically, the FAHCD model incorporates a Hierarchical Frequency Adaptation (HFA) module designed to suppress redundant high-frequency noise through multi-layer frequency decomposition and adaptive reconstruction, thereby preserving essential facial structures. Additionally, the SR loss is proposed to further mitigate the interference of high-frequency noise and enhance the smoothness of the generated landmark heatmaps. By cascading the FAHCD model with the SR loss, FAHCD-Net effectively leverages both statistical and frequency-based distribution characteristics of the data to progressively generate more accurate landmark heatmaps from noisy inputs. Extensive experiments on popular benchmarks demonstrate the effectiveness and robustness of the proposed method, achieving state-of-the-art performance in FLD tasks under challenging scenarios. The source code is available at https://github.com/HJWKryptonite/FAHCD-Net.

Abstract

Facial Landmark Localization

Explore similar work

FAHCD-Net: Frequency-Adaptive Heatmap-Conditional Diffusion Networks for Robust Facial Landmark Detection

Towards Unified Dynamic Face Landmark Detection

PLGSA-Transformer: Periocular Landmark-Guided Attention with Occlusion-Adaptive Cosine Thresholding for Cross-Modal Masked and Unmasked Face Recognition