DIME: Scaling Facial Representation Learning via Differential Masked Autoencoding

Hao Yu; Haoyu Chen; Hui Wei; Yan Jiang; Nicu Sebe; Guoying Zhao

cs.CVOct 12, 2026

DIME: Scaling Facial Representation Learning via Differential Masked Autoencoding

Authors: Hao Yu, Haoyu Chen, Hui Wei, Yan Jiang, Nicu Sebe, Guoying Zhao

Organizations: ELLIS Institute Finland and CMVS, University of Oulu, Finland · DISI, University of Trento, Italy

Abstract

Scaling up data and model capacity has driven substantial progress in visual representation learning. However, conventional pretext tasks exhibit diminishing returns on fine-grained facial analysis. We attribute this to a scaling bottleneck: these tasks model each image's appearance and are dominated by identity, pose, and illumination, so subtle within-identity facial differences contribute little to the training signal. We introduce DIME, a differential masked autoencoder for large-scale facial representation learning. DIME builds on a simple observation: two images of the same person share identity-specific appearance, so their difference exposes facial changes. DIME reconstructs both masked images in an identity-matched pair and, through an Explicit Differential Direction Supervision (EDDS) objective, aligns the direction of their reconstructed difference with the true difference, while source isolation makes each reconstruction depend only on its own image. Shared appearance cancels in this difference, and because EDDS ignores magnitude, subtle facial changes count as much as large pose or illumination changes. DIME uses identity labels only to form pretraining pairs, requires no dense annotations, and learns dense facial representations with emergent part-level semantics. We pretrain DIME on 20 million high-resolution facial images consolidated from existing face datasets. DIME's AU detection performance continues to improve as pretraining data scales from 6M to 20M images, with its BP4D F1 increasing by 5.9 points, while MAE gains only 0.5 points on the same subsets. DIME also outperforms specialized facial methods on AU detection, AU intensity estimation, cross-domain anti-spoofing, landmark localization, head pose estimation, face parsing, and face generation.

Figures & tables

Fig. 2 : Emergent capacity of DIME. (a) Cosine similarity between a marked query token (red cross) and all patches, demonstrating that DIME learns robust part-level semantic consistency across distinct facial parts (brighter colors indicate higher similarity). (b) UMAP-PCA [ 12 ] visualizations of dense features from DIME-L, DINOv3-L [ 13 ] , and MAE-L [ 3 ] . DIME exhibits more semantically differentiated and structurally aligned facial representations.

Fig. 3 : Overall architecture of DIME. For simplicity, we illustrate one mixed token grid. In practice, the two reciprocal mixed grids and their complementary decoder branches are batched and processed in parallel to improve efficiency.

Fig. 4 : EDDS in masked image modeling. (a) From the context $\mathcal{V}_{\ell}$ (dots) of two complementary masked views, the model infers the direction of change $\boldsymbol{\delta}_{\ell}$ at the hidden patch $\ell$ . EDDS scores a prediction by $1-\cos\theta_{\ell}$ , so the optimum points along the posterior mean $\mathbf{u}_{\ell}$ , whose length reflects predictability. (b) Under MSE, predicting no change costs $\propto\|\boldsymbol{\delta}_{\ell}\|^{2}$ , which nearly vanishes for subtle changes such as AUs; under EDDS it costs 1 at every supported patch, whatever the magnitude, recovering the lost signal.

Model	Backbone	Embed. dim	Depth	Decoder dim/depth	Params
DIME-S	ViT-S/16	384	12	$384/6$	$23$ M
DIME-B	ViT-B/16	768	12	$512/8$	$86$ M
DIME-L	ViT-L/16	1024	24	$512/8$	$307$ M

TABLE I : Architectures of DIME. The decoder column reports its embedding dimension and block depth. The Params column counts the encoder only; the decoder and prediction heads, used only in pretraining, are excluded.

Method	AU1	AU2	AU4	AU6	AU7	AU10	AU12	AU14	AU15	AU17	AU23	AU24	Avg. $\uparrow$	SD $\downarrow$
CISNet [ 48 ]	54.8	48.3	57.2	76.2	76.5	85.2	87.2	66.2	50.9	65.0	47.7	56.5	64.3	13.4
D-PAttNet [ 49 ]	50.7	42.5	59.0	79.4	79.0	85.0	89.3	67.6	51.6	65.3	49.6	54.5	64.5	14.9
CaFNet [ 50 ]	55.1	49.3	57.7	78.3	78.6	85.1	86.2	67.4	52.0	64.4	48.3	56.2	64.9	13.4
ME-GraphAU [ 51 ]	52.7	44.3	60.9	79.9	80.1	85.3	89.2	69.4	55.4	64.4	49.8	55.1	65.5	14.4
MAE-Face [ 6 ]	65.2	55.7	66.9	79.5	79.5	85.2	90.0	67.5	59.7	67.5	54.8	60.9	69.4	11.1
FMAE [ 26 ]	59.2	50.0	62.7	80.0	79.2	84.7	89.8	63.5	52.8	65.1	55.3	56.9	66.6	12.8

TABLE II : Per-AU F1 scores (%) on BP4D [ 1 ] . Best and second-best results are in vermilion and deep blue , respectively.

Method	AU1	AU2	AU4	AU6	AU7	AU10	AU12	AU14	AU15	AU17	AU23	AU24	Avg. $\uparrow$	SD $\downarrow$
FACS3D-Net [ 58 ]	43.0	38.1	49.9	82.3	85.1	87.2	87.5	66.0	48.4	47.4	50.0	31.9	59.7	19.8
ML-GCN [ 59 ]	40.2	36.9	32.5	84.8	88.9	89.6	89.3	81.2	53.3	43.1	55.9	28.3	60.3	23.6
MS-CAM [ 60 ]	38.3	37.6	25.0	85.0	90.9	90.9	89.0	81.5	60.9	40.6	58.2	28.0	60.5	24.9
SEV-Net [ 61 ]	47.9	40.8	31.2	86.9	87.5	89.7	88.9	82.6	39.9	55.6	59.4	27.1	61.5	23.3
MAE-Face [ 6 ]	53.0	44.5	49.4	87.7	88.8	91.4	90.8	83.2	45.2	50.0	61.2	45.3	65.9	19.6
FMAE [ 26 ]	53.9	45.5	45.9	86.2	88.3	91.2	89.9	82.3	51.3	56.3	60.7	42.7	66.2	18.8

TABLE III : Per-AU F1 scores (%) on BP4D+ [ 2 ] . Best and second-best results are in vermilion and deep blue , respectively.

Table 7

Fig. 5 : Per-AU F1 scores on BP4D (left), BP4D+ (middle), and DISFA (right). Dashed boxes highlight groups containing challenging AUs, where DIME improves detection beyond generic visual encoders and prior facial self-supervised models.

	$ICM\!\rightarrow\!O$		$OCM\!\rightarrow\!I$		$OCI\!\rightarrow\!M$		$OMI\!\rightarrow\!C$		Average
Method	HTER $\downarrow$	AUC $\uparrow$	HTER $\downarrow$	AUC $\uparrow$	HTER $\downarrow$	AUC $\uparrow$	HTER $\downarrow$	AUC $\uparrow$	HTER $\downarrow$	AUC $\uparrow$
LBP-TOP [ 7 ]	30.17	77.61	29.14	71.69	36.90	70.80	33.52	73.15	32.43	73.31
SA-FAS [ 68 ]	10.00	96.23	6.58	97.54	5.95	96.55	8.78	95.37	7.83	96.42
PDLE [ 69 ]	11.42	95.52	8.62	97.60	5.41	98.85	10.05	94.27	8.88	96.56
UDG-FAS [ 70 ]	10.97	95.36	5.86	98.62	5.95	98.47	9.82	96.76	8.15	97.30
GAC-FAS [ 71 ]	8.60	97.16	4.29	98.87	5.00	97.56	8.20	95.16	6.52	97.19

TABLE VI : Cross-domain face anti-spoofing results. The generic visual encoders and DIME variants are evaluated using a unified downstream protocol [ 9 ] . Best and second-best results are highlighted in vermilion and deep blue , respectively.

	WFLW [ 79 ]
Method	NME $\downarrow$	FR $\downarrow$	AUC $\uparrow$
HRNet-W18 [ 80 ]	4.60	–	–
STAR [ 76 ]	4.02	2.32	60.50
ADNet [ 81 ]	4.14	2.72	60.22
LDEQ [ 82 ]	3.92	2.48	62.40
Faceptor-Full [ 83 ]	4.03	1.92	60.24

TABLE VII : Facial geometry estimation: sparse geometry (landmark detection), global orientation (head pose estimation), and dense semantic structure (face parsing). Best and second-best results are in vermilion and deep blue .

Fig. 6 : Selected face images generated with the DIME-L encoder in the RAEv2 framework with 100K training steps.

		RAEv2 [ 99 ]		REPA [ 100 ]
Rep. encoder	Steps	256/FID	512/FID	256/FID	512/FID
DINOv3-L [ 13 ]	100K	12.48	14.83	13.46	16.15
DINOv3-L [ 13 ]	200K	9.83 $\downarrow$ 2.65	11.53 $\downarrow$ 3.30	10.61 $\downarrow$ 2.85	11.80 $\downarrow$ 4.35
DINOv3-L [ 13 ]	400K	8.53 $\downarrow$ 3.95	8.94 $\downarrow$ 5.89	8.22 $\downarrow$ 5.24	8.36 $\downarrow$ 7.79
DINOv2-L [ 16 ]	100K	14.83 $\uparrow$ 2.35	16.79 $\uparrow$ 1.96	15.31 $\uparrow$ 1.85	18.82 $\uparrow$ 2.67
MAE-L [ 3 ]	100K	16.72 $\uparrow$ 4.24	18.22 $\uparrow$ 3.39	18.34 $\uparrow$ 4.88	21.31 $\uparrow$ 5.16

TABLE VIII : Face-generation FID [ 101 ] (lower is better) on FFHQ at

256{\times}256

and

512{\times}512

under RAEv2 [ 99 ] and REPA [ 100 ] . Gray arrows give the FID change relative to DINOv3-L at 100K steps. Best and second-best results are in vermilion and deep blue , respectively. Steps: training steps.

Fig. 7 : Face-generation dynamics on FFHQ. (a) Training loss and FID of RAEv2 using MAE, DINOv3, and DIME encoders. DIME consistently converges faster and achieves lower FID. (b) Generated samples from 1K to 50K training steps using DIME (top) and DINOv3 (bottom). DIME produces coherent facial structure earlier and resolves fine details sooner, consistent with its faster FID reduction.

	Transfer
Unit	F1 $\uparrow$	ICC $\uparrow$	NME $\downarrow$	HTER $\downarrow$
Single (MAE [ 3 ] )	61.9	.619	4.31	11.32
Same image	62.3	.623	4.29	10.91
Random ID	62.8	.630	4.23	10.38
Same ID	62.9	.630	4.22	10.32
Same–random	$+0.1$	$+.001$	$-.01$	$-0.06$

TABLE IX : Controlled ablation study of DIME. Panels (a)–(c) examine the pretraining unit, the interaction between identity matching and EDDS, and source isolation. Panels (d)–(f) evaluate complementary supervision, facial-change information retained in frozen features, and sensitivity to the source-mixing ratio. All variants: ViT-S/16, 100 epochs, 30% of Face-20M.

Visual Representation Learning

Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.

See all 224 papers

Emergent Region-Level Facial Correspondence in Frozen Vision Foundation Models
Jul 15, 2026Izaldein Al-Zyoud, Abdulmotaleb El SaddikSemantic Correspondence Vision Foundation Models
MCRLab, School of Electrical Engineering and Computer Science University of Ottawa, Ottawa, ON, Canada
FairReL: Deepfake Detection using Fairness-Aware Representation Learning
Aug 28, 2026Xiaoman Lu, Jiaqi Li, Shuntian Zheng +2Deepfake Detection Algorithmic Fairness
Department of Computer Science University of Warwick Coventry, United Kingdom · School of Computer Science University of Birmingham Birmingham, United Kingdom
DifFoundMAD: Foundation Models meet Differential Morphing Attack Detection
Apr 20, 2026Lazaro J. Gonzalez-Soler, André Dörsch, Christian Rathgeb +1Image Forgery Detection
da/sec - Biometrics and Security Research Group Darmstadt, Germany

Jul 15, 2026J/K move · Enter open · S save

Emergent Region-Level Facial Correspondence in Frozen Vision Foundation Models

Izaldein Al-Zyoud, Abdulmotaleb El Saddik

MCRLab, School of Electrical Engineering and Computer Science University of Ottawa, Ottawa, ON, Canada

Semantic Correspondence Vision Foundation Models Non-Rigid Shape Correspondence Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Frozen self-supervised vision models can align parts of generic objects, but it remains unclear whether this correspondence extends to human faces, where global layout is shared while identity-specific appearance varies sharply. We test whether frozen DINOv3 features define a region-level facial coordinate system: a feature space in which eyes, brows, nose, mouth, skin, and hair remain distinguishable across people and across time without face-specific training. Using DINOv3 ViT-L/16 patch embeddings and FaRL only as a face-part labeling interface, we evaluate cross-identity nearest-neighbor matching and temporal label propagation on 200 CelebDF-v2 real videos. DINOv3 achieves 83.0% region-level semantic accuracy under unconstrained cross-identity matching, compared with a 23.0% area-weighted random baseline, and 95.5% temporal tracking accuracy without a learned temporal module. A no-FaRL control collapses to 0.9%, showing that FaRL supplies semantic initialization while DINOv3 supplies dense spatial correspondence. The strongest correspondence appears at an intermediate layer: block 18 gives a 4.93x same-region versus cross-region discrimination ratio, compared with 1.48x at the final block. Against CLIP ViT-L/14, DINOv3 shows only a small aggregate advantage but a +16.8 pp gain on anatomical regions, indicating that image-level contrastive supervision captures coarse facial layout but not fine-grained anatomical identity. These results establish frozen DINOv3 as a strong zero-shot representation for region-level facial correspondence and identify intermediate self-supervised features as the most useful layer for dense face analysis.

Abstract

Visual Representation Learning

Explore similar work

Emergent Region-Level Facial Correspondence in Frozen Vision Foundation Models

FairReL: Deepfake Detection using Fairness-Aware Representation Learning

DifFoundMAD: Foundation Models meet Differential Morphing Attack Detection