AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relations

Ying Huang; Wencan Zhang; Brian Y. Lim

cs.MMAug 14, 2026

AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relations

Authors: Ying Huang, Wencan Zhang, Brian Y. Lim

Organizations: Department of Computer Science, National University of Singapore Singapore, Singapore

Abstract

Computer vision models for generated facial content, such as face editing and privacy protection, increasingly affect people, requiring similarity metrics that serve as faithful proxies for human perception. While perceptual evaluation has progressed from signal-based heuristics to representation-based metrics, current approaches are limited to behavioral modeling without cognitive alignment. They rely on implicit and spurious relations while assuming a universal observer, failing to account for inherent variations across diverse human populations. This leads to inaccurate evaluative models of stakeholders and misleading guidance for generative model debugging. Rather than treating perception as a black box, we leverage scientific findings from cognitive psychology of human face similarity perception: dependence on facial featural and configural attributes, nonlinear psychophysical response scaling, and own-group biases. We introduce the FACETS dataset and propose AlignFace, an interpretable, human-aligned, face similarity metric that encodes these cognitive principles through ante-hoc modeling. It employs visual-language modeling (VLM) to encode paired face images and text-based attributes, gated cross-attention (CA) to extract attribute-specific facial difference representations, concept bottleneck modeling (CBM) to constrain reasoning via interpretable face attributes, and neural generalized additive model (GAM) to model their nonlinear influence. Experiments found AlignFace significantly improves alignment with human subpopulation perceptions compared to baseline metrics, including recent domain-free learned perceptual metrics. By bridging learned representations and human cognitive processes, this work enables more transparent and aligned perceptual evaluation metrics for face images.

Figures & tables

Figure 1. a) AlignFace provides a face-pair similarity score that better matches human perception. b) It decomposes face similarity into attribute-level comparisons, capturing featural (e.g., lip color) and configural (e.g., pupillary distance) cues for fine-grained explanations. c) Decisions are explained via learned nonlinear functions that map attribute similarity to overall face similarity, while capturing group-dependent effects (own- vs. other- group). d) AlignFace outperforms metrics like LPIPS and SSIM in triplet comparisons—whether A or B is more similar to a reference—improving alignment with human perception. A description for accessibility. This should describe what is shown in the figure.

Category	Attribute	Scale (0–1)
Featural	Eyebrow shape	Rounded–Straight
	Eyebrow thickness	Thin–Thick
	Eye shape	Narrow–Round
	Eye size	Small–Large
	Eye color	Light–Dark
	Nose shape	Pointed–Flat

Table 1. Cognitively-grounded featural and configural attributes of human face perception adapted 1 from ( Abudarham and Yovel, 2016 ) .

Figure 2. Experiment apparatus of the 2AFC protocol for two user studies on [Face] overall similarity and per-[Attribute] similarity, with different survey questions based on the task.

Figure 3. Partial dependence plots of overall relative face selection by the 8 most salient attributes, split by own- and other-group. See Appendix Fig. 12 for full attributes.

Figure 4. AlignFace architecture. 1) vision-language model (VLM) encoding of face-pair images and text-based attributes into a shared representation, 2) attribute-gated cross-attention (CA) to extract attribute-specific visual features, 3) face-attribute concept bottleneck model (CBM) to constrain reasoning through disentangled face attributes, and 4) Attribute-Influence neural Generalized Additive Model (GAM) to represent each attribute’s nonlinear, additive contribution to the pairwise distance $\hat{d}$ .

Figure 5. AlignFace2AFC metric using AlignFace ( $M_{\text{AF}}$ ) with a triplet instance $\langle x_{A},x_{\text{Ref}},x_{B}\rangle$ to predict two sets of overall and attribute distances and compute their differences ( $\Delta\hat{d}$ , $\Delta\Delta\hat{\bm{c}}$ ). Differences are learned via supervised training with binary 2AFC human labels of the overall face $y_{d}$ or of all attributes $\bm{y}_{c}$ . See $y_{d}$ – $\bm{y}_{c}$ GAM-based relations in Fig. 3 .

Figure 6. Model-human agreement on overall face perception. The grey line represents random guessing. Error bars show 95% confidence intervals. Baselines without

\checkmark

are significantly lower than AF-FLIP (Dunnett’s test,

\alpha=0.05

Figure 7. Attribute perception agreement of a) averaged across all attributes, b) grouped into featural and configural categories, and c) at individual-level attributes.

Figure 8. Partial dependence plots (PDPs) of selected attributes, comparing human judgments with Linear, MLP and nGAM. The numbers in graphs are Pearson correlations $\rho$ between each model and humans. High correlations for nGAM (0.94–0.98) indicate strong human alignment of AlignFace.

Figure 9. Cross-group perception agreement evaluations on a) face and b) averaged attributes.

Setting	AF-CLIP		AF-FLIP
Setting	Face	Attr	Face	Attr
Within-dataset	79.9%	77.9%	81.3%	78.3%
Cross-dataset	79.4%	72.9%	81.3%	73.2%

Table 2. Agreement under within- and cross-dataset evaluation for AF-CLIP and AF-FLIP. Consistent performance indicates learned generalized perception across datasets.

Figure 10. Results of ablation studies on Attribute-Gated Cross-Attention: a) overall face model-human agreement, b) average attribute agreement; and on Attribute-Influence Neural GAM: c) overall face agreement.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Ethnicity	White		Asian
Gender	Female	Male	Female	Male
Multi-PIE	20	20	10	10
CelebA	20	20	10	10

Appendix

Table 3. Demographic distribution of face targets selected during dataset curation.

Figure 11. Statistical distribution of feature values for source and edited faces across 20 face attributes. These feature values are computed using geometry-based heuristic methods and serve as proxy evaluation metrics.

	Ethnicity					Gender			Age
	White	Asian	Black	Hispanic	Other	Female	Male	Other	Range	Median
Face perception	35	35	2	2	4	38	38	2	21-78	44.0
Attribute perception	342	187	53	9	20	324	286	1	21-84	44.0

Appendix

Table 4. Demographic statistics of participants in our psychophysical similarity measurement studies.

Attribute	Fleiss’ $\kappa$	Agreement
Overall	0.604	Moderate
Eyebrow shape	0.718	Substantial
Eyebrow thickness	0.703	Substantial
Eye shape	0.560	Moderate
Eye size	0.605	Moderate
Eye color	0.705	Substantial

Appendix

Table 5. Inter-rater agreement (Fleiss’

\kappa

) ( Landis and Koch, 1977 ) for overall face and 20 facial attributes.

Method	Comparison	Diff.	$p$ -value
PSNR	AF-FLIP	-0.184642	$<.0001$
SSIM	AF-FLIP	-0.171484	$<.0001$
SimCLR	AF-FLIP	-0.210958	$<.0001$
MoCo	AF-FLIP	-0.145168	$<.0001$
DINO	AF-FLIP	-0.145168	$<.0001$
LPIPS	AF-FLIP	-0.118852	0.0010

Appendix

Table 6. Dunnett’s test comparing AF-FLIP against baselines (

\alpha=0.05

Attribute	Own-group	Other-group
Eyebrow shape	0.035	0.033
Eyebrow thickness	0.027	0.029
Eye shape	0.043	0.079
Eye size	0.097	0.028
Eye color	0.076	0.056
Nose shape	0.020	0.063

Appendix

Table 7. Attribute importance and

R^{2}

of GAM models fitted to Own-group and Other-group human judgments.

Module	Alternative	Justification w.r.t. alternative (Benefit)
Vision-Language Model	Feature Engineering	More scalable: support open-domain (label-free) attributes instead of hard-coded features.
Attr-Gated Cross-Attention	Feature Concatenation	Interpretability and dimensionality reduction: constrain reasoning via attention to focus on information most relevant to each attribute, rather than finding relationships across the full face pair without differencing or focusing.
Concept Bottleneck Model	MLP	Interpretability: regularizes reasoning with domain-specific attributes, avoiding spurious features.
Neural GAM	MLP	Interpretability: regularizes reasoning independently and nonlinearly for each attribute, avoiding spurious relations.

Appendix

Table 8. Justifications for AlignFace modules compared to standard alternatives.

Category	Attribute	Pearson correlation $\rho$
Featural	Eyebrow thickness	0.916
	Eye size	0.970
	Nose size	0.969
	Mouth size	0.840
	Lip thickness	0.989
Configural	Face aspect ratio	0.934

Appendix

Table 9. Pearson correlation

\rho

between heuristic physical measurements and model-predicted attribute distances on single-attribute manipulations.

Baseline	Backbone	Pretrained
PSNR/SSIM	—	—
SimCLR	ResNet-50	ImageNet-1K
MoCo	ResNet-50	ImageNet-1K
DINOv2	ViT-B/32	LVD-142M
CLIP	ViT-B/32	WIT-400M
FLIP	ViT-B/32	FLIP-80M

Appendix

Table 10. Baseline implementation details.

Figure 12. Partial dependence plots of overall relative face selection by relative attribute selection across 20 attributes.

Figure 13. Partial dependence plots (PDPs) of overall relative face distance difference by relative attribute distance difference across 20 attributes. Numbers denote the Pearson correlation between human judgments and comparators.

Figure 14. Cross-dataset perception agreement for a) attribute categories and b) individual attributes.

Figure 15. Cross-group attribute perception agreement for a) attribute categories and b) individual attributes.

May 12, 2026

Human face perception reflects inverse-generative and naturalistic discriminative objectives

The perceptual representations supporting our ability to recognize faces remain a computational mystery. Deep neural networks offer mechanistic hypotheses for human face perception, but theoretically distinct models often make indistinguishable representational predictions for randomly sampled faces. To expose diagnostic differences among these hypotheses, we compared six neural network models sharing an architecture but trained on distinct tasks, using face pairs optimized to elicit contrasting model predictions ("controversial" pairs) alongside randomly sampled pairs. We tested model predictions against face-dissimilarity judgments from 864 human participants across stimulus sets differing in realism and pose variation. Models prioritizing high-level, invariant structures (trained via inverse rendering, face identification, or object classification) most robustly matched human judgments. Furthermore, models trained on natural images typically outperformed synthetic-trained counterparts. Together, these findings suggest that human face perception is shaped by mechanisms that infer latent causes of facial appearance, discount nuisance variation, and are tuned by natural image statistics.

Wenxuan Guo, Heiko H. Schütt, Kamila Maria Jozwik +3

Department of Psychology, Columbia University, New York, NY, USA. · Department of Behavioural and Cognitive Sciences, Université du Luxembourg, Esch-sur-Alzette, Luxembourg. · MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, England. +4

Face Recognition Human Visual Perception

Ask PDF

May 22, 2026

Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot

A central question in computational vision is whether human-like visual representations are better explained by discriminative or generative learning. Existing comparisons, however, often confound the learning objective with architecture, scale, and training data, leaving open whether the objective itself drives alignment. We address this confound using Joint Energy-Based Models (JEMs), which interpolate continuously between discriminative and generative training within a fixed architecture. By varying a single mixing coefficient, we isolate the effect of the learning objective and evaluate the resulting models across six human-alignment benchmarks spanning perceptual similarity, gloss perception, human response uncertainty, robustness, shape-texture cue conflict, and diagnostic feature attribution. Across this diverse suite, human alignment is consistently maximized at intermediate points of the generative-discriminative continuum, rather than at either endpoint. Hybrid JEMs combine the categorical structure induced by discriminative learning with the sensitivity to input structure induced by generative learning, yielding more human-like behavior across multiple levels of vision. These results suggest that the generative-discriminative dichotomy is the wrong axis for understanding human-aligned vision: alignment emerges not from choosing one objective over the other, but from balancing both.

Jorge Chang Ortega, Bastien Le Lan, Thomas Serre +1

ANITI · Brown University, ANITI · CNRS, ANITI

Visual Representation Learning Energy-Based Models

Ask PDF

Jun 26, 2026

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 10,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.

Yana Wei, Hongbo Peng, Yanlin Lai +14

Johns Hopkins University · StepFun · Tsinghua University +1

Rubric-Based Evaluation Automated Evaluation Multimodal Model Evaluation Fine-Grained Visual Perception

Ask PDF

Abstract

Explore similar work

Human face perception reflects inverse-generative and naturalistic discriminative objectives

Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception