True to Tone? Quantifying Skin Tone Fidelity and Bias in Photographic-to-Virtual Human Pipelines

Gabriel Ferri Schneider; Erick Menezes; Rafael Mecenas; Paulo Knob; Victor Araujo; Soraia Raupp Musse

cs.CVApr 2, 2026

True to Tone? Quantifying Skin Tone Fidelity and Bias in Photographic-to-Virtual Human Pipelines

Authors: Gabriel Ferri Schneider, Erick Menezes, Rafael Mecenas, Paulo Knob, Victor Araujo, Soraia Raupp Musse

Organizations: PUCRS Porto Alegre – RS – Brazil · INCT-SANI Brazil · UNIT Araujo – SE – Brazil · Kunumi Institute Brazil

Abstract

Accurate reproduction of facial skin tone is essential for realism, identity preservation, and fairness in Virtual Human (VH) rendering. However, most accessible avatar creation pipelines rely on photographic inputs that lack colorimetric calibration, which can introduce inconsistencies and bias. We propose a fully automatic and scalable methodology to systematically evaluate skin tone fidelity across the VH generation pipeline. Our approach defines a full workflow that integrates skin color and illumination extraction, texture recolorization, real-time rendering, and quantitative color analysis. Using facial images from the Chicago Face Database (CFD), we compare skin tone extraction strategies based on cheek-region sampling, following the literature, and multidimensional masking derived from full-face analysis. Additionally, we test both strategies with lighting isolation, using the pre-trained TRUST framework, employed without any training or optimization within our pipeline. Extracted skin tones are applied to MetaHuman textures and rendered under multiple lighting configurations. Skin tone consistency is evaluated objectively in the CIELAB color space using the $ΔE$ metric and the Individual Typology Angle (ITA). The proposed methodology operates without manual intervention and, with the exception of pre-trained illumination compensation modules, the pipeline does not include learning or training stages, enabling low computational cost and large-scale evaluation. Using this framework, we generate and analyze approximately 19,848 rendered instances. Our results show phenotype-dependent behavior of extraction strategies and consistently higher colorimetric errors for darker skin tones.

Figures & tables

Figure 1: From left to right, the first four columns represent subjects from the CFD dataset classified as ITA Class 1 (lighter skin tones), while the second group of four columns represents ITA Class 6 (darker skin tones). Each column corresponds to one extraction method for each group: $Cheek$ , $MMM$ , $T-Cheek$ , and $T-MMM$ . Additionally, each row (from top to bottom) shows the results for a different lighting configuration: CFD, Frontal, and Paramount.

Figure 2: Overview of our methodology for evaluating skin tone consistency. Starting with the ground truth images extracted from the CFD Dataset [ 21 , 22 , 18 ] (1), we extract the average skin color following four different approaches: $Cheek$ (2), $MMM$ (3), $T-Cheek$ (4), and $T-MMM$ (5). Considering our MetaHuman Setup (6), we apply a normalization (7) and a variation map (8) in the texture, before rendering the final image with three different lighting configurations: Frontal Light (9), Paramount Light (10), and CFD Light (11). Finally, we validate the resulting rendered images with two metrics: $\Delta E$ (12) and ITA Error (13).

Figure 3: Values of $\Delta E$ for 3 light conditions and 4 extraction methods. Values of ITA Error considering 6 ITA Classes and 4 extraction methods. The higher the ITA Class number, the darker the skin tone.

Figure 4: Values of ITA Error for 6 ITA Classes and 4 extraction methods. Values of ITA Error considering 6 ITA Classes and 4 extraction methods. The higher the ITA Class number, the darker the skin tone.

Figure 5: Values of ITA Error considering 6 ITA Classes and 3 light conditions. The higher the ITA Class number, the darker the skin tone.

Figure 6: Values of ITA Error considering 3 light conditions and 4 extraction methods.

Figure 7: Quadrants show $MMM$ , $T-MMM$ , $Cheek$ , and $T-Cheek$ . Columns correspond to ITA I–VI Classes, and rows show the reference image (CFD) followed by renderings under CFD Light, Frontal Light, and Paramount Light. Samples represent the maximum color difference ( $\Delta E$ ) for each ITA Class.

Figure 8: Quadrants show $MMM$ , $T-MMM$ , $Cheek$ , and $T-Cheek$ . Columns correspond to ITA I–VI Classes, and rows show the reference image (CFD) followed by renderings under CFD Light, Frontal Light, and Paramount Light. Samples represent the minimum color difference ( $\Delta E$ ) for each ITA Class.

Figure 9: Quadrants show $MMM$ , $T-MMM$ , $Cheek$ , and $T-Cheek$ . Columns correspond to three subjects in the CFD dataset, and rows show the reference image (CFD) followed by renderings under CFD Light, Frontal Light, and Paramount Light. This figure represents ITA classes I, III, and VI, respectively.

Figure 10: Confusion Matrix between ITA Class ground truth and ITA Class related to the Render.

Algorithmic Bias

Algorithmic bias studies how automated systems produce or amplify unfair outcomes across demographic groups, and how to measure and reduce them.

See all 100 papers

Mar 2, 2026

Skin Tone in the Wild: A Dataset and Benchmark for 10-tone MST Classification

Skin tone classification is a fundamental task in computer vision, with applications in quantifying medical features, assessing demographic representativeness, and mitigating algorithmic bias. However, existing datasets either rely on a restrictive 6-tone Fitzpatrick scale, prohibit training through restrictive licenses, or lack representation of extreme skin tones, while current methods often fail to disentangle true skin tone from in-the-wild imaging conditions, degrading to random-classifier performance. We introduce skin tone in the wild (STW), the first open-access dataset annotated with the 10-tone Monk Skin Tone (MST) scale, containing over 42K images and specifically curated to include underrepresented tones. Alongside a rigorous evaluation protocol, we propose SkinToneNet, a ViT-based model that leverages global image context for robust in-the-wild generalization. Experiments show that SkinToneNet achieves 90% weighted off-by-one accuracy, substantially outperforming prior baselines. We further use SkinToneNet to audit widely used face datasets, including CelebA and VGGFace2, revealing substantial representation imbalances across MST tones.

Vitor Matias, Márcus Vinícius Lobo Costa, Tiago Novello +1

Instituto de Ciências Matemáticas e de Computação (ICMC), Universidade de São Paulo (USP), São Carlos, São Paulo, Brazil · Instituto de Matemática Pura e Aplica, Rio de Janeiro, Brazil

Classification Algorithmic Fairness Image Classification Fine-Grained Image Classification

Ask PDF

Mar 11, 2026

Too Vivid to Be Real? Benchmarking and Calibrating Generative Color Fidelity

Recent advances in text-to-image (T2I) generation have greatly improved visual quality, yet producing images that appear visually authentic to real-world photography remains challenging. This is partly due to biases in existing evaluation paradigms: human ratings and preference-trained metrics often favor visually vivid images with exaggerated saturation and contrast, which make generations often too vivid to be real even when prompted for realistic-style images. To address this issue, we present Color Fidelity Dataset (CFD) and Color Fidelity Metric (CFM) for objective evaluation of color fidelity in realistic-style generations. CFD contains over 1.3M real and synthetic images with ordered levels of color realism, while CFM employs a multimodal encoder to learn perceptual color fidelity. In addition, we propose a training-free Color Fidelity Refinement (CFR) that adaptively modulates spatial-temporal guidance scale in generation, thereby enhancing color authenticity. Together, CFD supports CFM for assessment, whose learned attention further guides CFR to refine T2I fidelity, forming a progressive framework for assessing and improving color fidelity in realistic-style T2I generation. The dataset and code are available at https://github.com/ZhengyaoFang/CFM.

Zhengyao Fang, Zexi Jia, Yijia Zhong +5

Harbin Institute of Technology, Shenzhen · Independent Researcher · College of Computer Science and Aritificial Intelligence, Fudan University +2

Image Generation Image Generation Evaluation T2I Generation Evaluation

Ask PDF

Aug 17, 2026

Fidelity Preference, Not Demographic Preference: A Pixel-Level Attribute-Sensitivity Audit of Image Aesthetic/Preference Scorers

Text-to-image systems use learned aesthetic scorers to filter training data and guide generation, but whether these scores encode demographic attributes as objective quality is unclear. We audit four scorers (LAION-Aesthetics, PickScore, ImageReward, HPSv2) using pixel-level interventions on skin tone and body type in synthetic and real images. Our key finding is that along skin-lightness, the dominant effect is fidelity preference: unaltered images score highest, and perturbations in either direction are penalized (inverted-U). Placebo arms show this penalty is not an artifact of the skin operator, as applying the same CIELAB L* shift to non-skin regions yields similar penalty magnitudes. However, the penalty is operator-dependent and holds for all operators only for LAION-Aes. Critically, audits on synthetic images alone are misleading: LAION-Aes shows strong preference for darker skin on synthetic faces, but on 1470 real faces the preference reverses and becomes much smaller, and amplification becomes non-significant. Across scorers, synthetic results do not transfer -- reversing for LAION-Aes and HPSv2, attenuating for PickScore. We contribute a reproducible benchmark with artifact control and synthetic/real cross-validation, and an auditability criterion for pixel-level causal isolation (valid for skin tone, not for body type due to deformation). Population-stratified analysis shows fidelity-penalty asymmetry is not robust across groups after FDR correction except for HPSv2. Our findings show naive synthetic audits misjudge bias direction and magnitude, and only within-image causal isolation on real data can distinguish true demographic bias from fidelity preference.

Mingyang Xu

Peking University Beijing, China

Algorithmic Bias Algorithmic Fairness Image Quality Assessment Counterfactual Evaluation of VLMs

Ask PDF

Abstract

Algorithmic Bias

Explore similar work

Skin Tone in the Wild: A Dataset and Benchmark for 10-tone MST Classification

Too Vivid to Be Real? Benchmarking and Calibrating Generative Color Fidelity

Fidelity Preference, Not Demographic Preference: A Pixel-Level Attribute-Sensitivity Audit of Image Aesthetic/Preference Scorers