A Lightweight Differentiable Method for Real-Time Skin Rendering and Digitization
Authors: Mario Marquez-Balduque, Carlos Rodriguez-Pardo, Jorge Lopez-Moreno
Organizations: Universidad Rey Juan Carlos, Spain · Politecnico di Milano, Milan, Italy · Euro-Mediterranean Center on Climate Change (CMCC), Lecce, Italy · RFF-CMCC European Institute on Economics and the Environment (EIEE), Milan, Italy
Inverse rendering for skin typically relies on offline path tracing to estimate physical properties, which fail to map directly to the approximate shaders utilized in interactive applications, such as game engines. We bridge this gap by directly estimating subsurface scattering parameters tailored for real-time rendering pipelines. We introduce a differentiable Screen-Space Subsurface Scattering (SSSS) framework that enables the joint optimization of spatially varying base textures and scattering distances. To account for local variations in skin thickness, we estimate these parameters across distinct facial regions, generating a blended profile map ready for production runtime engines. Our pipeline utilizes a low-cost, single-light, single-view capture setup designed for accessible and practical appearance acquisition. Optimization is driven by a masked loss function targeting regions where subsurface scattering effects are dominant. We validate our approach on a real-world dataset of 45 individuals across the Monk Skin Tone Scale, ensuring demographic generalization. Furthermore, we complement this evaluation with synthetic data, a multi-light setup validating our single-light assumption, and a tilted-view configuration confirming angular consistency. Results demonstrate that our method effectively recovers scattering parameters from lightweight inputs, yielding realistic skin appearance directly compatible with standard real-time shaders.
Figures & tables
Figure 3: Illustration of subsurface scattering regimes. Left: a rendered ear where the yellow box highlights a single scattering dominant area (thin cartilage), while the green box shows a region dominated by multiple scattering, emphasizing the variability of light transport in human skin. Right: representation of single, intermediate, and multiple scattering. Adapted from [ 34 ] .
Figure 5: Texture attachments of our rendering engine: (a) thickness, (b) region, and (c) blended scattering distance. For the region texture (b), the red channel encodes discrete region tags, while the green channel stores the interpolation factor for scattering distance blending .
Figure 6: Left: Fibonacci sampling pattern used for importance sampling the diffusion profile. Right: Chromatic fringing observed in captures due to spectral dependencies in the projector light.
Figure 7: Final render produced with our engine (a). Insets highlight regions dominated by (b) multiple scattering and (c) single scattering.
Figure 8: Hardware acquisition setup and illumination patterns. (a) Physical configuration (left) and corresponding schematic ray-diagram (right) illustrating the scanning system geometry. (b) Temporal sequence of projected light patterns used during data acquisition: ambient baseline, structured stripes, inverted stripes, and uniform flat-field illumination.
Figure 9: Localized view of the captured raw frame sequence during acquisition. (a) Ambient baseline frame (brightness-enhanced for visibility) showcasing residual environmental light. (b) Active surface capture under the high-frequency structured stripe pattern. (c) Corresponding active capture under the inverted stripe pattern. (d) Fully illuminated frame under uniform flat-field projection.
Figure 10: Optimization input data evaluated across different anatomical skin regions (rows). From left to right: (a) Uniformly illuminated reference frames, (b) active strip-lit captures displaying subsurface scattering effects, (c) light masks, and (d) estimated monocular depth maps.
Configuration
Stage 1
Stage 2
Stage 3
Optimized Parameters
Δspec,T,ρT
AT,ρT,Δspec,T
ST
Frozen Parameters
AT,ST
ST
AT,ρT,Δspec,T
Iterations
700
500
600
Learning Rate
5×10−3
5×10−3
1×10−3
Active Mask ( Mopt )
Full Image ( 1 )
Full Image ( 1 )
Shadow Mask ( M )
Roughness Prior ( λprior )
1×10−2
1×10−2
None
Table 1: Optimization Schedule and Hyperparameters.
o r *4c Monk Range
FLIP ↓
LPIPS ↓
SSIM ↑
PSNR ↑
Monk 1-3
0.093±0.026
0.086±0.020
0.615±0.066
28.61±1.68
Monk 4-6
0.093±0.031
0.088±0.018
0.614±0.066
28.67±1.89
Monk 7-10
0.077±0.021
0.115±0.015
0.503±0.072
29.65±1.35
Table 2: Quantitative reconstruction errors across the Monk Scale, computed on masked image regions.
o l *4c Metric
Stage 1
Stage 2
Stage 3
Overall
Execution Time (s)
51.1
38.3
109.0
198.4
Iterations
700
500
600
1800
Time per Iteration (ms)
72.9
76.6
181.7
110.2
Peak VRAM (GB)
2.5
2.6
9.4
9.4
Table 3: Computational cost per skin patch ( 480×360 px, N=128 scattering samples) across optimization stages on an AMD Ryzen 9 7900 CPU (12 cores, 24 threads) and an NVIDIA GeForce RTX 4070 Ti GPU. The Overall column reports cumulative values for time and iterations, mean iteration latency, and maximum peak VRAM.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Map progression across optimization stages. Green and red denote active and frozen maps, respectively. Stage 1 fits global reflection, capturing unmodeled specular highlights into the residual map while shielding initialized albedo. Stage 2 unfreezes albedo to absorb diffuse spatial details, enabling the L2 prior to smooth roughness while preserving the specular residual. Stage 3 optimizes scattering distance over shadow-masked regions (Input) with surface maps frozen.
Figure 2: Tilted view validation. In synthetic cases (top), dopt closely tracks dref . For real captures (bottom), scattering distances remain stable across views, demonstrating that 3D depth-aware computations prevent parameter shrinkage caused by perspective foreshortening.
Figure 3: Full synthetic validation across the Monk Skin Tone Scale. Per tone (left to right): input, optimized reconstruction, and FLIP error map. Inset swatches denote dref (known reference value) and dopt (optimized value). Low perceptual errors confirm estimation invariance across diverse skin tones.
Figure 4: Unified confusion matrix cross-referencing participant self-identified (Perceived) skin tones against our automatic appearance-based color-matching algorithm (Computed) across the Monk Skin Tone Scale ( N=45 ). The clustering behavior highlights systematic discrepancies between subjective human self-assessment and objective physical albedo extraction, driven by visual perception biases and the uncalibrated nature of the original scale.
Figure 5: Physical color calibration setup under the projector’s flat-field white illumination. The 24 patch coordinates are automatically extracted to fit the 3×3 CCM.
In this paper we present a spectral optical inversion for skin for path tracing-based rendering of subsurface scattering. Skin is a complex multilayered medium, with appearance determined by the mixture of biophysical chromophores. However, current methods rely on medium homogeneization, with optical parameters obtained via albedo inversion from a reflectance texture and hand-tuned scattering distance and anisotropy. This results into significant art-skilled manual labor for authoring, and an inaccurate scattering profile for skin. To solve these problems, we generalize existing albedo inversion techniques, and propose a framework that predicts full-spectral skin scattering parameters from a single RGB diffuse albedo. Our method builds upon a new mixture-of-media representation, that approximates the aggregated multilayered appearance of skin by mixing the aggregated scattering of three uncorrelated media. We train a chained neural decoder that maps RGB diffuse albedo to the optical properties of the mixture of media, including anisotropy, scattering radius and scattering albedo. Then, we show this mixture can be used in a random-walk-based path tracer with minimal modifications, by simply randomly selecting the medium to traverse.
We propose a self-supervised pretraining framework for learning sub-surface scattering (SSS) light transport representations from minimal input. Our method leverages a stereo projector-camera setup that captures only eight high-frequency phase-shift profilometry (PSP) images per view to pretrain an encoder in a multi-view, multi-object setting. We introduce a tailored augmentation strategy for PSP-based SSS data, and show that it significantly outperforms standard ImageNet-style augmentations for SSL pretraining. The pretrained encoder learns generalizable SSS representations that transfer effectively to downstream tasks, including spatially varying relighting and representation evaluation using a kNN classifier. Combined with a decoder, the model reconstructs dense scattering footprint responses, trained using a dedicated cost function that improves accuracy, particularly for anisotropic footprints. Despite using only eight input images per view, our approach generalizes to unseen objects with complex geometry and material properties, achieving high-fidelity reconstructions while requiring orders of magnitude fewer images than prior methods.
Arjun Majumdar, Raphael Braun, Andreas Engelhardt +1
We present 3D Surface Splatting (3DSS), the first differentiable surface splatting renderer for physically-based inverse rendering from multi-view images. Our central insight is that the surface separation problem at the heart of surface splatting admits a direct formulation in terms of the reconstruction kernels themselves. From this foundation we derive a coverage-based compositing model whose per-layer opacity arises directly from the accumulated Elliptical Weighted Average reconstruction weight, yielding anti-aliased silhouettes and informative visibility gradients at sparsely covered edges. Combined with forward microfacet shading under co-optimized HDR environment lighting and density-aware adaptive refinement, 3DSS jointly recovers shape, spatially-varying BRDF materials, and illumination. Because the optimized representation is a set of oriented surface samples, it bridges natively to mesh-based workflows via surface reconstruction from oriented point cloud methods. We evaluate 3DSS against mesh-based, implicit, and Gaussian-splatting baselines across geometry reconstruction, novel-view synthesis, and novel-illumination relighting.