Shape-Bayes: Bayesian Inference of Structured Shapes under Visual Ambiguity

Mani Kumar Tellamekala; Tosh Brown; Michel Valstar

cs.CVOct 6, 2026

Shape-Bayes: Bayesian Inference of Structured Shapes under Visual Ambiguity

Authors: Mani Kumar Tellamekala, Tosh Brown, Michel Valstar

Organizations: Amsterdam, The Netherlands · BlueSkeye AI · University of Nottingham, U.K.

Abstract

Perceiving structured shapes, such as human faces, from pixels is an inherently ambiguous task in real-world conditions. Yet, shape inference is largely posed as a deterministic regression task predicting fixed spatial coordinates. We find that deterministic regression is brittle when visual evidence is ambiguous or incomplete; under severe occlusions deterministic models exhibit structural collapse, predicting incoherent shapes or reverting to generic averages. To address this, we introduce Shape-Bayes, a probabilistic framework that couples uncertainty-aware visual perception with Bayesian shape reasoning. Rather than forcing point estimates, Shape-Bayes dynamically weights visual evidence against geometric priors to infer a structurally valid shape posterior. Demonstrated on human face shape regression, a rigorous testbed featuring complex non-rigid deformations and strict anatomical constraints, Shape-Bayes comprises: (1) a base model predicting noisy landmarks alongside distilled aleatoric uncertainties; (2) a lightweight Transformer encoding these observations into an adaptive prior over a PCA shape manifold; and (3) a differentiable Bayesian solver computing closed-form posteriors by balancing the noisy predictions against this prior. By guaranteeing complete structural integrity, Shape-Bayes achieves an absolute improvement of up to ~34% IDR over state-of-the-art deterministic models. Simultaneously, it yields highly calibrated uncertainty bounds and reduces relative error by up to 12.5%, establishing a new state-of-the-art for robust 2D face shape regression under severe occlusion. The project page is at https://shape-bayes.github.io.

Figures & tables

Figure 1: Robust Bayesian shape inference under severe occlusion. ( Left ) Deterministic regression exhibits structural collapse when forced to predict point estimates for obscured regions. ( Right ) Shape-Bayes probabilistically reconciles noisy visual evidence and a latent shape prior via aleatoric uncertainty, and infers a well-calibrated posterior. This not only guarantees structural integrity but enables the sampling of multiple plausible shape hypotheses under severe ambiguity.

Figure 2: Schematic overview of Shape-Bayes . Given an initial noisy shape estimate, Shape-Bayes dynamically balances point-wise observation likelihoods ( $W$ ) against a data-driven structural prior ( $\Lambda$ ) via a closed-form Bayesian update to recover a calibrated posterior distribution $P(S)$ . Although demonstrated only on face shape regression here, Shape-Bayes is topology-agnostic and applicable to any other structured shape inference.

Figure 3: Shape-Bayes inference flow. (1) A base regression model predicts landmarks $\mathbf{S}^{\prime}$ and uncertainties $\bm{\sigma}^{\prime}$ , defining the likelihood precision $\mathbf{W}$ . (2) A Transformer maps these observations to an adaptive prior precision $\mathbf{\Lambda}_{\text{prior}}$ in PCA space. (3) A closed-form Bayesian update computes the shape posterior $\mathcal{N}(\mathbf{b}_{\mu},\tilde{\mathbf{\Sigma}}_{b})$ , yielding probable shapes via sampling and projection ( $\mathbf{S}=\bar{\mathbf{S}}+\mathbf{P}\mathbf{b}$ ).

Figure 4: Distilled Uncertainty Supervision. Teacher predictions across affine-augmented views of an input are inverse-transformed; their empirical variance provides aleatoric targets for the base model.

Figure 5: Qualitative illustration of the Shape-Bayes inference pipeline. From left to right: input image, base model shape prediction and its aleatoric uncertainty, Bayesian posterior mean, and shapes sampled from the posterior.

		300W (Dynamic Occlusion)					WFLW (Dynamic Occlusion)					COFW (Dynamic Occlusion)
Method	Cfg.	IDR $\uparrow$	NME ${}_{\text{occ}}$ $\downarrow$	NME ${}_{\text{all}}$ $\downarrow$	FR $\downarrow$	AUC $\uparrow$	IDR $\uparrow$	NME ${}_{\text{occ}}$ $\downarrow$	NME ${}_{\text{all}}$ $\downarrow$	FR $\downarrow$	AUC $\uparrow$	IDR $\uparrow$	NME ${}_{\text{occ}}$ $\downarrow$	NME ${}_{\text{all}}$ $\downarrow$	FR $\downarrow$	AUC $\uparrow$
	Base	83	9.87	7.06	11.8	36.8	90	12.27	9.08	23.3	30.9	–	–	–	–	–
LUVLi	+SB	100	8.86	6.67 / 6.17	9.9	39.5	100	11.33	8.73 / 7.97	22.0	32.3	–	–	–	–	–
	Base	–	–	–	–	–	98	12.08	8.81	25.3	28.3	–	–	–	–	–
DSLPT	+SB	–	–	–	–	–	100	11.46	8.66 / 8.25	24.6	29.4	–	–	–	–	–
	Base	78	10.36	7.07	14.4	36.5	84	10.49	8.24	21.3	33.0	100	9.56	6.61	10.8	37.1

Table 3: Generalization of Shape-Bayes (+SB) across base models and datasets (

\text{NME}_{\text{all}}

: mean /

\text{minNME}_{@100}

). Base models in gray natively output uncertainty, requiring no training.

Table 8

Figure 8: Posterior calibration on occluded 300W. (a) Error vs. uncertainty shows strong correlation ( $\rho=0.67$ ). (b) Reliability plot yields robust calibration ( $\text{ECE}=1.97$ ). (c) Sparsification curve tracks the oracle.

Architectural Configuration	NME $\downarrow$	minNME @100 $\downarrow$	FR $\downarrow$	AUC $\uparrow$
Base Deterministic Regression	9.03	9.03	32.15%	25.81%
Deterministic Manifold Projection	8.85	8.85	31.57%	26.97%
Static Prior + Distilled Aleatoric Uncertainty	8.08	6.79	25.69%	30.61%
Dynamic Prior (w/ Uniform Uncertainty)	8.00	6.68	25.47%	30.80%
Shape-Bayes (Dynamic Prior + Distilled Unc.)	7.91	6.60	24.67%	31.53%

Table 7: Ablation of Shape-Bayes on occluded 300W. Rows are ordered by architectural progression. Cell shading tracks performance from high failure rate ( red ) to low failure rate ( blue ).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Qualitative results on occluded 300W. HR-noSA ( Yang and Yeh, 2025 ) base model vs. Shape-Bayes .

Figure 10: Qualitative results on occluded COFW. HR-noSA ( Yang and Yeh, 2025 ) base model vs. Shape-Bayes .

Figure 11: Qualitative results on occluded WFLW. HR-noSA ( Yang and Yeh, 2025 ) base model vs. Shape-Bayes .

Base Model	Config.	Jaw	R-Brow	L-Brow	Nose	L-Eye	R-Eye	Mouth
ORFormer ( Chiang et al., 2025 )	Base	9.0	9.5	9.8	9.5	7.1	6.6	11.1
ORFormer ( Chiang et al., 2025 )	+SB	8.8	8.8	9.3	7.8	6.3	5.7	9.7
LUVLi ( Kumar et al., 2020 )	Base	10.7	14.1	12.9	9.0	10.2	9.3	9.3
LUVLi ( Kumar et al., 2020 )	+SB	9.9	11.3	11.1	7.4	8.0	7.7	8.8
STAR ( Zhou et al., 2023 )	Base	13.0	10.2	12.9	10.5	9.1	7.2	12.0
STAR ( Zhou et al., 2023 )	+SB	11.5	9.2	10.6	7.7	7.3	6.1	10.2

Appendix

Table 8: Region-wise shape prediction error in terms of NME

{}_{\text{occ}}

(

\downarrow

) on occluded 300W .

Base Model	Config.	Jaw	R-Brow	L-Brow	Nose	L-Eye	R-Eye	Mouth
HR-noSA ( Yang and Yeh, 2025 )	Base	11.8	11.9	11.4	11.3	9.6	10.4	12.9
HR-noSA ( Yang and Yeh, 2025 )	+SB	11.5	11.0	10.9	9.9	8.9	9.3	11.8
ORFormer ( Chiang et al., 2025 )	Base	11.5	11.2	11.4	9.0	9.5	8.7	10.5
ORFormer ( Chiang et al., 2025 )	+SB	11.0	10.9	11.2	8.4	9.1	8.4	10.2
LUVLi ( Kumar et al., 2020 )	Base	11.5	12.9	13.6	11.3	11.5	11.1	12.8
LUVLi ( Kumar et al., 2020 )	+SB	11.1	12.2	12.7	10.2	10.5	10.2	11.8

Appendix

Table 9: Region-wise shape prediction error in terms of NME

{}_{\text{occ}}

(

\downarrow

) on occluded WFLW .

Base Model	Config.	R-Brow	L-Brow	R-Eye	L-Eye	Nose	Mouth	Chin
HR-noSA ( Yang and Yeh, 2025 )	Base	12.0	14.4	10.0	12.1	10.7	11.1	13.4
HR-noSA ( Yang and Yeh, 2025 )	+SB	10.6	11.7	8.5	8.9	9.1	9.8	14.6
ORFormer ( Chiang et al., 2025 )	Base	10.1	11.6	8.0	8.8	9.3	9.9	8.7
ORFormer ( Chiang et al., 2025 )	+SB	9.0	10.4	7.5	7.9	8.6	9.5	8.1
STAR ( Zhou et al., 2023 )	Base	11.9	14.4	9.9	12.0	10.3	10.1	13.0
STAR ( Zhou et al., 2023 )	+SB	10.7	12.5	8.6	9.9	9.0	9.3	12.8

Appendix

Table 10: Region-wise shape prediction error in terms of NME

{}_{\text{occ}}

(

\downarrow

) on occluded COFW .

Component	Formula	Per-Unit	Subtotal
Input Projection
Linear $(4\to d)$	$4d+d$	320	320
Per Transformer Encoder Layer ( $\times L=5$ )
Multi-head self-attention
In-projection $W_{QKV}$	$3d^{2}+3d$	12,480
Out-projection $W_{O}$	$d^{2}+d$	4,160

Appendix

Table 11: Shape-Bayes Transformer parameter breakdown (300W,

N\!=\!68

K\!=\!60

d\!=\!64

d_{\text{ff}}\!=\!128

L\!=\!5

H\!=\!8

). All counts verified against torch.nn.TransformerEncoder .

Operation	Formula	FLOPs
Multi-Head Self-Attention
$Q,K,V$ projections	$2\cdot N\cdot d\cdot 3d$	1,671,168
Attention scores $QK^{\top}$	$2\cdot N^{2}\cdot d$	591,872
Attention $\times$ values	$2\cdot N^{2}\cdot d$	591,872
Output projection	$2\cdot N\cdot d^{2}$	557,056
Self-attention subtotal	$2N(4d^{2}+2Nd)$	3,411,968

Appendix

Table 12: Per-layer FLOPs breakdown for the Shape-Bayes Transformer (

N\!=\!68

d\!=\!64

d_{\text{ff}}\!=\!128

H\!=\!8

Stage	Formula	FLOPs
Input projection	$2\cdot N\cdot 4\cdot d$	34,816
Transformer encoder ( $L\!=\!5$ layers)	$L\times 5.64\text{M}$	28,200,960
Precision head	$2\cdot(Nd)\cdot K$	522,240
Neural network subtotal		28,758,016
Closed-Form Bayesian Solve (Section 2 )
$\mathbf{P}^{\top}\operatorname{diag}(\mathbf{w})$ scaling	$KM$	8,160

Appendix

Table 13: End-to-end FLOPs for the complete Shape-Bayes inference pipeline.

Model	Parameters	FLOPs	Relative Cost
StackedHGNetV1 (base)	$\sim$ 13.4M	$\sim$ 29.1 GFLOPs	1.00 $\times$
Shape-Bayes module (MAP)	429K	$\sim$ 29.8 MFLOPs	–
Shape-Bayes module (with sampling)	429K	$\sim$ 30.3 MFLOPs	–
Shape-Bayes / Base	3.2%	0.10%	$\sim$ $1000\times$ cheaper

Appendix

Table 14: Computational cost comparison: Shape-Bayes module vs. StackedHGNetV1 base model.

Component	Size	Notes
Model weights	1.64 MB	$428{,}988\times 4$ bytes
PCA basis $\mathbf{P}\in\mathbb{R}^{M\times K}$	32.6 KB	$136\times 60\times 4$ bytes, stored once
Mean shape $\bar{\mathbf{s}}\in\mathbb{R}^{M}$	0.53 KB	$136\times 4$ bytes
Activations (peak)	$\sim$ 170 KB	Dominated by $L\times N\times d$ attention maps
Precision matrix $\bm{\Lambda}\in\mathbb{R}^{K\times K}$	14.4 KB	Per-sample; used in linalg.solve
Total	$<$ 2 MB	vs. $\sim$ 51 MB for StackedHGNetV1 weights

Appendix

Table 15: Inference-time memory overhead of the Shape-Bayes module (FP32, batch size 1).

Mode	Anchor Point	Face Area %	Representative Assets
Global Scene	Full image	100%	Chainlink fence, frost, foliage, prison bars
Helmet	Inter-ocular midpoint	110–140%	Navy/silver, white/blue helmets
Eyewear	Inter-ocular midpoint	15–35%	Goggles, VR headset, masquerade mask
Faceguard	Nose–chin midpoint	25–40%	Football helmet faceguard
Mouth	Mouth centroid	5–40%	Mug, burger, pizza, microphones, fruit
Camera	Random eye/midpoint	20–40%	DSLR, telephoto lens, TLR camera

Appendix

Table 16: Object overlay placement modes and representative assets.

Explore similar work

ShapeY: A Principled Framework for Measuring Shape Recognition Capacity via Nearest-Neighbor Matching
Apr 27, 2026Jong Woo Nam, Amanda S. Rios, Bartlett W. MelInvariant Representation Learning Fine-Grained Visual Perception
DinoComplete: 3D Shape Completion with Distilled Semantic Priors and State Space Models
May 26, 2026Furkan Mert Algan, Eckehard Steinbach3D Reconstruction 3D Shape Completion
Chair of Media Technology · Munich Institute of Robotics and Machine Intelligence · School of Computation Information and Technology, Technical University of Munich
Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations
Apr 29, 2026Andrii Zadaianchuk, Leonardo Barcellona, Lennard Schuenemann +73D Reconstruction Object Pose Estimation
University of Amsterdam · KU Leuven · Toyota Research Institute +1

Apr 27, 2026, cs.CVJ/K move · Enter open · S save

ShapeY: A Principled Framework for Measuring Shape Recognition Capacity via Nearest-Neighbor Matching

Jong Woo Nam, Amanda S. Rios, Bartlett W. Mel

Invariant Representation Learning Fine-Grained Visual Perception

Open paper AskPDF arXiv

Object recognition (OR) in humans relies heavily on shape cues and the ability to recognize objects across varying 3D viewpoints. Unlike humans, deep networks often rely on non-shape cues such as texture and background, leading to vulnerabilities in generalization and robustness. To address this gap, we introduce ShapeY, a novel and principled benchmarking framework designed to evaluate shape-based recognition capability in OR systems. ShapeY comprises 68,200 grayscale images of 200 3D objects rendered from multiple viewpoints and optionally subjected to non-shape ``appearance'' changes. Using a nearest-neighbor matching task, ShapeY specifically probes the fine-grained structure of an OR system's embedding space by evaluating whether object views are clustered by 3D shape similarity across varying 3D viewpoints and other non-shape changes. ShapeY provides a suite of quantitative and qualitative performance readouts, including error rate graphs, viewpoint tuning curves, histograms of positive and negative matching scores, and grids showing ordered best matches, which together offer a comprehensive evaluation of an OR system's shape understanding capability. Testing of 321 pre-trained networks with diverse architectures reveals significant challenges in achieving robust shape-based recognition: even state-of-the-art models struggle to generalize consistently across 3D viewpoint and appearance changes, and are prone to infrequent but egregious matches of objects of obviously completely different shape. ShapeY establishes a principled framework for advancing artificial vision systems toward human-like shape recognition capabilities, emphasizing the importance of disentangled and invariant object encodings.

Abstract

Explore similar work

ShapeY: A Principled Framework for Measuring Shape Recognition Capacity via Nearest-Neighbor Matching

DinoComplete: 3D Shape Completion with Distilled Semantic Priors and State Space Models

Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations