Recovering Black-Box Point Cloud Models from Pose Shifts via Derivative-Free Optimization

Seonghwan Park; Min-Gyu Park

cs.CVOct 26, 2026

Recovering Black-Box Point Cloud Models from Pose Shifts via Derivative-Free Optimization

Authors: Seonghwan Park, Min-Gyu Park

Organizations: Korea Electronics Technology Institute (KETI)

Abstract

Point cloud models are increasingly deployed through score-only black-box interfaces. Under this access model, test-time rotations outside the training distribution can substantially degrade performance. Although existing mitigations such as equivariant retraining and gradient-based adaptation are effective, they are infeasible without access to model weights or gradients. To address this limitation, we propose CanDO, a test-time canonicalization method that recovers lost performance using only forward queries. Specifically, CanDO searches for an input transformation that maximizes model confidence and uses derivative-free optimization to perform this search without gradients. We further identify three conditions for successful recovery: a training augmentation gap, an accuracy-aligned confidence surrogate, and global search over a multi-modal objective. Across multiple architectures and benchmarks, CanDO recovers 76% of the oracle gap under azimuthal misorientation and 51% under arbitrary SO(3) rotations for object part segmentation. These results show that frozen score-only point cloud models can recover from rotation-induced distribution shifts through post-deployment canonicalization alone.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Rotation-induced shifts within each frozen point cloud model’s augmentation gap collapse performance, and CanDO recovers a large share of the gap to the supervised oracle from forward queries alone. (b) A frozen point cloud model on S3DIS collapses under tilt despite yaw augmentation; the star marks the mIoU recovered by CanDO over the shaded deployment band, without labels or gradients.

Benchmark	Transform	Backbone	Corrupted	Recovered	Oracle	% Gap
			mIoU / Accuracy
ShapeNetPart	Azimuth	PointNet++	$0.387{\pm}0.005$	$\mathbf{0.752}{\pm}0.002$	$0.869{\pm}0.000$	76
	Azimuth	PointNeXt-S	$0.396{\pm}0.005$	$\mathbf{0.718}{\pm}0.003$	$0.871{\pm}0.000$	68
	$SO(3)$	PointNet++	$0.352{\pm}0.001$	$\mathbf{0.602}{\pm}0.003$	$0.845{\pm}0.001$	51
	$SO(3)$	PointNeXt-S	$0.372{\pm}0.002$	$\mathbf{0.637}{\pm}0.003$	$0.843{\pm}0.001$	56
S3DIS	Tilt	PointNet++	$0.305{\pm}0.005$	$\mathbf{0.528}{\pm}0.046$	$0.735{\pm}0.023$	52

Table 1: Test-time pose recovery performance of frozen point cloud models under pose corruptions across various tasks, scales, and architectures. Relying exclusively on forward queries, CanDO bridges a substantial portion of the oracle gap across all evaluated benchmarks and backbones.

Method	Gradient	Update	Label	ShapeNetPart [ 66 ]	S3DIS [ 1 ]	ModelNet40 [ 63 ]
				mIoU		Accuracy
Source (No Adaptation)	✗	✗	✗	$0.351\pm 0.004$	$0.338\pm 0.011$	$0.128\pm 0.007$
PCA Canonicalization [ 64 ]	✗	✗	✗	$0.237\pm 0.003$	$0.047\pm 0.004$	$0.221\pm 0.008$
PCA + Confidence	✗	✗	✗	$0.293\pm 0.004$	$0.047\pm 0.004$	$0.260\pm 0.009$
External Prior (Classifier)	✗	✗	✗	$0.359\pm 0.005$	N/A	N/A
DUA [ 27 ]	✗	✓	✗	$0.356\pm 0.005$	$0.357\pm 0.011$	$0.118\pm 0.007$

Table 2: Performance comparison of pose recovery against established baselines and a supervised oracle, under

SO(3)

rotation. Using solely forward queries, CanDO achieves the highest recovery, outperforming even the white-box baselines. Bold denotes the best results.

Figure 2: Qualitative results of pose recovery on ShapeNetPart. From left to right, each column depicts the misoriented input, the pose recovery achieved by CanDO , and the clean upright target.

Transform	Augmented	Accuracy Drop	Recovery $\Delta$	Recoverable
Yaw	Yes	0.01	$+0.00$	✗ (no headroom)
Scale	No	0.08	$+0.00$	✗ (no headroom)
$z$ -offset	No	0.22	$-0.02$	✗ (misaligned confidence)
Color	No	0.06	$+0.03$	✗ (no headroom)
Tilt	No	0.42	$+0.22$	✓

Table 3: Conditions for recoverability on the S3DIS PointNet++ model. Recovery requires performance headroom (an accuracy drop) and an accuracy-aligned confidence surrogate. The unseen tilt (green) best satisfies both prerequisites. Recovery

\Delta

denotes the accuracy change after label-free recovery.

Augmentation	Clean	Yaw	Tilt	$SO(3)$
None	0.828	$0.390{\to}0.562$	$0.505{\to}0.613$	$0.447{\to}0.558$
Yaw	0.825	$0.824{\to}0.829$	$0.529{\to}0.655$	$0.539{\to}0.668$
Tilt	0.826	$0.733{\to}0.765$	$0.813{\to}0.819$	$0.779{\to}0.798$
$SO(3)$	0.820	$0.804{\to}0.811$	$0.808{\to}0.815$	$0.805{\to}0.812$

Table 4: Causal link between augmentation gaps and recoverability on ShapeNetPart. Cells report corrupted

\to

recovered mIoU, shaded by recovery gain so that darker red indicates larger improvements. Recovery is largest when the corruption is absent from training augmentation and reduced when it is covered.

Figure 3: (Left) For one shape, the confidence objective is multi-modal yet peaks at the same pose as the true mIoU, so a local search is trapped while a global one is not. (Right) Across shapes and poses the objective tracks accuracy, with a pooled Pearson correlation of

0.51

Optimizer	Family	% of Oracle Gap
Coordinate Finite Differences [ 6 ]	Local	17
NES [ 60 ]	Local	18
Score-Function [ 61 ]	Local	19
SPSA [ 47 ]	Local	27
Bayesian Optimization [ 43 ]	Global	57
Random Search [ 2 ]	Global	53

Table 5: Optimizer comparison on

SO(3)

part segmentation recovery. Performance is measured as the percentage of the oracle gap recovered under a matched budget of approximately

300

queries per shape. Global search strategies significantly outperform local estimators, which plateau at spurious local optima.

Budget	Bayesian [ 43 ]	Random Search [ 2 ]	CMA-ES [ 16 ]
60	41	42	52
120	46	43	54
240	56	51	53

Table 6: Query efficiency of global search algorithms on

SO(3)

recovery. Performance is reported as the percentage of the oracle gap recovered across varying query budgets. Algorithmic rankings are budget-dependent: CMA-ES dominates constrained regimes, whereas Bayesian optimization prevails at higher query counts.

Objective	Transform $\Delta$	Noise $\Delta$	Density $\Delta$	Mean $\Delta$	Identity Kept
Part Segmentation	$+0.001$	$+0.002$	$+0.001$	$+0.002$	$89\%$
Classification	$+0.015$	$-0.070$	$-0.028$	$-0.034$	$62\%$

Table 7: Robustness across PointCloud-C corruptions. The dense per-point segmentation objective remains stable under confidence maximization, whereas the single classification softmax is more susceptible to overconfidence and performance degradation.

Corruption	Family	$\Delta$ Confidence	$\Delta$ Marginal Entropy
Rotate	Transform	$+0.055$	$+0.169$
Scale	Transform	$-0.026$	$-0.037$
Jitter	Noise	$-0.054$	$-0.125$
Add (Global)	Noise	$-0.093$	$-0.150$
Add (Local)	Noise	$-0.064$	$-0.150$
Dropout (Global)	Density	$-0.018$	$-0.038$

Table 8: Change in accuracy (

\Delta

) across ModelNet-C corruptions when optimized using confidence and marginal-entropy objectives. Performance gains are predominantly localized to rotational transformations, and the marginal-entropy formulation fails to mitigate the performance degradation observed under non-pose corruptions.

Panel of Figure 4 — Figure 4: Recovery across initial misorientation angles $\alpha$ under random-axis $SO(3)$ rotations on ShapeNetPart and ModelNet40. The identity prior preserves canonical inputs but struggles with near-inversions. Conversely, omitting the prior improves large-rotation recovery but degrades well-posed inputs.

Frozen model	Equivariance	Clean	$SO(3)$	$+$ CanDO
PointNet++ [ 35 ]	✗	0.847	0.352	0.602
PointNeXt-S [ 36 ]	✗	0.841	0.372	0.637
VN-PointNet [ 7 ]	✓	0.794	0.794	0.794
VN-DGCNN [ 7 ]	✓	0.774	0.774	0.774

Table 9: Performance of frozen ShapeNet models under

SO(3)

corruption. Equivariant Vector-Neurons [ 7 ] remain strictly invariant, rendering CanDO an identity operation (green). Conversely, non-equivariant backbones suffer severe degradation, from which CanDO recovers a substantial portion of the oracle gap (red).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Recovery versus optimization budget on a common model-evaluation axis. Gradient ascent plateaus well below global search, whereas CMA-ES is the most query-efficient.

Method	Budgets	Results at Each Budget
TENT [ 57 ]	$1$ / $10$ / $50$	0.345	0.346	0.301
MEMO [ 67 ]	$1$ / $10$ / $50$	0.344	0.357	0.337
Gradient Ascent	$32$ / $128$ / $8192$	0.509	0.515	0.528
Random Search [ 2 ]	$32$ / $128$ / $8192$	0.541	0.575	0.567
CMA-ES [ 16 ]	$32$ / $128$ / $8192$	0.552	0.602	0.577

Appendix

Table 10: Budget-saturation study. Each row varies its own budget and is comparable only within the row. Increasing the budget provides little additional benefit across methods.

Figure 6: Recovery versus query budget under random-axis

SO(3)

. Recovery shows diminishing returns and remains bounded by the oracle. Shaded bands denote standard error.

Augmentation	Clean	Yaw	Tilt	$SO(3)$
None	0.842	$0.376{\to}0.540$	$0.520{\to}0.621$	$0.469{\to}0.586$
Yaw	0.838	$0.840{\to}0.842$	$0.539{\to}0.628$	$0.547{\to}0.682$
$SO(3)$	0.835	$0.820{\to}0.828$	$0.822{\to}0.827$	$0.819{\to}0.827$

Appendix

Table 11: Augmentation ablation on ShapeNetPart part segmentation. Each cell reports corrupted

\to

recovered mIoU. Recovery decreases as augmentation coverage increases.

Drop Under	Tilt	Large Scale	Color
PointNet [ 34 ]	0.48	–	–
PointNet++ [ 35 ]	0.42	0.20	0.09
PointNeXt-S [ 36 ]	0.53	0.20	0.08

Appendix

Table 12: Accuracy drops under tilt, scale, and color across architectures. All backbones remain robust to augmented transformations but degrade substantially under the unseen tilt.

Backbone	Family	Clean	$SO(3)$	$+$ CanDO	% Gap	Corr
PointNet++ [ 35 ]	MLP	0.92	0.13	0.34	29	0.31
PointNeXt-S [ 36 ]	Conv	0.93	0.16	0.48	42	0.52
PointVector-S [ 8 ]	Conv	0.93	0.15	0.47	40	0.54
Point-MAE [ 30 ]	Transformer	0.93	0.15	0.41	34	0.26

Appendix

Table 13: Classification recovery under

SO(3)

rotations across architecture families. CanDO consistently recovers a substantial portion of the oracle gap, and confidence-accuracy correlation remains positive across backbones.

Corruption	Family	Source	$+$ CanDO	$\Delta$
Rotate	Transform	0.805	0.806	$+0.001$
Scale	Transform	0.839	0.840	$+0.001$
Jitter	Noise	0.791	0.795	$+0.004$
Add (Global)	Noise	0.813	0.815	$+0.002$
Add (Local)	Noise	0.837	0.839	$+0.002$
Dropout (Global)	Density	0.850	0.852	$+0.001$

Appendix

Table 14: ShapeNet-C recovery with the part-segmentation model. The dense per-point confidence surrogate remains reliable across all corruption families, with every change between

+0.001

and

+0.004

mIoU.

Objective	Prob.	Temp. $0.5$	Temp. $2$	Temp. $4$	Log-prob.	Neg. Entropy
Recovered	$0.309$	$0.297$	$0.309$	$0.277$	$0.309$	$0.307$

Appendix

Table 15: Calibration study of

SO(3)

recovery on ModelNet40. Recovery remains stable across temperature scaling and is identical under logit and probability access.

Figure 7: Per-category analysis of

SO(3)

recovery. (Left) Categories with distinctive canonical frames exhibit the largest gains, whereas rotationally ambiguous categories show limited or negative recovery. (Right) Recovery increases with corruption magnitude, indicating that residual failures arise primarily from geometric ambiguity rather than optimization.

Task	Prior	Mean $\Delta$	$>1\%$ harmed	$>5\%$ harmed	$>10\%$ harmed
Part Seg.	$\lambda{=}0.08$	$-0.005$	$6.2\%$	$3.2\%$	$2.6\%$
Part Seg.	$\lambda{=}0$	$-0.143$	$41.0\%$	$31.6\%$	$29.1\%$
Classif.	$\lambda{=}0.08$	$-0.053$	$6.0\%$	$6.0\%$	$6.0\%$
Classif.	$\lambda{=}0$	$-0.231$	$23.4\%$	$23.4\%$	$23.4\%$

Appendix

Table 16: Tail risk on clean inputs for part segmentation and classification: mean accuracy change and the fraction of inputs degraded by more than

1

5

, and

10\%

, with and without the identity prior.

$\lambda$	Rotate	Scale	Jitter	Add (Global)
0.2	$+0.086$	$-0.031$	$-0.086$	$-0.148$
0.4	$+0.047$	$-0.016$	$-0.055$	$-0.102$
0.7	$+0.031$	$-0.016$	$-0.047$	$-0.062$

Appendix

Table 17: Identity-prior ablation: change in accuracy versus the prior weight

\lambda

on ModelNet-C. Larger

\lambda

preserves the rotation gain while reducing harm on non-pose corruptions.

Corruption	Family	$\tau{=}0$	$\tau{=}0.05$	$\tau{=}0.1$	$\tau{=}0.2$
Rotate	Transform	$+0.060$	$+0.058$	$+0.067$	$+0.054$
Jitter	Noise	$-0.067$	$-0.058$	$-0.048$	$-0.067$
Add (Global)	Noise	$-0.073$	$-0.065$	$-0.067$	$-0.060$
Dropout (Global)	Density	$-0.017$	$-0.017$	$-0.002$	$-0.002$

Appendix

Table 18: Confidence-margin gate on ModelNet-C: change in accuracy versus the acceptance margin

\tau

, with

\tau{=}0

the ungated method. Larger

\tau

reduces harm on corruptions with misaligned confidence while preserving the rotation gain, but cannot fix outlier addition.

Task	$\tau$	Net Gain	Clean $\Delta$	Mild $\Delta$	Severe $\Delta$	Identity Kept
Part Seg.	$0.00$	$+0.036$	$-0.002$	$+0.081$	$+0.067$	$70\%$
	$0.05$	$+0.030$	$-0.004$	$+0.066$	$+0.064$	$98\%$
	$0.10$	$+0.021$	$-0.003$	$+0.039$	$+0.051$	$99\%$
	$0.20$	$+0.003$	$+0.000$	$+0.001$	$+0.013$	$100\%$
Classif.	$0.00$	$+0.028$	$-0.051$	$+0.156$	$+0.060$	$70\%$
	$0.05$	$+0.029$	$-0.049$	$+0.154$	$+0.060$	$83\%$

Appendix

Table 19: Mixed-stream single-policy sweep, averaged over three seeds. Part segmentation admits a do-no-harm operating point with positive net gain, whereas classification trades canonical harm for the rotated gain at every threshold.

Distribution Shift Robustness

Distribution shift robustness studies how to keep machine-learning predictions reliable when deployment data differ from training data.

See all 280 papers

Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring
Jun 23, 2026Dominik Lindner, Johann Schmidt, Tom Siegl +2Distribution Shift Robustness
Artificial Intelligence Lab, Otto-von-Guericke University Magdeburg, Germany · University of Rostock, Germany · Becker Lab, University of Marburg, Germany
LexiCan: Lexicographic Canonicalization of Point Clouds for Machine Learning Applications
Oct 10, 2026Daniel Sanchez, Djordje Mihajlovic, Federica Valeau +4Invariant Representation Learning Molecular Representation Learning
Technion - Israel Institute of Technology
COSTA: A Cluster-Centric Paradigm for Annotation-Free Open-Set Semantic Segmentation of Aerial Point Clouds with Domain Shifts
Aug 19, 2026Yanghong Lin, Li Fang, Tianyu Li +2Graph-Based Label Propagation 3D Point Cloud Segmentation
Fujian Institute of Research on the Structure of Matter, Chinese Academy of Sciences, Fuzhou, 350108, Fujian, China. · Spatial Intelligence and Urban Computing, Institute of Urban Environment, Chinese Academy of Sciences, Xiamen, 361021, Fujian, China. · State Key Lab for Ecological Security of Regions and Cities, Institute of Urban Environment, Chinese Academy of Sciences, Xiamen, 361021, Fujian, China. +2

Jun 23, 2026J/K move · Enter open · S save

Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring

Dominik Lindner, Johann Schmidt, Tom Siegl, Martin Becker+1

Artificial Intelligence Lab, Otto-von-Guericke University Magdeburg, Germany · University of Rostock, Germany · Becker Lab, University of Marburg, Germany

Distribution Shift Robustness

Open paper AskPDF arXiv

Pretrained vision models often misclassify inputs that are rotated, scaled, or sheared, even though these affine transformations leave the object class unchanged. Robustness is usually restored either by building equivariance into the architecture or by retraining with augmentation, both of which require changing or retraining the model. Test-time canonicalization instead leaves the classifier untouched. It undoes the transformation of each input, mapping it to a canonical form near the training distribution before classification. Existing canonicalizers, however, rely on a narrow set of logit-based energy scores and bespoke search procedures, leaving the design space of scoring functions and optimizers unexplored. We reframe canonicalization as out-of-distribution (OOD) detection, which lets any OOD score serve as the energy minimized over transformations. Across benchmarks ranging from handwritten characters and sketches to natural images and 3D point clouds, we systematically evaluate around twenty OOD scores and nine search algorithms, finding that distance-based scores paired with random search and local refinement perform best overall. Because canonicalizing an already-aligned input can hurt accuracy, we add a gated mechanism that transforms an input only when its OOD score indicates this is needed, preserving most in-distribution accuracy while retaining the robustness gains on transformed inputs. Code is available at github.com/johschm/its.

Abstract

Distribution Shift Robustness

Explore similar work

Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring

LexiCan: Lexicographic Canonicalization of Point Clouds for Machine Learning Applications

COSTA: A Cluster-Centric Paradigm for Annotation-Free Open-Set Semantic Segmentation of Aerial Point Clouds with Domain Shifts