What Do Scan-Derived Class Prototypes Add? Disentangling Supervision, Prototype Content and Query Protocol in Recognition over Frozen Foundation Features

Chenxi Tao; Hong-In Won; Seung-Kyum Choi

cs.CVSep 3, 2026

What Do Scan-Derived Class Prototypes Add? Disentangling Supervision, Prototype Content and Query Protocol in Recognition over Frozen Foundation Features

Authors: Chenxi Tao, Hong-In Won, Seung-Kyum Choi

Organizations: George W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, 30332, GA, USA

Abstract

A scan supplies labeled images and a geometric reference. We separate their contributions in a recognizer whose scan-derived prototype matrix acts as a supervised head's fixed output layer. On T-LESS, HOPE and 18 self-collected industrial parts, we test real, random and exactly permuted prototypes, matched geometry-free classifiers, stronger appearance rules and paired background protocols. Across DINOv2-giant and MetaCLIP-H with real-background queries, the largest fused-accuracy advantage of the real prototypes over either control is one percentage point; larger differences favor controls, by up to 2.8 points in arm means. On HOPE with DINOv2-giant the head alone is 2.8 points above exact permutations (95% interval: 0.8-4.7); this advantage does not reach fusion and is not observed on MetaCLIP-H. On DINOv2-giant, matched logistic regression comes within 0.5 points of fusion on T-LESS and exceeds it on HOPE and the self-collected parts. Against white cutouts, real HOPE query backgrounds lower image-prototype accuracy by 43 points on DINOv2-giant and 13 on MetaCLIP-H. The audit separates prototype content, label supervision and query protocol.

Figures & tables

Method	Query and reference	Decision path / object-specific training
CNOS, SAM-6D	RGB; CAD-rendered templates. SAM-6D also uses depth for pose.	Frozen template matching; none.
FoundPose	RGB; CAD RGB-D templates.	Template retrieval, 2D–3D correspondences and PnP; none.
GFreeDet	RGB; Gaussian-Splatting-rendered templates.	Frozen template matching; none.
FreeZe	RGB-D; model point cloud with frozen geometric and visual features.	Registration of query and model clouds; none.
ULIP, OpenShape	Point cloud or image; joint image–text–point-cloud embedding.	Retrieval in the joint embedding; none per object.
Audited system	RGB; per-class $45$ -D scan descriptors ( $G$ ).	$G$ is a fixed classifier output layer; head $h$ learns from labeled onboarding frames.

Table 1: Reference geometry and object-specific supervision. A class permutation directly tests prototype assignment in the final row.

Figure 1: The audited computation. (A) Labeled onboarding scan of $60$ – $160$ frames per object, without CAD. (B) Frozen image encoder $f$ and one mean image prototype per class (blue star); the query $x$ is an RGB crop with its background kept. (C) 3DGS reconstruction, point cloud and $45$ -D descriptor per class, stacked into the fixed prototype matrix $G$ (orange star). (D) The head $h$ , the only trained module, fitted on the scan labels with logits $z^{\top}g_{c}/0.10$ . (E) Fusion of the image score with $0.35$ times the geometry score. (F) Top-1 class. The controls act at $G$ (Sections 4.1 and 4.5 ), on lanes C–E (Section 4.2 ), on the image prototype (Section 4.3 ), on the query crop (Section 4.4 ) and on $f$ (Section 4.5 ).

Dataset (classes)	Onboarding	Per class	Queries
T-LESS ( $28$ )	$4{,}480$	$160$	$6{,}395$
HOPE ( $28$ )	$1{,}680$	$60$	$915$
FGIP ( $18$ )	$2{,}839$	$\leq 160$	$1{,}927$

Table 2: Datasets. Onboarding uses foreground crops; queries retain the scene within the object box. FGIP queries come from an independent video.

Figure 2: Onboarding references for six objects each of T-LESS (top), HOPE (middle) and FGIP (bottom). T-LESS rows: a segmented onboarding crop with the foreground on white (labeled “real crop”), a novel view rendered from the 3DGS reconstruction, and the extracted point cloud. HOPE rows: a segmented onboarding crop and the extracted point cloud. FGIP rows: a segmented frame of the onboarding video on white and the extracted point cloud. The FGIP block shows two gears, two nuts and two screws of the $18$ parts: gears A and B, nuts A and B, screws A and B. T-LESS clouds carry the Gaussian colors; the other clouds are drawn in one color. All images are onboarding data. The point cloud is the only reconstruction product that enters the class descriptors (Section 3.2 ).

Coverage	Views	Minutes	GB
One equatorial ring	$32$	$2.41$	$0.14$
Three rings	$96$	$2.66$	$0.15$
Five rings (main)	$160$	$2.83$	$0.17$

Table 3: T-LESS reconstruction cost per object (

28

objects, splatfacto-big ,

20

k iterations, one NVIDIA RTX 5000 Ada). Each fixed-elevation ring contains

32

azimuth views.

Dataset	Output	Real	Random	Permuted
T-LESS	Image-only	$0.560$
	head	$0.566{\pm}0.015$	$0.560{\pm}0.009$	$0.557{\pm}0.013$
	fused	$0.591{\pm}0.004$	$0.588{\pm}0.006$	$0.586{\pm}0.007$
	best_val	$0.999{\pm}0.001$	$0.998{\pm}0.001$	$0.999{\pm}0.001$
HOPE	Image-only	$0.399$
	head	$0.459{\pm}0.014$	$0.429{\pm}0.013$	$0.431{\pm}0.016$

Table 4: DINOv2-giant: audited system and prototype controls on real-background queries (top-1, mean

\pm

SD). Real:

5

head seeds; random:

5

prototype matrices

\times

3

head seeds; permuted:

5

exact row permutations

\times

5

head seeds. Fusion uses

\lambda=0.35

; best_val is held-out onboarding accuracy.

Dataset	Level	Real $-$ random [95%]	Real $-$ permuted [95%]
T-LESS	head	$+0.6$ [ $-1.3$ , $+2.4$ ]	$+0.8$ [ $-1.0$ , $+2.7$ ]
	fused	$+0.3$ [ $-0.5$ , $+1.1$ ]	$+0.5$ [ $+0.1$ , $+0.8$ ]
HOPE	head	$+3.0$ [ $+1.0$ , $+5.0$ ]	$+2.8$ [ $+0.8$ , $+4.7$ ]
	fused	$-0.3$ [ $-1.1$ , $+0.5$ ]	$-0.3$ [ $-0.6$ , $+0.1$ ]
FGIP	head	$+1.8$ [ $-2.2$ , $+5.8$ ]	$+2.0$ [ $-2.2$ , $+6.1$ ]
	fused	$-0.8$ [ $-2.3$ , $+0.7$ ]	$-1.3$ [ $-2.3$ , $-0.4$ ]

Table 5: DINOv2-giant: real minus control, in percentage points with

95\%

intervals. Permutations are paired by head seed; random-matrix intervals combine between-matrix and real-head seed variation (

n=5

4

degrees of freedom).

Figure 3: Prototype controls on DINOv2-giant with real-background queries, for T-LESS, HOPE and FGIP (columns). Top row: top-1 accuracy of the head alone; bottom row: fused accuracy at

\lambda=0.35

. Each point is one training run and each bar the mean of its arm: real prototypes (blue,

5

seeds), random prototypes (orange,

5

matrices

\times

3

seeds) and exact row permutations

P_{k}G_{s}

(purple,

5

permutations

\times

5

seeds). Green markers right of the dotted line are geometry-free references on the same features and labels, repeated in both rows: logistic regression (LR) and softmax head trained on all onboarding frames (circle, square) and on the matched subset (triangles; mean and SD over

5

seeds). The dashed line is image-only accuracy.

Queries	Training / classifier	T-LESS	HOPE	FGIP
Real	Matched LR	$0.586{\pm}0.002$	$0.433{\pm}0.002$	$0.916{\pm}0.002$
	Matched softmax	$0.529{\pm}0.006$	$0.421{\pm}0.012$	$0.947{\pm}0.004$
	Full LR	$0.585$	$0.454$	$0.921$
	Full softmax	$0.522{\pm}0.003$	$0.409{\pm}0.007$	$0.948{\pm}0.003$
	Geometry head	$0.566{\pm}0.015$	$0.459{\pm}0.014$	$0.904{\pm}0.026$
	Fused	$0.591{\pm}0.004$	$0.421{\pm}0.003$	$0.860{\pm}0.007$

Table 6: Supervision-only references on DINOv2-giant (top-1, mean

\pm

SD over

5

seeds where supplied). Matched uses the head optimization subset and, for softmax, its validation selection rule; full budget uses all onboarding frames and the final softmax epoch. W denotes white-cutout queries; other rows use real backgrounds.

Figure 4: Appearance rules with and without the geometry head on DINOv2-giant, real-background queries, for T-LESS, HOPE and FGIP (top-1 accuracy). Three rules are shown: mean prototype, best single template, and final-layer patch bag-of-words with best-five matching. Light blue bars give the rule alone. Orange bars give the rule fused with the real geometry head at a weight calibrated on held-out onboarding frames (mean and SD over

5

head seeds). Black bars give the head alone and green bars full-budget logistic regression without geometry. The label “

\lambda=0

3/5

” marks a rule whose calibrated weight is zero in three of five seeds. Table 7 lists all nine rules.

T-LESS
Rule	Alone	$+$ head
Mean prototype	$0.560$	$0.594{\pm}0.008$
Best template	$0.581$	$0.611{\pm}0.006$
Mean of best five	$0.584$	$0.610{\pm}0.006$
$k$ -means ( $8$ centers)	$0.578{\pm}0.001$	$0.607{\pm}0.007$
Soft vote, all templates	$0.575$	$0.604{\pm}0.007$

Table 7: Appearance rules with real-background queries: alone and fused with the real geometry head (top-1; fused mean

\pm

SD,

5

seeds). Each rule calibrates its fusion weight on held-out onboarding frames. HOPE

k

-means alone is

0.410

at seed

0

and ranges from

0.402

0.423

. FGIP mean-prototype features were re-encoded, giving

0.804

instead of

0.803

. Geometry-only and full-budget LR references are in Table 6 .

	Alone		$+$ real head		$+$ random
HOPE rule	White	Real	White	Real	Real
Mean prototype	$0.832$	$0.399$	$0.908$	$0.448$	$0.440$
MetaCLIP mean	$0.879$	$0.745$	—	—	—
Best template	$0.863$	$0.408$	$0.925$	$0.463$	$0.461$
Mean of best five	$0.882$	$0.420$	$0.930$	$0.468$	$0.468$
$k$ -means ( $8$ )	$0.861$	$0.410$	$0.924$	$0.457$	$0.458$

Table 8: Paired query protocols. White pastes the masked foreground on a white canvas; Real keeps the scene content of the box (Section 3.1 ). HOPE uses the same

915

boxes; calibrated fusion uses

5

real-head seeds or

5

random-matrix heads (one seed each). The HOPE white

k

-means result is

0.861\pm 0.011

. FGIP parentheses count correct screw A/B queries. T-LESS uses real backgrounds throughout; its corresponding results are in Tables 6 and 7 .

Figure 5: Paired query protocols on DINOv2-giant features (top-1 accuracy). (a, b) The same

915

HOPE query boxes scored as white cutouts and with the scene background kept. (a) Rules alone: the mean prototype (blue) and final-layer patch bag-of-words with best-five matching (vermilion) are labeled; gray lines are the other seven rules. References: full-budget logistic regression (green), head alone (black), fixed fusion at

\lambda=0.35

(orange) and the MetaCLIP-H mean prototype (dotted purple). (b) The nine rules fused with the real geometry head at the onboarding-calibrated weight (mean of

5

seeds). (c) FGIP, the same

1{,}927

frames: mean-prototype and best-template rules on all classes (solid) and on the screw A/B pair (dashed).

Dataset	Level	Real	Random	Permuted
T-LESS	image-only	$0.559$ / $0.560$
	head	$0.589{\pm}0.012$	$0.569{\pm}0.020$	$0.578{\pm}0.013$
	fused	$0.613{\pm}0.007$	$0.604{\pm}0.016$	$0.612{\pm}0.010$
	best_val	$0.995$	$0.992$	$0.992$
HOPE	image-only	$0.745$ / $0.399$
	head	$0.760{\pm}0.015$	$0.782{\pm}0.011$	$0.781{\pm}0.034$

Table 9: MetaCLIP-H prototype audit (top-1, mean

\pm

SD, real backgrounds;

\lambda=0.35

). Real uses

5

head seeds; random uses

5

prototype matrices at head seed

0

; permuted uses

5

exact permutations

P_{k}G_{0}

at head seed

0

. Image-only gives MetaCLIP-H / DINOv2-giant; best_val is held-out onboarding accuracy.

Figure 6: Prototype controls on MetaCLIP-H with real-background queries, for T-LESS, HOPE and FGIP (columns; the HOPE axis is broken). Top row: top-1 accuracy of the head alone; bottom row: fused accuracy at

\lambda=0.35

. Each point is one training run and each bar the mean of its arm: the real prototype matrix (blue,

5

head seeds),

5

random prototype matrices (orange) and

5

exact permutations

P_{k}G_{0}

(purple), both at head seed

0

. Green markers right of the dotted line are geometry-free references on MetaCLIP-H features: logistic regression (LR) and softmax head trained on all onboarding frames (circle, square) and on the matched subset (triangles; mean and SD over

5

seeds). The dashed line is MetaCLIP-H image-only accuracy. Short gray lines give the DINOv2-giant arm means under the same design.

Dataset	Level	random	permuted	random	permuted
		Real minus, over runs ( $t$ )		Real minus, seed $0$
T-LESS	head	$+2.0$ ( $1.9$ )	$+1.1$ ( $1.4$ )	$+1.1$	$+0.3$
	fused	$+0.8$ ( $1.1$ )	$+0.0$ ( $0.0$ )	$+0.5$	$-0.4$
HOPE	head	$-2.2$ ( $-2.6$ )	$-2.1$ ( $-1.3$ )	$-0.9$	$-0.8$
	fused	$-2.8$ ( $-10.6$ )	$-2.2$ ( $-7.2$ )	$-3.0$	$-2.4$
FGIP	head	$+1.0$ ( $0.9$ )	$+2.0$ ( $1.3$ )	$-1.4$	$-0.3$

Table 10: MetaCLIP-H: real minus control in percentage points. Run comparisons give Welch

t

in parentheses; the last two columns match head seed

0

MetaCLIP-H, matched optimization subset
Reference / arm	T-LESS	HOPE	FGIP
Logistic regression	$0.609{\pm}0.004$	$0.794{\pm}0.009$	$0.834{\pm}0.005$
Softmax head	$0.580{\pm}0.007$	$0.813{\pm}0.005$	$0.826{\pm}0.006$
Geometry head	$0.589{\pm}0.012$	$0.760{\pm}0.015$	$0.832{\pm}0.020$
DINOv2-giant: head / fused
Real ( $5$ seeds)	$0.566$ / $0.591$	$0.459$ / $0.421$	$0.904$ / $0.860$

Table 11: Matched references on MetaCLIP-H (mean

\pm

SD,

5

seeds) and the second-backbone control design applied to DINOv2-giant. DINO real uses

5

head seeds; random and exact permutations use head seed

0

. All queries retain their backgrounds; fusion uses

\lambda=0.35

Assignment (runs)	Test	Train
Semantic ( $5$ )	$0.571{\pm}0.003$	$0.822$
Permuted ( $5\times 5$ )	$0.500{\pm}0.003$	$0.789$

Table 12: HPN on CIFAR-100: top-1 accuracy of the semantic class assignment and of its derangements. Test entries are mean

\pm

SD over five training seeds; permuted runs are first averaged over the five permutations within a seed. Train: unaugmented training-set accuracy at the final epoch.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Prototype-matrix diagnostics on T-LESS, HOPE and FGIP. (a) Mean absolute cosine between class rows of

G

(left) and effective rank of

G

out of

45

(right): real prototype matrix (blue diamond, value labeled), five random prototype matrices (orange points) and exact permutations (open purple diamond, identical to real by construction). T-LESS and HOPE share the random matrices. (b) Fused top-1 accuracy on the test queries (DINOv2-giant, real backgrounds) against the fusion weight

\lambda

: mean over

5

real-matrix runs (blue, solid) and

15

random-matrix runs (orange, dashed). Labels give image-only accuracy at

\lambda=0

and the real-matrix value at

\lambda=1

. The dotted line marks

\lambda=0.35

; the grid is scored on test labels as a diagnostic.

Encoder	Image-only	Geometry head	Fused
SigLIP2-g †	$0.561$	$0.611{\pm}0.010$	$0.629{\pm}0.004$
MetaCLIP-H	$0.559$	$0.589{\pm}0.010$	$0.613{\pm}0.006$
SigLIP-L	$0.545$	$0.555{\pm}0.002$	$0.592{\pm}0.005$
DINOv2-giant	$0.560$	$0.566{\pm}0.013$	$0.591{\pm}0.004$
SigLIP2-so400m	$0.531$	$0.553{\pm}0.013$	$0.586{\pm}0.007$
OpenCLIP-H	$0.546$	$0.516{\pm}0.016$	$0.578{\pm}0.011$

Appendix

Table 1: T-LESS frozen-backbone comparison, sorted by fused accuracy (

\lambda=0.35

; mean

\pm

SD over

5

seeds;

{}^{\dagger}n=4

). Image-only is deterministic. SDs in this table use the divisor

n

; Tables 4 and 9 use

n-1

, which changes the last digit. Feature families and checkpoints are listed below the table.

Pair details: raw RGB (left); assisted CAD / 3DGS (right)
Arm	Pose-free	Pose-assisted
RGB patch BoW	$0.671$	—
CAD surface + query depth	$0.486$	$0.824$
3DGS scan + query depth	$0.492$	$0.833$
Onboarding depth + query depth	$0.503$	$0.774$
CAD box + query depth	$0.486$	$0.644$

Appendix

Table 1: Measured-depth pilot on T-LESS: object-balanced two-way accuracy,

273

development queries across six pairs;

15

unusable-depth queries count as wrong. Geometry receives the true visibility mask; pose-assisted arms also receive the true identity pose. RGB receives only the raw crop. Perturbation summaries use

87

usable-depth queries from pairs

25/26

and

21/22

Explore similar work

Class Geometry as Supervision for Sample-Efficient Open-World Detection
Aug 13, 2026Akash Rao, Zhou Chen, Revanth Reddy Palem +3Open-Vocabulary Object Detection Open-Set Recognition
CSSE Department, Auburn University, Auburn, AL, 36849 · Department of Veterinary Pathobiology, Oklahoma State University, Stillwater, OK, 74078
Beyond Points: Spherical Distributional Part Prototypes for Interpretable Classification
Jun 25, 2026Duarte Leão, Diogo Pereira Araújo, Catarina Barata +1Learnable Prototypes Gradient-Weighted Class Activation Mapping
Instituto Superior Técnico, Universidade de Lisboa · Carnegie Mellon University
Can Frozen Hyperspherical Features Guide the Selection of Pseudo Masks?
Sep 24, 2026Xinge Guo, Fengyang Xiao, Dingming Zhang +6Heterogeneous Modeling Tools
Duke University · Carnegie Mellon University · University of Alabama at Birmingham

Aug 13, 2026, cs.CVJ/K move · Enter open · S save

Class Geometry as Supervision for Sample-Efficient Open-World Detection

Akash Rao, Zhou Chen, Revanth Reddy Palem, Udhav Ramachandran+2

CSSE Department, Auburn University, Auburn, AL, 36849 · Department of Veterinary Pathobiology, Oklahoma State University, Stillwater, OK, 74078

Open-Vocabulary Object Detection Open-Set Recognition Learnable Prototypes Object Detection Geometric Constraints

Open paper AskPDF arXiv

Open-world object detection requires models to recognize known categories, reject unfamiliar objects, and incorporate new classes over time. This is especially challenging in scarce-data settings such as biomedical and scientific imaging, where rare categories may have only a few annotated examples and fine-grained classes differ by subtle morphology. Prototype-based detectors are natural for this regime, but they typically learn class prototypes as independent anchors, ignoring relational structure among classes. We propose class-geometry supervision (CGS), a general framework that constrains learned prototype or class-representation spaces to preserve visual or semantic class dissimilarities estimated from training data. CGS introduces a dissimilarity-preserving objective that aligns pairwise distances among learned class representations with a target class-geometry matrix while retaining the standard task loss. We instantiate the same objective across prototype recognition, few-shot biomedical object detection, open-set detection, novel-class insertion, and OWOD adaptation on COCO. Experiments show that CGS improves sample efficiency in recognition and ova detection, substantially strengthens novel-class insertion, and improves unknown recall on COCO while retaining much of the known-class detection performance. Ablations show that meaningful visual geometry provides the most reliable gains, while random geometry can help novel separation but is less consistent for few-shot detection. These results suggest that relational class geometry is an effective supervisory signal for building calibrated and extensible open-world detectors under limited supervision.

Abstract

Explore similar work

Class Geometry as Supervision for Sample-Efficient Open-World Detection

Beyond Points: Spherical Distributional Part Prototypes for Interpretable Classification

Can Frozen Hyperspherical Features Guide the Selection of Pseudo Masks?