Uncertainty Gates the Gradient: Adaptive Updates for Generalized Few-Shot 3D Segmentation

Yifei Zhao; Fanyu Zhao; Shengtang Wu; Han Zhang; Ruike Cao; Yinsheng Li

cs.CVOct 17, 2026

Uncertainty Gates the Gradient: Adaptive Updates for Generalized Few-Shot 3D Segmentation

Authors: Yifei Zhao, Fanyu Zhao, Shengtang Wu, Han Zhang, Ruike Cao, Yinsheng Li

Organizations: Fudan University · University of Science and Technology of China

Abstract

In generalized few-shot 3D point cloud segmentation, support annotations are often compromised by sparsity and occlusion. Blindly adapting to such low quality samples risks learning spurious geometric patterns rather than robust novel concepts. Although 3D Vision-Language Models have been introduced to sup plement supervision, the potential of uncertainty to actively govern adaptation dynamics remains underexplored. Existing signal-level methods like loss reweight ing fail to prevent unreliable episodes from steering the optimization trajectory. To address this, we propose Uncertainty-Gated Adaptive (UGA), to the best of our knowledge the first framework that elevates uncertainty to update-level gating for generalized few-shot 3D point cloud segmentation. The core mod ule, Uncertainty-Gated Update (UG-Update), estimates episodic reliability via intra-class dispersion over merged support and pseudo-labeled novel supervision, and selectively gates residual gradient updates. This encourages the model to learn novel semantics from reliable episodes while attenuating updates during unreliable ones. Complementing this, we introduce an Uncertainty-Based Pseudo Label Filter (UB-Filter) to enforce learning from high-confidence dense signals. On ScanNet200 and ScanNet++ (1-/5-shot), UGA sets new state of the art with an average +3.08 HM IoU over the strongest prior baseline.

Figures & tables

Figure 1: Blind adaptation vs. robust adaptation.

Figure 2: Phase 2 UGA schematic: UB-Filter (left) filters dense VLM pseudo labels; UG-Update (right) gates the residual gradient $\mathbf{g}_{\perp}$ . Full description is shown in Sec. 3.1 .

Method	ScanNet200								ScanNet++
	5-shot				1-shot				5-shot				1-shot
	B	N	A	HM	B	N	A	HM	B	N	A	HM	B	N	A	HM
Fully Supervised	68.70	39.32	45.51	50.02	68.70	39.32	45.51	50.02	65.45	37.24	48.53	47.47	65.45	37.24	48.53	47.47
PIFS [ 4 ]	28.78	3.82	9.07	6.71	17.84	2.87	6.02	4.88	39.98	5.74	19.44	10.03	36.66	4.95	17.63	8.71
attMPTI [ 40 ]	37.13	4.99	11.76	8.79	54.84	3.28	14.14	6.17	55.89	4.19	24.87	7.78	53.16	3.55	23.40	6.66
COSeg [ 3 ]	57.67	5.21	16.25	9.54	47.03	4.03	13.09	7.42	59.34	6.96	27.91	12.45	58.49	6.24	27.14	11.26

Table 1: Results on ScanNet/++ (1/5-shot). Best: bold ; second: underline .

Figure 3: Per-scene qualitative results on $5$ ScanNet200 episodes, $\bar{u}_{\mathrm{e}}$ (left bar) and gate $\bar{s}$ (right bar; whiskers $\pm 1\sigma$ over 5 runs). Red boxes highlight regions where UGA suppresses mutual base–novel cross-contamination.

(a) Components on/off
UG-Update	UB-Filter	Gate proxy	B	N	A	HM
		–	67.57	31.67	39.23	43.12
✓		dispersion	68.06	32.00	39.59	43.53
	✓	–	68.81	32.34	40.02	44.00
(b) Both components active; varying UG-Update gate proxy
✓	✓	VLM Agg. Entropy	68.42	31.87	39.57	43.49

Table 2: Module ablation on ScanNet200 5-shot, mIoU (%). (a) Components on/off (UG-Update gate proxy = our default intra-class dispersion). (b) Both components active; varying the UG-Update gate proxy. Bold : best per column; underline : second; plain: third or below.

Figure 4: UG-Update: (a) gradient visualization; (b) gate $s$ vs. $u_{\mathrm{e}}$ ; (c) gated decomposition.

Figure 5: Visualization of pseudo-label refinement via UB-Filter. Each row shows a query scene.

Figure 6: UB-Filter. Class-mean uncertainty before ( $x$ ) vs. after ( $y$ ); color=IoU; red $y{=}x$ .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	B	N	A	HM
GFS-VL (Reproduced) ( $\lambda{=}0$ endpoint)	67.64	28.80	36.98	40.39
OGD [ 8 ] ( $\Delta\phi=-\eta g_{\perp}$ ; dual of $\lambda\!\to\!\infty$ )	68.84	29.22	37.56	41.03
UGA-Margin (ours)	69.14	29.25	37.65	41.11
UGA-Max (ours)	69.30	30.16	38.40	42.02
UGA-Entropy (ours)	68.89	32.14	39.88	43.83

Appendix

Table 3: Empirical comparison of the plain endpoint (

\lambda{=}0

, GFS-VL) and a faithful OGD baseline (geometric dual of UGA’s

\lambda\!\to\!\infty

limit; not a member of our proximal family) against UGA’s finite, episode-adaptive

\lambda(u_{\mathrm{e}})

on ScanNet200 (1-shot). Results in mIoU (%). Best: bold ; second: underline .

Input: Controlled parameters

\phi

(subset

\mathcal{P}_{\mathrm{UG}}

), fixed basis

B

, batch embeddings

\{f_{i}\}

and labels

\{y_{i}\}

, gate slope

k

, threshold

\tau

Output: Controlled gradient

g_{\mathrm{UG}}

applied to

\mathcal{P}_{\mathrm{UG}}

1. Eligible novel classes. Construct

\mathcal{C}

as in Eq. ( 1 ).

2. Prototypes. For each

c\in\mathcal{C}

, compute

\mu_{c}

using Eq. ( 1 ).

3. Episodic uncertainty. Compute raw uncertainty

u_{\mathrm{e,raw}}

by feature dispersion (Eq. ( 2 )); if enabled, normalize to

u_{\mathrm{e}}

via Eq. ( 5 ).

4. Backprop gradient. Compute

g\leftarrow\nabla_{\phi}\mathcal{L}_{\mathrm{seg}}

5. Decompose gradient. Project onto the base-gradient subspace (Eq. ( 9 )):

\hskip 18.49988ptg_{\parallel}\leftarrow B(B^{\top}g),\hskip 18.49988ptg_{\perp}\leftarrow g-g_{\parallel}.

Appendix

Table 4: Algorithm: Uncertainty-Gated Residual Update ( UG-Update , Phase 2).

Input: Per-point VLM probabilities

\{p(x)\}

(Eq. ( 11 )), novel class set

\mathcal{Y}_{\mathrm{novel}}

, retention ratio

\rho

, uncertainty function

u(\cdot)

Output: Filtered pseudo label map

\tilde{y}_{\mathrm{vlm}}(\cdot)

(uncertain points mapped to ignore ).

1. Pseudo labeling. For each point

x

, set

\hat{y}(x)\leftarrow\arg\max_{k}p_{k}(x)

2. Uncertainty scoring. For each point

x

, compute

u(x)

using one choice among: entropy (Eq. ( 12 )), margin (Eq. ( 13 )), or max-probability (Eq. ( 14 )).

3. Initialize output. Set

\tilde{y}_{\mathrm{vlm}}(x)\leftarrow\hat{y}(x)

for all points

x

4. Class-wise thresholds (novel only). For each

c\in\mathcal{Y}_{\mathrm{novel}}

\hskip 18.49988pt\mathcal{X}_{c}\leftarrow\{x\mid\hat{y}(x)=c\},\hskip 18.49988ptq_{c}\leftarrow\mathrm{Q}_{\rho}\big(\{u(x)\mid x\in\mathcal{X}_{c}\}\big)\hskip 9.24994pt(\text{Eq.~\eqref{eq:quantile}}).

5. Filtering. For each

x\in\mathcal{X}_{c}

: if

u(x)>q_{c}

, set

\tilde{y}_{\mathrm{vlm}}(x)\leftarrow\texttt{ignore}

Appendix

Table 5: Algorithm: Class-wise Quantile Filtering ( UB-Filter , Phase 2).

Phase 1: base pretraining
Setting	ScanNet200	ScanNet++
Backbone	PTv3	same
Optimizer	AdamW	same
Learning rate	$6\times 10^{-3}$	same
Weight decay	$5\times 10^{-2}$	same
Phase 2: adaptation (UGA enabled)

Appendix

Table 6: Key settings for reproducing Phase 1 pretraining and Phase 2 adaptation.

Setting	Method	HM
ScanNet200 5-shot	GFS-VL (Reproduced)	$42.92{\pm}0.49$
ScanNet200 5-shot	UGA-Entropy	$45.65{\pm}0.48$
ScanNet200 1-shot	GFS-VL (Reproduced)	$40.39{\pm}0.75$
ScanNet200 1-shot	UGA-Entropy	$43.83{\pm}0.51$
ScanNet++ 5-shot	GFS-VL (Reproduced)	$31.93{\pm}2.29$
ScanNet++ 5-shot	UGA-Entropy	$36.78{\pm}1.91$

Appendix

Table 7: Five-run HM variability for the reproduced baseline and default UGA-Entropy. Entries are mean

\pm

sample std in HM mIoU (%) over regis1 – regis5 .

Setting	Method	B	N	A	HM
1-shot	GFS-VL	78.56	49.72	69.45	60.88
1-shot	UGA	79.84	51.60	70.92	62.69
5-shot	GFS-VL	78.30	51.22	69.75	61.91
5-shot	UGA	79.22	53.17	70.99	63.63

Appendix

Table 8: ScanNet with 13 base and 6 novel classes. Results in mIoU (%). Bold : best per column.

Table 15

Figure 7: Hyper-parameter sensitivity on ScanNet200:

k

(left, 5-shot),

\tau

(middle, 5-shot), and Phase-2 batch size (right, 1-shot). Shaded regions denote 95% confidence intervals over 5 runs. Dashed lines indicate the GFS-VL [ 1 ] HM baselines.

Figure 8: Zoom-in qualitative comparison with base and novel predictions visualized separately; purple/red circles mark base/novel errors.

3D Point Cloud Segmentation

3D point cloud segmentation assigns semantic, instance, or part labels to points for scene understanding, mapping, robotics, and scientific measurement.

See all 68 papers

Learning Hierarchical Orthogonal Prototypes for Generalized Few-Shot 3D Point Cloud Segmentation
Mar 20, 2026Yifei Zhao, Fanyu Zhao, Zhongyuan Zhang +33D Point Cloud Segmentation 3D Semantic Segmentation
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
Uncertainty-aware Prototype Learning with Variational Inference for Few-shot Point Cloud Segmentation
Mar 20, 2026Yifei Zhao, Fanyu Zhao, Yinsheng LiFew-Shot Semantic Segmentation 3D Point Cloud Segmentation
Fudan University, Shanghai, China
SCOPE: Scene-Contextualized Incremental Few-Shot 3D Segmentation
Mar 6, 2026Vishal Thengane, Zhaochong An, Tianjin Huang +5Few-Shot Semantic Segmentation 3D Semantic Segmentation
University of Surrey, UK · University of Wollongong, Australia · University of Copenhagen, Denmark +2

Mar 20, 2026J/K move · Enter open · S save

Learning Hierarchical Orthogonal Prototypes for Generalized Few-Shot 3D Point Cloud Segmentation

Yifei Zhao, Fanyu Zhao, Zhongyuan Zhang, Shengtang Wu+2

College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China

3D Point Cloud Segmentation 3D Semantic Segmentation Few-Shot Semantic Segmentation Prototype Learning

Open paper AskPDF arXiv

Generalized few-shot 3D point cloud segmentation aims to adapt to novel classes from only a few annotations while maintaining strong performance on base classes, but this remains challenging due to the inherent stability-plasticity trade-off: adapting to novel classes can interfere with shared representations and cause base-class forgetting. We present HOP3D, a unified framework that learns hierarchical orthogonal prototypes with an entropy-based few-shot regularizer to enable robust novel-class adaptation without degrading base-class performance. HOP3D introduces hierarchical orthogonalization that decouples base and novel learning at both the gradient and representation levels, effectively mitigating base-novel interference. To further enhance adaptation under sparse supervision, we incorporate an entropy-based regularizer that leverages predictive uncertainty to refine prototype learning and promote balanced predictions. Extensive experiments on ScanNet200 and ScanNet++ demonstrate that HOP3D consistently outperforms state-of-the-art baselines under both 1-shot and 5-shot settings. The code is available at https://fdueblab-hop3d.github.io/.

Abstract

3D Point Cloud Segmentation

Explore similar work

Learning Hierarchical Orthogonal Prototypes for Generalized Few-Shot 3D Point Cloud Segmentation

Uncertainty-aware Prototype Learning with Variational Inference for Few-shot Point Cloud Segmentation

SCOPE: Scene-Contextualized Incremental Few-Shot 3D Segmentation