Stochastic Nonlinearities Improve Uncertainty Estimation

Mohammad Sarhangzadeh; Amirreza M. Shebly; Doruk Oner

cs.CVOct 21, 2026

Stochastic Nonlinearities Improve Uncertainty Estimation

Authors: Mohammad Sarhangzadeh, Amirreza M. Shebly, Doruk Oner

Organizations: NeuraVision Lab, Bilkent University Ankara, Turkey

Abstract

Reliable uncertainty estimation is important across visual recognition tasks, providing information about the confidence and reliability of model predictions. Many uncertainty estimation methods obtain predictive diversity through mechanisms that can suppress intermediate features, perturb the input, or require substantial additional computation. Stochastic activation functions, where network units randomly select between multiple nonlinearities during a forward pass, have previously been used to improve optimization behavior, mitigate limitations of fixed ReLU activations, and increase output diversity. In this work, we reframe stochastic activation switching as a lightweight mechanism for uncertainty estimation. The proposed method samples between nonlinear activation functions, inducing predictive diversity while keeping intermediate feature pathways active. This enables uncertainty estimation from a single trained model without the training cost of ensembles, task-specific uncertainty heads, or architectural redesign. We evaluate the proposed approach across classification, semantic segmentation, and out-of-distribution detection using convolutional and transformer-based architectures on diverse datasets. Across these settings, stochastic activation switching improves uncertainty quality while preserving competitive primary-task performance, showing that stochastic nonlinearities provide an effective and information-preserving source of predictive variation for visual recognition.

Figures & tables

Figure 1: Uncertainty from stochastic nonlinearities. Stochastic forward passes are shown for a network trained to predict road maps. Clear roads produce similar outputs and low variance, while ambiguous roads and road-like buildings produce higher variance.

Figure 2: Uncertainty quality under test-time stress. The left panel shows the input, prediction, ground truth, and errors; the right panels compare uncertainty maps from stochastic activation switching and MC Dropout, with brighter values indicating higher uncertainty. We increase stochasticity only at inference time to examine its effect on uncertainty. MC Dropout removes intermediate features, causing uncertainty to spread broadly, whereas stochastic activation switching keeps feature pathways active and concentrates uncertainty near boundaries and prediction errors. This supports our motivation that activation-level stochasticity can induce predictive diversity without disrupting intermediate representations.

Figure 3: Toy regression under heteroscedastic noise. A 3-layer MLP is trained on noisy sinusoidal samples, and stochastic activation switching is compared with MC-Dropout under the same setup. Both methods use repeated stochastic passes. Our method shows clearer combined predictive uncertainty, remaining more certain in well-supported regions while uncertainty increases with stronger noise and outside the observed range.

Figure 4: Visualization of our stochastic activation based uncertainty estimation. From left to right: input image, ground‐truth mask, model prediction, and the uncertainty map computed from the variance across $5$ stochastic activation trajectories.

Dataset	Method	Correct. (↑)	Complete. (↑)	Qual. (↑)	F1 (↑)
Massachusetts	MC Dropout	91.00	87.20	81.90	78.62
	DDU	91.44	86.10	80.52	79.23
	TTA	91.46	86.73	81.19	79.43
	Masksembles	81.63	87.67	81.94	80.02
	Deep Ensemble	92.37	88.11	82.67	80.54
	Laplace Redux	91.45	86.11	80.52	79.23

Table 1: Segmentation performance across aerial and medical benchmarks.

Dataset	Method	Corr. (↑)	ECE (↓)	Brier (↓)	Train	Infer
Massachusetts	MC Dropout	79.73	0.026	0.033	$\times 1$	$\times 5$
	DDU	75.64	0.027	0.031	$\times 1$	–
	TTA	79.90	0.025	0.029	$\times 1$	$\times 5$
	Masksembles	79.62	0.024	0.031	$\times 1$	$\times 5$
	Deep Ensemble	80.03	0.024	0.027	$\times 5$	$\times 5$
	Laplace Redux	77.51	0.034	0.034	$\times 1$	–

Table 2: Segmentation uncertainty, calibration, and relative computational cost.

Method	Mass. $\rightarrow$ RoadTracer		DRIVE $\rightarrow$ CHASE		CREMI $\rightarrow$ ISBI
Method	AUROC (↑)	AUPR (↑)	AUROC (↑)	AUPR (↑)	AUROC (↑)	AUPR (↑)
MC Dropout	53.07	48.51	82.21	53.07	87.34	94.64
DDU	54.63	52.54	85.99	61.77	88.79	94.82
TTA	55.05	56.62	89.94	67.31	93.10	94.88
Masksembles	54.00	51.91	86.35	64.51	87.72	94.14
Deep Ensemble	57.32	57.85	89.56	66.33	85.73	94.69

Table 3: Cross-dataset segmentation OOD detection.

Dataset	Method	Acc (↑)	ECE (↓)	Brier (↓)	AUC (↑)	AUPR (↑)	FPR95 (↓)
CIFAR-10	MC Dropout	93.54	0.011	0.094	84.22	87.36	27.10
	DDU	94.12	0.018	0.090	85.20	89.26	24.11
	TTA	94.23	0.011	0.083	93.64	97.29	17.97
	Masksembles	94.61	0.011	0.080	91.69	92.88	18.47
	Deep Ensemble	94.94	0.009	0.074	91.92	93.15	14.70
	Laplace Redux	93.83	0.090	0.103	92.60	94.98	18.83

Table 4: Classification accuracy and OOD detection on natural and medical benchmarks.

Granularity	Acc. (↑)	NLL (↓)	ECE (↓)	Brier (↓)
Layer-level	93.04	0.204	0.005	0.101
Channel-level	93.66	0.200	0.009	0.094
Node-level	94.68	0.162	0.009	0.088

Table 5: Ablation on stochastic activation granularity.

Activation Pair	Acc. (↑)	NLL (↓)	ECE (↓)	Brier (↓)
ReLU + Tanh [ LeCun et al.(2002)LeCun, Bottou, Orr, and Müller ]	94.43	0.176	0.008	0.085
ReLU + SiLU [ Elfwing et al.(2018)Elfwing, Uchibe, and Doya ]	94.68	0.162	0.009	0.088
ReLU + GELU [ Hendrycks(2016) ]	94.12	0.184	0.012	0.095
ReLU + Mish [ Misra(2020) ]	93.98	0.192	0.014	0.101
ReLU + ELU [ Clevert et al.(2016)Clevert, Unterthiner, and Hochreiter ]	93.67	0.206	0.018	0.109

Table 6: Effect of the activation pair used for stochastic switching.

Figure 5: IDR comparison across datasets. Lower is better.

Passes $T$	Accuracy (↑)	NLL (↓)	ECE (↓)	Brier (↓)	Time (s)
1	93.81	0.223	0.026	0.096	3.11
3	94.26	0.186	0.013	0.085	7.06
5	94.68	0.162	0.009	0.088	11.06
10	94.37	0.176	0.007	0.082	20.99

Table 7: Effect of the number of inference passes on acc, uncertainty quality, and runtime.

Backbone	Dataset	Optimizer	Batch	LR / schedule	Epochs
U-Net	Massachusetts	Adam [ Diederik(2014) ]	32	$1{\times}10^{-3}$ , cosine	200
U-Net	DRIVE	Adam	8	$1{\times}10^{-3}$ , cosine	200
U-Net	LiTS	Adam	64	$1{\times}10^{-4}$ , cosine	200
Swin-V2	CREMI	AdamW [ Loshchilov and Hutter(2017) ]	32	$1{\times}10^{-4}$	200
ResNet-18	CIFAR-10	SGD ( $m{=}0.9$ )	256	$0.1$ , cosine	200
ResNet-50	MiniImageNet	SGD ( $m{=}0.9$ )	128	$0.1$ , cosine	200

Table 8: Per-dataset training hyperparameters used for all methods. Weight decay is

5{\times}10^{-4}

for SGD and

1{\times}10^{-4}

for AdamW.

Method	Training Time (s) $\downarrow$	Inference Time (s) $\downarrow$
Deterministic	40.05	2.83
MC Dropout	40.97	13.66
DDU	49.93	2.86
TTA	40.05	18.43
Masksembles	43.30	3.16
Deep Ensemble	194.52	13.27

Table 9: Training and inference time on CIFAR-10.

Method	Massachusetts		DRIVE		LiTS		CREMI
Method	Train	Infer.	Train	Infer.	Train	Infer.	Train	Infer.
Deterministic	54.41	1.52	37.30	24.30	28.27	10.98	36.98	9.22
MC Dropout	54.54	4.46	38.28	140.43	32.92	55.78	39.27	52.90
DDU	74.41	3.01	51.70	26.40	48.38	11.09	52.45	9.36
Masksembles	57.91	1.90	39.40	29.60	33.73	12.51	37.64	12.51
Deep Ensemble	261.79	4.25	214.74	135.11	103.84	54.47	189.48	49.42

Table 10: Training and inference time on the segmentation benchmarks. All times are reported in seconds.

Figure 6: Accuracy under CIFAR-10-C Gaussian noise corruption at severities $1$ , $3$ , and $5$ . MC Dropout gives slightly higher accuracy under heavier corruption, likely because feature masking acts as a stronger noise-robust regularizer. Our method remains close in accuracy while preserving information flow through activation switching instead of feature removal.

Severity	Method	Acc. (↑)	NLL (↓)	ECE (↓)	Brier (↓)
1	MC Dropout	82.95	0.517	0.015	0.243
	Deep Ensemble	84.96	0.486	0.042	0.220
	TTA	29.20	3.030	0.401	1.019
	Laplace Redux	28.29	2.315	0.213	0.872
	Ours	83.96	0.521	0.040	0.217
3	MC Dropout	43.89	1.889	0.184	0.756

Table 11: CIFAR-10-C Gaussian noise corruption results at severities

1

3

, and

5

Panel of Figure 7 — Figure 7: Uncertainty comparison of ID and OOD samples. ID scores are concentrated at lower uncertainty, while OOD scores are dispersed and shifted toward higher uncertainty.

Figure 8: Qualitative OOD segmentation examples on ISBI 2012 using a model trained on CREMI. Each row shows the original image, ground-truth mask, prediction, and variance-based uncertainty map.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 652 papers

Activation-Space Uncertainty Quantification for Pretrained Networks
Feb 16, 2026Richard Bergna, Stefan Depeweg, Sergio Calvo-Ordoñez +3Uncertainty Quantification Uncertainty Calibration
Department of Engineering, University of Cambridge, Cambridge, UK · Siemens AG, Munich, Germany · Mathematical Institute and Oxford-Man Institute, University of Oxford, Oxford, UK
U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration
May 14, 2026Michael Smith, Frank P. FerrieImage Segmentation Panoptic Segmentation
Centre for Intelligent Machines, McGill University
Revisiting Neural Activation Coverage for Uncertainty Estimation
Apr 24, 2026Benedikt Franke, Nils Förster, Frank Köster +3Uncertainty Quantification
DLR Institute for AI Safety and Security, Ulm - Germany · Ruhr University Bochum, Bochum, Germany

Feb 16, 2026J/K move · Enter open · S save

Activation-Space Uncertainty Quantification for Pretrained Networks

Richard Bergna, Stefan Depeweg, Sergio Calvo-Ordoñez, Jonathan Plenk+2

Department of Engineering, University of Cambridge, Cambridge, UK · Siemens AG, Munich, Germany · Mathematical Institute and Oxford-Man Institute, University of Oxford, Oxford, UK

Uncertainty Quantification Uncertainty Calibration

Open paper AskPDF arXiv

Reliable uncertainty estimates are crucial for deploying pretrained models; yet, many strong methods for quantifying uncertainty require retraining, Monte Carlo sampling, or expensive second-order computations and may alter a frozen backbone's predictions. To address this, we introduce Gaussian Process Activations (GAPA), a post-hoc method that shifts Bayesian modeling from weights to activations. GAPA replaces standard nonlinearities with Gaussian-process activations whose posterior mean exactly matches the original activation, preserving the backbone's point predictions by construction while providing closed-form epistemic variances in activation space. To scale to modern architectures, we use a sparse variational inducing-point approximation over cached training activations, combined with local k-nearest-neighbor subset conditioning, enabling deterministic single-pass uncertainty propagation without sampling, backpropagation, or second-order information. Across regression, classification, image segmentation, and language modeling, GAPA matches or outperforms strong post-hoc baselines in calibration and out-of-distribution detection while remaining efficient at test time.

Abstract

Uncertainty Quantification

Explore similar work

Activation-Space Uncertainty Quantification for Pretrained Networks

U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration

Revisiting Neural Activation Coverage for Uncertainty Estimation