$TCP_α$: Margin-Controlled Confidence estimation for reliable Music Information Retrieval

Parampreet Singh; Anushka Singh; Sumit Kumar; Vipul Arora

eess.ASAug 20, 2026

$TCP_α$ : Margin-Controlled Confidence estimation for reliable Music Information Retrieval

Authors: Parampreet Singh, Anushka Singh, Sumit Kumar, Vipul Arora

Organizations: Indian Institute of Technology, Kanpur, India

Abstract

Deep neural networks are often overconfident, assigning high confidence even to incorrect predictions. Consequently, users lack a reliable signal for deciding when a prediction can be trusted. Post-hoc confidence estimation addresses this by training a lightweight auxiliary head over a frozen classifier. Existing targets, however, suffer from inherent ambiguity: they assign overlapping confidence values to correct and incorrect predictions, while errors near the decision boundary receive confidence scores indistinguishable from correct predictions. In this work, we propose $TCP_α$ , a novel confidence target that resolves these limitations by introducing a margin-controlled penalty for misclassified samples. We prove that $TCP_α$ guarantees complete separation between the target values of correct and incorrect predictions, with a separation margin that is independent of the number of classes and increases monotonically with the penalty parameter. Since accurate classifiers naturally produce very few errors, learning these targets results in a severely imbalanced regression problem. We therefore present a systematic study of training strategies for learning under this imbalance and identify an effective training configuration through extensive ablation studies. We evaluate the proposed approach on rāga identification, investigate its robustness under domain shift, and further validate it on frame-wise ornamentation detection without modifying the selected configuration. Across all settings, $TCP_α$ consistently outperforms existing confidence targets for failure prediction. Rejecting only the least-confident 8% of predictions improves the base model's macro-F1 from 0.89 to 0.98, while fine-tuning the confidence head with only 5% labeled samples from a new corpus effectively restores performance under domain shift.

Figures & tables

Fig. 1 : Overview of the post-hoc confidence estimation module. The backbone and classifier are frozen after base training; only the confidence head is trained, against the TCP α target derived from the frozen classifier’s softmax output and the ground-truth label.

Fig. 2 : Comparison of the target distributions induced by MCP, TCP n , and the proposed TCP α for PIM [ 7 ] dataset. The x-axis represents the targets and y-axis represents the density. TCP α produces complete separation between success and error targets.

Criterion	$\mathcal{T}^{+}$	$\mathcal{T}^{-}$	Overlap	Margin $\Delta$
MCP	$[\tfrac{1}{K},1]$	$[\tfrac{1}{K},1]$	always	$0$
TCP	$[\tfrac{1}{K},1]$	$[0,\tfrac{1}{2})$	$[\tfrac{1}{K},\tfrac{1}{2})$	$0$
TCP n	$\{1\}$	$(0,1)$	at boundary	$0$
TCP α (ours)	$\{1\}$	$\bigl(0,\tfrac{1}{2(1+\alpha)}\bigr)$	none	$1-\tfrac{1}{2(1+\alpha)}$

TABLE I : Theoretical properties of confidence targets.

\mathcal{T}^{+},\mathcal{T}^{-}

are the achievable target ranges for successes and errors;

K

is the number of classes;

\alpha\geq 0

is the penalty of TCP α .

Dataset	Unit	$K$	$\|\mathcal{D}^{+}\|\!:\!\|\mathcal{D}^{-}\|$	Base F1
PIM	30 s chunk	12	$\approx$ 8.1:1	0.89
ROD	frame	7	$\approx$ 2.2:1	0.69
Saraga	30 s chunk	11	$\approx$ 3.2:1	0.76

TABLE II : Dataset statistics and the induced imbalance seen by the confidence head.

|\mathcal{D}^{-}|

is the number of base-model errors available for training the confidence head. K is the number of classes. Base F1 shows performance of the base classifier on which the confidence head is being trained.

Method	FPR@95 $\downarrow$	AUPR-E $\uparrow$	AUPR-S $\uparrow$	AUROC $\uparrow$
MCP	42.25	50.59	96.94	83.60
Entropy	41.00	53.70	97.10	84.49
T2M	43.00	44.25	96.70	82.53
Energy	37.50	56.87	97.47	86.25
MC-D	40.75	52.37	97.13	84.70
TCP (R)	46.02	25.35	98.85	85.11

TABLE III : Performance on rāga identification (PIM). All values are percentages, best in bold. (R) denotes the target trained with vanilla regression, (P) denotes training with our proposed training strategy (Sec: IV-C ). T2M: Top-2-Margin score, MC-D: MC Dropout

Fig. 3 : Distribution of predicted confidence scores for $\mathrm{TCP}_{\alpha}$ on the PIM test set. The proposed target produces clear separation between correct and incorrect predictions.

Fig. 4 : Macro-F1 of the retained predictions as increasingly low-confidence samples are rejected using confidence scores from baselines and proposed method.

Strategy	FPR@95 $\downarrow$	AUPR-E $\uparrow$	AUPR-S $\uparrow$	AUROC $\uparrow$
reg	53.68	74.30	98.89	91.26
FDS	54.71	65.92	98.89	90.07
EU	66.14	29.91	98.34	81.67
LDS	21.98	85.70	98.94	94.42
$\mathrm{TCP}_{\alpha}$ (P)	1.60	95.96	99.95	99.46
Binary	76.08	25.92	97.13	76.75

TABLE IV : Ablation studies for training

\mathrm{TCP}_{\alpha}

target on PIM. All rows use the same target and the same confidence head, and differ only in how the imbalance is handled. EU: Error Upsampling,

\mathrm{TCP}_{\alpha}

-P: proposed (Sec IV-C )

r	FPR@95 $\downarrow$	AUPR-E $\uparrow$	AUROC $\uparrow$
5.4	39.36	91.09	95.08
3.3	10.62	97.41	97.50
2.2	1.60	95.96	99.46
1.6	7.53	89.21	98.14
1.1	7.63	82.19	98.25
0.6	10.17	73.69	97.31

TABLE V : Effect of the success-to-error ratio

r

in each training batch on PIM. Batch size is fixed at 64.

$\alpha$	FPR@95 $\downarrow$	AUPR-E $\uparrow$	AUROC $\uparrow$
$0$	11.02	88.10	97.42
$1/2K$	3.48	91.15	97.29
$1/K$	1.60	95.96	99.46
$2/K$	6.31	90.20	96.90
$0.5$	8.10	87.49	96.77
$1$	13.09	89.33	98.02

TABLE VI : Effect of the penalty

\alpha

TCP_{\alpha}

performed on PIM dataset.

Model	FPR@95 $\downarrow$	AUPR-E $\uparrow$	AUPR-S $\uparrow$	AUROC $\uparrow$
MCP	73.38	60.39	89.85	80.61
reg	52.83	80.14	92.05	88.56
$\mathrm{TCP}_{\alpha}$ $(P_{0})$	87.5	40.16	78.15	62.61
$\mathrm{TCP}_{\alpha}$ $(P_{5})$	5.0	97.90	99.50	98.21
$\mathrm{TCP}_{\alpha}$ $(P_{10})$	3.12	97.28	99.46	98.78
$\mathrm{TCP}_{\alpha}$ $(P_{20})$	2.50	97.92	99.37	98.60

TABLE VII : Performance on Saraga dataset. The base classifier is the PIM model in all rows and is never retrained. reg represents vanilla regression training.

\mathrm{TCP}_{\alpha}

(

P_{s}

): best model trained on PIM fine-tuned on Saraga using s% labeled training samples, test set being fixed for all.

Method	FPR@95 $\downarrow$	AUPR-E $\uparrow$	AUPR-S $\uparrow$	AUROC $\uparrow$
MCP	68.66	43.34	91.98	77.14
TCP	33.46	76.62	97.60	92.31
$\mathrm{TCP_{n}}$	47.88	71.21	96.77	89.22
$\mathrm{TCP}_{\alpha}$	19.13	86.01	98.70	95.80
Binary	35.30	81.58	97.74	92.90
Focal	35.10	81.41	97.91	92.96

TABLE VIII : Failure prediction on ornamentation detection. The configuration is the one selected on rāga identification; no additional search was performed.

Fig. 5 : Rejection Curves for Ornamentation detection task

Fig. 6 : Density plot for the best performing model for Ornamentation detection

Confidence Estimation

See all 42 papers

Aug 10, 2026

ReliableNet: A Chance-Constrained Approach to Trustworthy Classification in Deep Learning

A prediction that is both confident and wrong is a critical reliability failure because it can bypass abstention and human review precisely when the model is mistaken. Empirical risk minimization (ERM) controls average loss but not this failure directly, while calibration, uncertainty estimation, conformal risk control, and selective prediction methods target related reliability properties rather than bounding the joint failure event during training. We propose ReliableNet, which constrains the Joint Confident-Wrong (JCW) probability, the probability that a prediction is simultaneously confident and incorrect, below a user-specified risk budget

α\in(0,1)

. We formulate this as a chance-constrained ERM problem, use a conservative smooth inner approximation whose population feasibility implies the original JCW constraint. Across four tabular and two image datasets, ReliableNet is the only method certified within the JCW budget for every dataset and seed in distribution, when compared against baselines spanning ERM, post-hoc calibration, conformal risk control, and selective prediction. Under demographic, ambiguity, spurious-correlation, novel-class, and covariate shifts, it achieves the lowest empirical JCW among the compared methods while remaining very competitive in accuracy, coverage, calibration, and selective prediction. Risk-coverage results further indicate that ReliableNet achieves better selective ranking than the benchmark methods on most datasets. Overall, ReliableNet provides a principled approach to trustworthy classification.

Ange-Clément Akazan, Ineza Remy Mugenga, Abebe Geletu +2

School of Agriculture and Science, University of KwaZulu-Natal, Pietermaritzburg, KwaZulu-Natal, South Africa · 2AIMS Research and Innovation Centre, African Institute for Mathematical Sciences, Kigali, Rwanda · 3African Institute for Mathematical Sciences, Kigali, Rwanda

Chance-Constrained Optimization Classification Empirical Risk Minimization

Ask PDF

Oct 18, 2026

TrustAgent-RFF: Agentic Diagnostic Planning for Reliable RF Fingerprinting under Domain Shift

Radio frequency fingerprinting (RFF) models can achieve accuracy under source domain conditions but become unreliable when the receiver, distance, collection day, or environment changes. Incorrect predictions in unseen domains may receive high softmax confidence, making confidence thresholding insufficient for trustworthy deployment. We propose TrustAgent-RFF, a prediction level reliability framework for RFF that operates with a frozen in-phase and quadrature (I/Q) classifier without target domain labels. For each device prediction, the framework acquires radio frequency evidence: counterfactual stability, prototype consistency, acquisition domain similarity, and signal quality. A source calibrated trust estimator combines these diagnostics with classifier confidence, prediction margin, and evidence availability indicators. Source derived lower tail penalties and cross module interactions reduce trust when diagnostic signals indicate weak support. A large language model (LLM) based planner adaptively selects the next diagnostic action based on evidence, trust state, action costs, and remaining budget. A deterministic source calibrated decision layer updates the prediction state as Accept, Reject, or Defer. Experiments across five RFF datasets and classifier backbones reveal an average source to target accuracy drop of 38.7% and show that many incorrect target domain predictions remain highly confident.

Chuan Liu, Tianya Zhao, Xuyu Wang

Knight Foundation School of Computing and Information Sciences, Florida International University, Miami, FL 33199, USA

RF Fingerprint Identification Distribution Shift Robustness Confidence Estimation Selective Prediction Adaptive Inference

Ask PDF

May 12, 2026

A Semi-Supervised Framework for Speech Confidence Detection using Whisper

Automatic detection of speaker confidence is critical for adaptive computing but remains constrained by limited labelled data and the subjectivity of paralinguistic annotations. This paper proposes a semi-supervised hybrid framework that fuses deep semantic embeddings from the Whisper encoder with an interpretable acoustic feature vector composed of eGeMAPS descriptors and auxiliary probability estimates of vocal stress and disfluency. To mitigate reliance on scarce ground truth data, we introduce an Uncertainty-Aware Pseudo-Labelling strategy where a model generates labels for unlabelled data, retaining only high-quality samples for training. Experimental results demonstrate that the proposed approach achieves a Macro-F1 score of 0.751, outperforming self-supervised baselines, including WavLM, HuBERT, and Wav2Vec 2.0. The hybrid architecture also surpasses the unimodal Whisper baseline, yielding a 3% improvement in the minority class, confirming that explicit prosodic and auxiliary features provide necessary corrective signals which are otherwise lost in deep semantic representations. Ablation studies further show that a curated set of high confidence pseudo-labels outperforms indiscriminate large scale augmentation, confirming that data quality outweighs quantity for perceived confidence detection.

Adam Wynn, Jingyun Wang

Department of Computer Science, Durham, United Kingdom, DH1 3DF

Audio Understanding Pseudo-Labeling Semi-Supervised Learning

Ask PDF

Abstract

Confidence Estimation

Explore similar work

ReliableNet: A Chance-Constrained Approach to Trustworthy Classification in Deep Learning

TrustAgent-RFF: Agentic Diagnostic Planning for Reliable RF Fingerprinting under Domain Shift

A Semi-Supervised Framework for Speech Confidence Detection using Whisper