Semi-Supervised Learning

Semi-supervised few-shot learning (SSFSL) resembles real-world applications such as auto-annotation, as it aims to learn a model from a few labeled and abundant unlabeled task-specific examples to annotate the unlabeled ones. Despite the availability of powerful open-source Vision-Language Models (VLMs) and open-world data, existing SSFSL literature largely neglects these resources. In contrast, the related area few-shot learning (FSL) has already exploited them to boost performance. Arguably, to solve real-world auto-annotation, SSFSL should leverage such open resources. To bridge this gap, we explore established SSL methods to finetune a VLM. Unexpectedly, they significantly underperform FSL baselines that do not use unlabeled data. Our in-depth analysis reveals the root cause of failure: VLMs produce flat distributions of softmax probabilities, resulting in zero utilization of unlabeled data and weak supervision signals. To address this challenge, we propose an embarrassingly simple solution that uses temperatures to sharpen the softmax output, which not only increases the confidence scores of pseudo-labels to improve the utilization of unlabeled data, but also strengthens training supervision for effective finetuning. Furthermore, we exploit task-relevant open data, e.g., those retrieved from VLMs' publicly available pretraining set. To mitigate the imbalance and domain gaps in retrieved data, we employ a stage-wise training strategy. Building on the successful finetuning of VLMs and the exploitation of open data, we present a simple yet effective SSFSL method, Stage-Wise Finetuning with Temperatures (SWIFT). Across five benchmarks, SWIFT outperforms recent FSL and SSL methods by $\sim$5 accuracy points. SWIFT even rivals supervised learning, which finetunes a VLM assuming unlabeled data having ground-truth labels!

Weakly supervised learning has emerged as a practical alternative to fully supervised learning when complete and accurate labels are costly or infeasible to acquire. However, many existing methods are tailored to specific supervision patterns -- such as positive-unlabeled (PU), unlabeled-unlabeled (UU), complementary-label (CLL), partial-label (PLL), or similarity-unlabeled annotations -- and rely on post-hoc corrections to mitigate instability induced by indirect supervision. We propose a principled, unified framework that bypasses such post-hoc adjustments by directly formulating a stable surrogate risk grounded in the structure of weakly supervised data. The formulation naturally subsumes diverse settings -- including PU, UU, CLL, PLL, multi-class unlabeled, and tuple-based learning -- under a single optimization objective. We further establish a non-asymptotic generalization bound via Rademacher complexity that clarifies how supervision structure, model capacity, and sample size jointly govern performance. Beyond this, we analyze the effect of class-prior misspecification on the bound, deriving explicit terms that quantify its impact, and we study identifiability, giving sufficient conditions -- most notably via supervision stratification across groups -- under which the target risk is recoverable. Extensive experiments show consistent gains across class priors, dataset scales, and class counts -- without heuristic stabilization -- while exhibiting robustness to overfitting.

Modern artificial intelligence has revolutionized how we extract representations from scientific data, yet the statistical properties of these representations remain poorly controlled, causing misspecified anomaly detection methods to falter. The hardest anomalies to detect are the rare, weakly separable ones hiding within the nominal distribution-a regime that grows in importance as models mature and easily separable signals are exhausted. We identify structural desiderata for detection in this regime under minimal prior information: sparsity, to enforce parsimony; locality, to preserve geometric sensitivity; and competition, to promote efficient allocation of model capacity. These principles define a class of self-organizing local kernels that adaptively partition the representation space around regions of statistical imbalance. As an instantiation, we introduce SparKer, a sparse ensemble of Gaussian kernels trained in a semi-supervised Neyman-Pearson framework to locally model the likelihood ratio between a sample that may contain anomalies and an anomaly-free reference. We provide theoretical insights into the mechanisms driving detection and self-organization, and demonstrate the approach on realistic high-dimensional problems in scientific discovery, open-world novelty detection, intrusion detection, and generative-model validation. Ensembles of only a handful of kernels identify statistically significant anomalies in representation spaces of thousands of dimensions while remaining sensitive across regimes, underscoring the interpretability, efficiency, and scalability of the approach.

Real-world time-series regression often involves non-stationarity, heteroscedasticity, and regime changes, under which recent prediction errors may contain structured information about local temporal mismatch between model predictions and observations. Learning how to represent and reuse these errors can therefore provide useful information for subsequent prediction. We introduce the Variability-Aware Recursive Neural Network (VARNN), a residual-aware architecture for supervised time-series regression that learns an explicit residual-memory state from recent prediction errors and uses it to condition subsequent predictions. Specifically, VARNN maps scalar prediction innovations into a learned nonlinear, vector-valued residual representation over a short context. Across nine datasets spanning energy, healthcare, and environmental domains, VARNN achieves lower test MSE than the compared static, lag-based, and sequence-model baselines. Targeted ablations further show that learned projected residual memory improves predictive accuracy over direct scalar residual feedback, supporting the benefit of a learned nonlinear representation of prediction deviations.

Conventional dimensionality reduction methods mainly optimize variance or correlation, leaving statistical dependence, data diversity, contrast, and interpretability under addressed. We propose three new independence criteria for designing supervised and unsupervised dimensionality reduction (DR) methods, aiming to improve feature extraction and representation quality. Our framework combines linear and nonlinear formulations and is evaluated using contrast, classification accuracy, and interpretability measures. The interpretability of eigenfaces helps to effectively summarize dominant class-specific structures and trends within representative images. Evaluated on MNIST and a Gender face dataset for classification and reconstruction, our methods achieve significant improvements in contrast (up to $+$20.1%), accuracy (up to $+$17.4%), and interpretability (up to $+$120.0%) over Principal Component Analysis (PCA), t-distributed Stochastic Neighbor Embedding (t-SNE), Linear Discriminant Analysis (LDA), and Variational Autoencoder (VAE) baselines, while also improving VAE reconstruction performance by 9.5%. These results suggest a promising direction for interpretable representation learning based on statistical dependence and independence criteria.

Uncertainty quantification is crucial in machine learning, yet most (axiomatic) studies of uncertainty measures focus on classification, leaving a gap in regression settings with limited formal justification and evaluations. In this work, we provide a formal way of representing uncertainty in continuous space, using a general parametric formulation, allowing for tractable analysis and evaluation of uncertainty measures. Within this framework, we propose a set of axioms that enable rigorous assessment of total, aleatoric, and epistemic uncertainty measures. Together, this allows for a theoretical examination of uncertainty measures and their corresponding properties. As a specific example, we compare the widely used entropy- and variance-based measures with respect to established predictive models and analyze their limitations and challenges in uncertainty quantification. Our work provides a principled way to understand and develop uncertainty measures in supervised regression, offering theoretical insights and practical guidelines for reliable uncertainty assessment.

Supervised deep learning achieves strong performance in biomedical image segmentation but relies on costly pixel-wise annotations, motivating semi-supervised approaches that exploit unlabeled data. We introduce a diffusion-based teacher--student framework in which segmentation predictions are used to condition image denoising, encouraging the production of more informative pseudo-labels. The teacher is first pretrained through an unsupervised reconstruction task using diffusion-style corruption, timestep conditioning, and denoising. Starting from a corrupted empty mask, the model predicts an intermediate segmentation that conditions image denoising, encouraging the predicted mask to capture structural information useful for recovering the original image. The resulting teacher is then co-trained with a student using supervised segmentation on labeled samples and cross pseudo-supervision on unlabeled data. We further introduce a multi-round extension during co-training, in which the teacher generates multiple stochastic image reconstructions and corresponding segmentation predictions, providing additional reconstruction and alignment signals to improve its pseudo-labels. We evaluate the proposed framework on three public 2D biomedical segmentation datasets and a 3D left atrial segmentation benchmark. Across several labeling regimes, our method achieves competitive or superior performance compared with state-of-the-art semi-supervised approaches, with the largest gains observed under severe label scarcity.

In this paper, the method of gaps, a technique for deriving closed-form expressions in terms of information measures for the generalization error of supervised learning algorithms, is introduced. This method relies on the notion of gaps, which characterize the variation of the expected empirical risk (when either the model or dataset is kept fixed) with respect to changes in the probability measure on the varying parameter. This distinction results in two classes of gaps: algorithm-driven gaps (fixed dataset) and data-driven gaps (fixed model). The method relies on two central observations: (i) the generalization error is the expectation of an algorithm-driven gap or a data-driven gap. In the first case, the expectation is with respect to a measure on the datasets; in the second case, it is with respect to a measure on the models. (ii) Both algorithm-driven gaps and data-driven gaps exhibit closed-form expressions in terms of relative entropies. In particular, algorithm-driven gaps involve a Gibbs probability measure on the set of models, which represents a supervised Gibbs algorithm. Alternatively, data-driven gaps involve a worst-case data-generating (WCDG) probability measure on the set of data points, which is also a Gibbs probability measure. Interestingly, such Gibbs measures, which are exogenous to the analysis of generalization, place the supervised Gibbs algorithm and the WCDG probability measure as natural references for the analysis of supervised learning algorithms. New exact expressions and all existing exact expressions for the generalization error of supervised learning algorithms can be obtained with the proposed method. Such new expressions are intended as structural and conceptual characterizations, not computational shortcuts. Finally, these expressions unveil strong connections among generalization, hypothesis testing, information measures, and Pythagorean identities.