Learnability

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 126

Jun 16, 2026cs.LG

No-Free-Fairness: Fundamental Limits and Trade-offs in Learning Systems

In this paper, we establish a set of theoretical impossibility results, termed the No-Free-Fairness theorems, that identify three fundamental sources of disparity in learning systems. First, we show that when a task exhibits irreducible cost on a subgroup, any decision rule must trade off overall performance with disparity, yielding an inherent fairness--cost frontier. Second, we prove that even in ideal, noise-free settings where a perfectly fair and accurate solution exists, finite-sample learning alone induces nontrivial subgroup disparity, ruling out distribution-free fairness guarantees. More seriously, enforcing strict relative fairness creates a statistical bottleneck: achieving low cost may require exponentially many samples. Third, we show that limitations of the model class can independently induce disparity: if the model cannot represent accurate solutions for a subgroup, fairness remains unattainable regardless of data or training procedure. Overall, these results demonstrate that unfairness is not solely a consequence of biased data or suboptimal optimization, but arises from the intrinsic structure of decision problems, the constraints of finite data, and the expressivity of models. Our framework applies broadly beyond standard supervised learning, and suggests that achieving fairness requires explicit trade-offs and should be treated as a core design consideration.

Khoat Than

Hanoi University of Science and Technology

Algorithmic Fairness Learnability Subgroups Learning-Augmented Algorithms Biases Fundamental Limits+1

Jun 14, 2026cs.LG

ALCL: An Adaptive Log-Correntropy Loss for Robust Learning under Non-Gaussian Noise

Robust deep learning under heavy-tailed and impulsive noise remains challenging because conventional losses such as mean squared error (MSE) exhibit unbounded sensitivity to outliers. Although correntropy-based objectives improve robustness, existing formulations rely on fixed kernel parameters that must be empirically tuned and remain static during training. To address these limitations, we propose an Adaptive Log-Correntropy Loss (ALCL), a heavy-tailed loss formulation that adaptively learns its robustness geometry during optimization. ALCL introduces a logarithmic residual model whose shape and scale parameters are learned jointly with network weights through differentiable reparameterization. This yields a principled maximum likelihood formulation whose influence function is formally bounded and redescending, allowing the loss geometry to adapt dynamically to evolving residual statistics while suppressing extreme outliers. Comparative experiments on four widely used benchmark datasets spanning grayscale and red-green-blue (RGB) image data under mixed heavy-tailed and impulsive noise demonstrate that ALCL consistently outperforms MSE and optimally tuned generalized correntropy losses in both reconstruction fidelity and downstream classification accuracy. While performance differences remain small under low-noise conditions, under high-noise regimes ALCL improves median accuracy by up to 4.75% on grayscale benchmarks and 4.51% on RGB datasets, with reduced variance across runs. These results demonstrate that adaptive robustness through joint learning of loss parameters provides a computationally efficient alternative to static correntropy-based losses for deep learning in non-Gaussian environments.

Mainak Kundu, Ria Kanjilal, Ismail Uysal

Department of Electrical Engineering, University of South Florida, Tampa, FL 33620 USA · Department of Computer Engineering, California Polytechnic State University, San Luis Obispo, CA 93407 USA

Noise-Aware Learnability

Jun 12, 2026cs.LG

Learning High Coverage Discriminative Parsimonious Rulesets

Learning systems based on IF-THEN rule representations readily offer interpretability, making them a crucial focus in contemporary AI research. A key objective for such rule sets is to achieve both high discriminative power and interpretability. While existing state-of-the-art algorithms implicitly prioritize predictive accuracy, they often fall short on one or more quality metrics that ensure interpretability, such as coverage and parsimony of rule sets. Motivated by this, this paper propose the development of CDPR, which aims to create highly accurate and interpretable rule sets for classification problems. To the best of our knowledge, this represents the first attempt to establish such an approach. In this study, we introduce two algorithms rooted in submodular maximization, which not only provide provable guarantees on coverage but also yield rule sets that are both discriminative and parsimonious. We empirically demonstrate that rule sets learned through our approaches achieve higher accuracy and interpretability and has more than a 2.5-fold improvement in average coverage rates when compared to the next best algorithm.

Mariamma Antony, Raman Sankaran, Chiranjib Bhattacharyya +1

Indian Institute of Science, CV Raman Rd, Bengaluru, Karnataka 560012 · Compass, Bengaluru, Karnataka 560012

Interpretability Discriminative Learnability Artificial Intelligence Research

Jun 11, 2026cs.LG

Learning with Simulators: No Regret in a Computationally Bounded World

Understanding the minimal assumptions necessary for generalization is the fundamental question in learning theory. Unfortunately, most results rely heavily on independence (or some proxy thereof) of the data-generating process, while results for strongly dependent data are far more limited. Towards addressing this gap, we introduce the framework of simulatable processes, where the learner has access to a simulator that approximates the distribution generating the data (which may be an arbitrarily complex and dependent process). Surprisingly, given access to such a simulator, we show that we can recover the same learning guarantees as in the classical setting with independent data, namely, error bounds that depend on the VC dimension. Further, we use this framework to study the power of conditional sampling and show strict statistical and computational advantages in this setting. As a highlight of our framework, we exhibit a single algorithm that simultaneously learns any given VC class under all processes samplable in bounded polynomial time, with regret controlled by the time-bounded Kolmogorov complexity of the process. This provides a significant conceptual broadening of the classical PAC model.

Sasha Voitovych, Abhishek Shetty, Noah Golowich +1

MIT · Microsoft Research

Learnability Regret

Jun 11, 2026cs.LG

Information Lattice Learning as Probabilistic Graphical Model Structure Learning

Information lattice learning (ILL) learns interpretable rules of a signal by alternately projecting the signal onto a partition lattice that encodes a hierarchy of abstractions and lifting selected rules back to the signal domain. When the signal is a probability mass function, we show the probabilistic rules learned by ILL admit a natural probabilistic graphical model (PGM) interpretation and develop this interpretation in detail. A partition in ILL induces a deterministic quotient variable, and a rule is the marginal law of that quotient variable. A rule set is therefore a collection of marginal constraints over interpretable abstractions. General lifting is the feasible family of all joint distributions satisfying those constraints, while special lifting chooses a maximum-ignorance reconstruction, implemented in ILL by an L2 uniformity principle closely related to maximum entropy. Under a Shannon-entropy lifting, the same constraints yield a log-linear factor graph whose factors are indexed by learned abstractions. The information lattice itself, however, is not a Bayesian network: its edges encode refinement and coarsening of abstractions, not conditional dependence. Thus ILL is best viewed as structure learning for interpretable constraint-based factor graphs over quotient variables. This view clarifies how ILL relates to graphical models and maximum entropy models, while suggesting new directions for inference, identifiability, and hybrid symbolic-probabilistic learning.

Haizi Yu, Lav R. Varshney

Kocree, Inc. · AI Innovation Institute, Stony Brook University

Probabilistic Inference Factor Graphs Latent Variable Learnability Graph Representation Learning Neuro-Symbolic Framework

Jun 9, 2026cs.LG

Learning from almost nothing: How neural networks survive heavy input corruption

Learning from imperfect data is a central theme in machine learning, connecting practical questions of robustness to fundamental questions of learnability. Here we examine attribute noise: learning from corrupted inputs while keeping the labels intact, a setting that has received considerably less analytical attention than its label-noise counterpart. We consider two types of corruption models: additive noise and replacement noise. Through experiments with multi-layer perceptrons (MLPs) on corrupted classification datasets, we find that neural networks remain robust, maintaining well-above-chance accuracy even when inputs are >90% corrupted -- far beyond human recognition. To understand this robustness, we analyze infinite-width networks in the heavy-corruption regime using a mean-field-inspired approach and derive a leading-order decision rule for the classification outcome: the network implements a prototype rule, the nearest-class-mean, assigning each test point to the class whose training-set average it most closely resembles. This leading-order decision rule is universal across a broad range of MLP architectures, holding for any depth, as well as a wide class of activation functions and noise distributions. The same centroid mechanism closely matches finite-width network behavior in our experiments and provides an interpretable and analytically tractable account of why learning can succeed even when individual training examples carry almost no signal.

Justin Tahmassebpur, Asadullah Bhuiyan, Hyejin Kim +1

Department of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · Department of Physics, Cornell University, Ithaca, NY 14853, USA

Learnability Neural Network

Jun 9, 2026cs.LG

Robust Regression of General ReLUs with Queries

We study the task of agnostically learning general (as opposed to homogeneous) ReLUs under the Gaussian distribution with respect to the squared loss. In the passive learning setting, recent work gave a computationally efficient algorithm that uses

poly(d,1/ε)

labeled examples and outputs a hypothesis with error

O(opt)+ε

, where

opt

is the squared loss of the best fit ReLU. Here we focus on the interactive setting, where the learner has some form of query access to the labels of unlabeled examples. Our main result is the first computationally efficient learner that uses

d polylog(1/ε)+\tilde{O}(\min\{1/p, 1/ε\})

black-box label queries, where

p

is the bias of the target function, and achieves error

O(opt)+ε

. We complement our algorithmic result by showing that its query complexity bound is qualitatively near-optimal, even ignoring computational constraints. Finally, we establish that query access is essentially necessary to improve on the label complexity of passive learning. Specifically, for pool-based active learning, any active learner requires

\tildeΩ(d/ε)

labels, unless it draws a super-polynomial number of unlabeled examples.

Ilias Diakonikolas, Daniel M. Kane, Mingchen Ma

University of Wisconsin-Madison · University of California, San Diego

Learnability Rectified Linear Unit Unlabeled Data Structural Complexity Queries Gaussian Primitives

Jun 8, 2026cs.CL

Causally Evaluating the Learnability of Formal Language Tasks

Language models, as multi-task learners, acquire a wide range of abilities during training. A fundamental question is how much task-specific data is needed to learn a given task. Answering this for natural language is difficult: tasks are hard to delineate and can confound one another. To rigorously investigate the relationship between data frequency and learnability, we turn to a controlled setting using formal languages induced from probabilistic finite automata. These serve as a methodological testbed to demonstrate that standard correlational evaluation practices are inherently flawed. To enable causal analysis, we introduce the binning semiring, an algebraic object that lets us control how often a targeted property occurs in a sampled corpus. We formulate the experimental pipeline as a causal graphical model and derive decomposed Kullback-Leibler divergence metrics to measure the learnability of specific sub-tasks. Our experiments show that evaluating learnability without causal intervention leads to incorrect conclusions due to confounders in correlational analysis, and serve as a warning about correlational pitfalls in natural-language settings.

Vésteinn Snæbjarnarson, Anej Svete, Josef Valvoda +3

1ETH Zürich · University of Copenhagen · *Now at Google

Legal Natural Language Processing Benchmarks Learnability Evaluation Protocols Multi--Task Learning Automata Causal Graph

Jun 8, 2026cs.LG

Tight Sample Complexity of Transformers

We tightly characterize the VC dimension of depth-

L

Transformers with a total of

W

parameters, mapping an input sequence of length

T

to a single output, establishing an upper bound of

O(L W \log (T W))

and a nearly matching lower bound of

Ω(L W \log (T W / L))

. We further tightly characterize the sample complexity of chain-of-thought learning using such a Transformer, showing teacher forcing (i.e. selecting a predictor consistent with the entire chain-of-thought on training data) learns with sample complexity

O\left(L W \log \left(\left(T+T^{\prime}\right) W\right)\right)

and that any learning rule that uses chain-of-thought data requires at least

Ω\left(L W \log \left(\left(T+T^{\prime}\right) W / L\right)\right)

examples, where

T

is the input length and

T^{\prime}

is the number of autoregressive steps.

Chenxiao Yang, Nathan Srebro, Zhiyuan Li

Toyota Technological Institute at Chicago (TTIC)

Transformer Architectures Sample Complexity Optimal Sample Complexity Learnability Chain-of-Thought Reasoning Autoregressive Model+1

Jun 7, 2026cs.LG

Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions

Transformers consistently fail to learn certain simple functions that are provably expressible with specific parameter settings. This gap between learnability and expressivity is particularly prominent for sensitive functions -- functions whose output is likely to change if a single bit of the input is flipped -- for example, PARITY. While prior work has established that transformers exhibit a bias toward functions with low average sensitivity, the precise mechanism underlying this bias remains poorly understood. To shed light on this phenomenon, we study the geometry of transformers' parameter space. We show that sensitive functions -- even when representable -- occupy a vanishingly small region that random initialization is very likely to miss. Specifically, we shift the focus from average sensitivity to the full sensitivity profile -- the distribution of sensitivity values across all inputs -- and prove that randomly initialized transformers almost surely compute functions which have low-sensitivity strings. Consequently, any function that lacks such strings is provably unlearnable.

Blanka Köver, Alexandra Butoi, Anej Svete +2

Department of Mathematics, ETH Zürich, Zürich, Switzerland · Department of Computer Science, ETH Zürich, Zürich, Switzerland · Department of Language Science and Technology, Saarland University, Saarbrücken, Germany.

Transformer Architectures Activation Functions Learnability

May 31, 2026cs.LG

Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teacher

Weak-to-strong generalization studies how to improve a strong student using supervision from a weaker teacher when reliable labels are scarce. We view this primarily as a data selection problem, where the key challenge is to identify which weak labels are reliable enough to serve as a training signal. To address this, we introduce trust functions that assign each weak label a scalar trust score and use these scores to filter weak supervision. Across several domains, including world knowledge, quantitative reasoning, and strategy games, trust filtering yields students that match and sometimes surpass ground-truth supervision, achieving near-lossless weak-to-strong generalization. Moreover, trust functions enable an iterative weak-to-strong chain that compounds gains by training a student and reusing it as the next teacher, amplifying the gains. There are several mechanisms to which advantage of trust functions can be attributed.

Arda Uzunoglu, Alvin Zhang, Daniel Khashabi

Department of Computer Science, Johns Hopkins University, Maryland, United States.

Learnability Teacher AI Trustworthiness

May 29, 2026stat.ML

Hedging on the Frontier: Learning New Tasks with Few Samples

When a learner faces a new task with few samples, it must leverage any available side information. In practice, this often comes in the form of model evaluations on related tasks in public benchmarks. A key question then is how to model task relatedness such that it is both realistic and the benchmark evaluations lead to provable gains. Empirically, we observe that weak monotonicity is often approximately satisfied: if a model dominates another on many benchmarks, it also tends to outperform on the new task. We explore the statistical complexity of learning under (approximate) weak monotonicity, leveraging it within two learning paradigms: transfer learning and model selection aggregation. We show that not only can we prune the model class based on monotonicity, but we can also further adapt to the geometry of the available trade-offs by hedging on the frontier.

Tobias Wegel, Federico Di Gennaro, Geelon So +1

Department of Computer Science, ETH Zurich · Department of Computer Science and Engineering, UC San Diego

Learnability Monotonicity Transfer Learning Frontiers Model Selection New Tasks+1

May 28, 2026cs.LG

LARK: Learnability-Grounded Trajectory Selection for Efficient Reasoning Distillation

We study trajectory selection for reasoning distillation, where teacher-generated reasoning trajectories are selectively used as supervision for a student model. Existing methods rely on heuristics such as trajectory quality or model confidence, but they often overlook whether a trajectory is learnable by the student. In this paper, we present LARK, a learnability-grounded method for reasoning trajectory selection. LARK selects trajectories that the student can learn efficiently while preserving the generalization of the full training distribution. At the core of LARK is a learnability factor

ρ

, which characterizes the rate at which the student's training loss decreases. To estimate this rate efficiently and maintain generalization, we introduce a learnability proxy and a

χ^2

-regularized selection policy that balances learnability and distributional coverage, both with strong theoretical guarantees on their estimation error. Empirically, LARK consistently outperforms data selection baselines across multiple base models and reasoning tasks. Diagnostic analyses show that the LARK score predicts downstream training utility and that LARK-selected trajectories induce faster supervised fine-tuning loss reduction. Our code is available at https://github.com/Tianrun-Yu/LARK.

Tianrun Yu, Kaixiang Zhao, Chih-Chun Chen +5

Brigham Young University · The Pennsylvania State University · University of North Carolina at Chapel Hill

Reasoning Trajectory Learnability Trajectory Generation

May 28, 2026cs.LG

Universal Multiclass Transductive Online Learning

We consider the problem of universal transductive online classification with a possibly unbounded label space. This setting considers online learning, with the sequence of instances (without labels) known to the learner in advance. We say a concept class

\mathcal{H}

is learnable if there is a learning algorithm

\mathcal{A}

, such that for every realizable sequence, the number of mistakes made by

\mathcal{A}

grows at most sublinearly with the number of predictions. We characterize the learnability of this setting and show that there are only two possible optimal rates for the learnable classes: either bounded or increasing logarithmically. We introduce a new combinatorial structure, called ``Level-Constrained-Littlestone-Littlestone (LCLL) tree'', which, along with the indifference property, characterizes the learnability. We also extend the learnability result to the agnostic case and the case where only the stochastic process that generates the instance sequence is known.

Steve Hanneke, Hongao Wang

Department of Computer Science, Purdue University, West Lafayette, IN 47907, USA

Transductive Learning Learnability Unlabeled Data Multiclass Classification

May 28, 2026cs.DS

On Language Generation in the Limit with Bounded Memory

We study language generation in the limit under bounded memory. In this task, a learner observes examples from an unknown target language one at a time and must eventually output only new valid examples. Prior work assumes access to the entire history, a strong assumption since realistic algorithms retain limited past information. Classical work in learning theory shows memory constraints dramatically alter learnability; we extend this to language generation. First, we study memoryless generators. Under a mild enumeration restriction, every countable collection of infinite languages remains generable without memory. Without this restriction, we exactly characterize when memoryless generation is possible. For finite collections, we characterize the optimal minimax density achievable by memoryless generators -- the best density guaranteed against any collection of a given size. This combinatorial bound relies on Sperner's theorem and symmetric chain decompositions. We further show that a sliding window of the last

W

examples does not improve this worst-case density, whereas allowing it to store

b

adaptively chosen past examples improves the achievable density for every

b \geq 1

. Finally, we revisit identification in the limit, where the learner must converge to a single correct hypothesis for the target language. We focus on its incremental variant, where the learner remembers only its previous guess. Here, although exact identification fails on a collection of just three languages, a mild relaxation requiring convergence to an ``approximate'' version of the target is achievable for every finite collection. These results show bounded memory affects these tasks differently: generation remains achievable for every countable collection, while density and identification are confined to finite collections, with guarantees weakening as the collection grows.

Jon Kleinberg, Anay Mehrotra, Amin Saberi +1

Cornell University · Stanford University · Google Research

Bounded-Memory Learnability Infinities Worst-Case

May 28, 2026cs.LG

The Interplay Between Interpolation and Aggregation in Regression: Optimal Sample Complexity

This work investigates theoretically the interplay between interpolation and aggregation in regression. We establish that the

γ

-graph dimension characterizes learnability for a broad class of natural aggregation procedures. Furthermore, we prove that an extremely simple aggregation procedure, combining three interpolating hypotheses via the median, is optimal among all these aggregation procedures, and is strictly more powerful than proper learning. Finally, we show that some hypothesis classes are learnable only by aggregating infinitely many hypotheses or by using non-interpolating aggregation rules (which may predict outside the range of their inputs), and any finite interpolating aggregation fails to achieve even trivial performance.

Mikael Møller Høgsgaard, Kasper Green Larsen, Liang-Yu Zou

Department of Computer Science, Aarhus University, Aarhus, Denmark · Department of Statistics, University of Oxford, Oxford, United Kingdom

Stochastic Interpolants Learnability Optimal Sample Complexity Hypothesis Class Interpolation

May 28, 2026cs.FL

The Inclusion Depth of Pattern Languages: An Open Problem in Algorithmic Learning Theory

Pattern languages are a classical model in formal language theory and algorithmic learning theory. This note formulates the problem of computing the inclusion depth of a pattern language: the length of the longest strict inclusion chain from the universal pattern language to the language generated by a given pattern. Inclusion depth captures the mind-change complexity of pattern identification from positive data. The central open question is whether the inclusion depth ID_Sigma(p) is computable for every pattern p over every finite alphabet Sigma with at least two symbols, and whether it is computable in polynomial time. A simple conjectured formula, ID_Sigma(p) = 2|p| - #var(p) - 1, would imply a linear-time algorithm. The problem connects pattern language inclusion, combinatorics on words, language identification in the limit, and mind-change-bounded learning.

Wei Luo

School of Information Technology, Deakin University

Tractability Context-Free Grammars Learnability Induction Open Problems

May 28, 2026stat.ML

Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Data

Recent work in random matrix theory (RMT) has developed the notion of deterministic equivalents: typically linear surrogate models that approximate the spectral behavior of large nonlinear random matrices, such as nonlinear feature maps in neural networks (NNs). Such equivalents make theoretical predictions tractable by reducing a complex model to a simpler one with properties that fall under the umbrella of classical RMT tools. However, this leaves open the question of whether this idealized linear equivalence remains meaningful for classification of high-dimensional nonlinearly separable data. Motivated by this, we consider the conjugate kernel (CK), which is the nonlinear feature map of a one-layer feedforward NN, under a canonical nonlinearly separable dataset for the XOR problem; and we use the study of informative outlier eigenvalues in the CK and whether their corresponding eigenvectors asymptotically align with XOR labels as a proxy for nonlinear learnability. We develop a robust quadratic equivalent of the CK matrix that enables a precise analysis of emergent informative spikes, as one modifies various knobs common in ML practice: sample complexity, signal-to-noise ratio (SNR), nonlinear activation choice, and pretrained features. We identify regimes in which these knobs move the CK beyond the linear equivalent and produce BBP-type transitions to label-aligned outlier eigenspaces. Our analysis helps bring deterministic-equivalence tools from RMT to bear on problems of practical relevance in ML.

Collin Cranston, Zhichao Wang, Todd Kemp +1

Department of Mathematics University of California, San Diego, USA · ICSI and Department of Statistics University of California, Berkeley, USA · ICSI, LBNL and Department of Statistics University of California, Berkeley, USA

Kernel Method Spectral Decomposition Separability Feature Learning Exponential Family Learnability+3

May 27, 2026cs.LG

Unification and Optimization of Robust Supervised Learning

The literature has proposed various robust alternatives to empirical risk minimisation to address failure modes such as distribution shift, label noise and finite-sample degeneracies. Examples include distributionally robust optimization, label smoothing, vicinal risk minimization, and Mixup. However, such approaches are typically developed in isolation, forcing practitioners to commit a priori to a single failure mode even when the dominant mode for the task is unclear. To address this, we organize a broad class of existing methods along three common design axes and derive a tractable training procedure that decomposes robust learning into sequential stages (reference distribution enrichment, input-space perturbation, label-space perturbation, and sample-level aggregation), each with a choice of stance (pessimistic, neutral, or optimistic). This results in a unified design space in which joint hyperparameter optimization can compose and configure robustness strategies suited to the task at hand. Across tabular, image, and reward modeling benchmarks, joint hyperparameter optimization is competitive with the best single-method baseline in each setting, offering a reliable default for practitioners who do not know a priori which failure mode dominates their task.

Jonas Hanselle, Valentin Margraf, Clemens Damke +1

LMU Munich, MCML · LMU Munich, MCML, DFKI

Empirical Risk Minimization Learnability Semi-Supervised Learning Enrichment Analysis

May 27, 2026cs.LG

On the Learnability of Test-Time Adaptation: A Recovery Complexity Perspective

Test-time adaptation (TTA) aims to adapt models to maintain reliable performance on non-stationary test streams without requiring labeled data. Despite its empirical success, the learnability of TTA under non-stationary streams remains unexplored. A key challenge is the lack of a principled theoretical framework that simultaneously aligns with the TTA objective and captures both continuously evolving distribution shifts and intrinsic information constraints. To address this gap, we propose the first theoretical framework for studying the learnability of TTA and introduce

(ε,δ)

-Recovery Complexity and

(ε,ρ)

-TTA Learnability. Recovery complexity measures the post-shift time needed to maintain excess risk below a target level with high probability, and is further extended to TTA learnability, which measures the long-term reliability of TTA. Within this framework, we introduce a novel discrete surrogate for non-stationary test streams, enabling a unified and tractable analysis of both gradual and abrupt shifts. We derive order-wise matching lower and upper bounds on recovery complexity, revealing fundamental limits of TTA and an intrinsic adaptivity-information trade-off. These results provide unified learnability guarantees for TTA that complement regret-based analyses.

Zhi Zhou, Ming Yang, Shi-Yu Tian +3

State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China

Stable Test-Time Adaptation Test-Time Adaptation Learnability Non-Stationarity Perspective Regret

May 26, 2026cs.LG

Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning

Self-Distillation Policy Optimization (SDPO) provides dense token-level credit assignment for reinforcement learning with large language models by leveraging the model's own feedback-conditioned predictions as a self-teacher. Unlike GRPO, however, whose group-relative advantage naturally concentrates learning on a sweet spot of intermediate-difficulty questions, SDPO's KL-based advantage lacks an implicit notion of difficulty awareness. We analyze this gap through the lens of GRPO's advantage normalization. Extending the learnability framework to normalized rewards, we show that normalization absorbs the variance term

p(1-p)

, equalizing leading-order learnability across questions and leaving

\sqrt{p(1-p)}

as the sole residual scaling factor in the per-question gradient. This analysis yields a simple prescription: weight each question's SDPO loss by

[\hat{p}(1-\hat{p})]^{1/2}

, resulting in SC-SDPO, a scale-consistent variant of SDPO. The proposed weights are obtained as a zero-cost byproduct of on-policy rollouts with batch-adaptive normalization, inducing an implicit curriculum that dynamically tracks the model's evolving competence. Experiments on scientific reasoning and tool-use benchmarks demonstrate that SC-SDPO consistently improves over SDPO, yielding gains of +3.2/+4.3 (mean@16/maj@16) on Qwen3-8B and +1.8/+3.0 on OLMo-3-7B, while preserving stable training dynamics throughout optimization.

Zehao Liu, Yuanpu Cao, Jinghui Chen +1

College of Information Sciences and Technology Pennsylvania State University

Unsupervised On-Policy Self-Distillation Self-Distillation Framework LLM Reasoning Strategies Credit Assignment Learnability

May 23, 2026cs.LG

A computational phase transition for learning-to-sample from Ising models

We study \emph{learning-to-sample} -- a basic algorithmic task underlying generative modeling -- for Ising models, a standard testbed for algorithmic ideas in both theoretical computer science and machine learning. Given i.i.d. samples of an unknown target distribution, the goal of learning-to-sample is to learn a computationally efficient generation procedure that produces new samples following approximately the same distribution. We construct a family of Ising models of constantly bounded-width which lie just beyond the spectral threshold

λ_{\max}(J)-λ_{\min}(J)=1

, and show that learning-to-sample for this family is computationally hard under standard cryptographic assumptions, even when the learner is given both polynomially many i.i.d. samples from the model and explicit access to its parameters. Combined with results of [AJKPV24,KLV25] showing tractability of learning-to-sample below the spectral threshold, this establishes a sharp computational phase transition at the spectral threshold. Moreover, combined with prior results on parameter learning for bounded-width Ising models [KM17,WSD19,VML20], this shows that learning-to-sample can be more difficult than parameter learning. Finally, we show that any efficient learner for these hard instances exhibits a natural memorization-hallucination dichotomy: the learner must either output configurations that, after a simple transformation, match the (transformed) training data or place substantial mass on configurations of negligible probability under the target distribution.

Andrej Risteski, Thuy-Duong Vuong

Machine Learning Department, Carnegie Mellon University. · Department of Computer Science and Engineering, UC San Diego.

Spatial Photonic Ising Machines Phase Transitions Learnability Adaptive Sampling Generative Models Spectral

May 21, 2026cs.LG

Toward Understanding Adversarial Distillation: Why Robust Teachers Fail

Adversarial Distillation aims to enhance student robustness by guiding the student with a robust teacher's soft labels within the min-max adversarial training framework, yet its success is notoriously inconsistent: a more robust teacher often fails to improve, or even harms, the student's robust generalization. In this paper, we identify a key mechanism of this teacher dependency: the misalignment between the teacher's supervisory confidence and the student's representational limitations on a consistent subset of training data -- the Robustly Unlearnable Set. We present a theoretical framework analyzing the feature learning dynamics of a two-layer neural network, demonstrating that this mismatch creates a dichotomy in distillation outcomes. We prove that when a teacher provides confident supervision on unlearnable samples, it compels the student to memorize spurious noise patterns that eventually overpower the learned robust signal, thereby driving robust overfitting. Conversely, a teacher that exhibits high uncertainty on these samples effectively suppresses noise memorization, allowing the student to rely solely on the learnable signal for robust generalization. We empirically validate our theory across both synthetic simulations and real-image classification datasets, confirming that robust overfitting is driven by the teacher's interaction with unlearnable samples. Finally, we demonstrate that a teacher's predictive entropy on unlearnable samples serves as a strong indicator of student robustness, validating our theoretical framework and offering a principled guideline for robust teacher selection.

Hongsin Lee, Hye Won Chung

School of Electrical Engineering, KAIST, Daejeon, Korea.

Teacher-Student Distillation Dataset Distillation Teacher Learnability Overfitting Memorization+1

May 20, 2026cs.LG

Polynomial-Time Robust Multiclass Linear Classification under Gaussian Marginals

We study the task of agnostic learning of multiclass linear classifiers under the Gaussian distribution. Given labeled examples

(x, y)

from a distribution over

\mathbb{R}^d \times [k]

, with Gaussian

x

-marginal, the goal is to output a hypothesis whose error is comparable to that of the best

k

-class linear classifier. While the binary case

k=2

has a well-developed algorithmic theory, much less is known for

k \ge 3

. Even for

k=3

, prior robust algorithms incur exponential dependence on the inverse of the desired accuracy in both complexity and representation size. In this work, we develop new structural results for multiclass linear classifiers and use them to design fully polynomial-time robust learners with dimension-independent error guarantees. Our first result shows that the standard multiclass perceptron algorithm requires super-polynomially many samples and updates, even with clean labels and Gaussian marginals, revealing a basic obstruction absent in the binary case. Our main positive result is a pairwise improper-learning framework which yields an efficient learner with error

\widetilde O(k^{3/2}\sqrt{\mathrm{opt}})+ε

for general

k

. Additionally, we develop a sharper localization-based framework which leads to error

O(\mathrm{opt})+ε

for

k=3

, and error

\mathrm{poly}(k)\mathrm{opt}+ε

for geometrically regular

k

-class linear classifiers.

Ilias Diakonikolas, Giannis Iakovidis, Mingchen Ma

University of Wisconsin-Madison

Multiclass Classification Learnability

May 20, 2026cs.LG

A Sharper Picture of Generalization in Transformers

We study transformers' generalization behavior on boolean domains from the perspective of the Fourier spectra of their target functions. In contrast to prior work (Edelman et al., 2022; Trauger & Tosh, 2024), which derived generalization bounds from Rademacher complexity, we investigate the feasibility of obtaining generalization bounds via PAC-Bayes theory. We show that sparse spectra concentrated on low-degree components enable low-sharpness constructions with good generalization properties. Our idea is to show the existence of flat minima implementing any boolean function of sparsity no greater than the context length, and then apply a PAC-Bayes bound to an idealized low-sharpness learner, resulting in a non-vacuous generalization bound. We use this to give a formal account of why chain-of-thought improves generalization for high-degree target functions, and show that the complexity parameters in our bound can be efficiently estimated via property testing. We evaluate predictions empirically and conduct a mechanistic interpretability study to support the realism of our theoretical construction in real transformers.

Paul Lintilhac, Sair Shaikh

Thayer School of Engineering Dartmouth College · Dartmouth College

Generalization Bounds Sharpness-Aware Minimization Transformer Architectures Learnability Pac-Bayesian Theory

May 19, 2026stat.ML

Sample Complexity of Transfer Learning: An Optimal Transport Approach

Transfer learning is an essential technique for many machine learning/AI models of complex structures such as large language models and generative AI. The essence of transfer learning is to leverage knowledge from resolved source tasks for a new target task, especially when the sample size

m

of the training data for the latter is low. In this work, we rigorously analyze the potential benefit of transfer learning in terms of sample efficiency. Specifically, taking an optimal transport viewpoint of transfer learning, we find that when the data dimension

d

is higher than

3

, the sample complexity for transfer learning is

O(m^{-(α+1)/d})

, with

α

indicating the smoothness of the data distribution, as opposed to the

O(m^{-p/d})

sample complexity for direct learning with

p

indicating the smoothness of the optimal target model. Our finding theoretically supports a better sample efficiency for transfer learning, when the target task is optimizing over a family of not-so-smooth models (i.e., highly complex networks with the possible use of non-smooth activation functions). Using image classification as an example, we numerically demonstrate the sample efficiency for transfer learning, that is, in the data hungry regime, the model performance can be significantly improved by transfer learning.

Haoyang Cao, Xin Guo, Wenpin Tang +1

Department of Applied Mathematics and Statistics, John Hopkins University, Baltimore, MD, USA · Department of Industrial Engineering and Operations Research, University of California, Berkeley, CA, USA · Department of Industrial Engineering and Operations Research, Columbia University, New York, NY, USA +1

Transfer Learning Learnability Differentiable Optimal Transport Sample Complexity Optimal Transport Approach Activation Functions

May 18, 2026cs.LG

Efficient and Noise-Tolerant PAC Learning of Multiclass Linear Classifiers

Noise-tolerant PAC learning of linear models has been of central interests in machine learning community since the last century. In recent years, many computationally-efficient algorithms have been proposed for the problem of learning linear threshold functions under multiple noise models. Yet, when the problem is considered under multiclass learning settings, i.e. when the number of classes

k

is at least

3

, it is unknown whether there exist computationally-efficient PAC learning algorithms when the data sets are maliciously corrupted. In this paper, we consider that the marginal distribution is a mixture of bounded variance distributions and the data sets satisfy a margin condition at the same time. We show that there exists a computationally-efficient algorithm that PAC learns multiclass linear classifiers

\{h_w:x\mapsto \arg\max_{y\in[k]}w_y\cdot x, x\in \mathbb{R}^d, w\in\mathbb{R}^{kd}\}

using at most

O(k^2\cdot (d\log d+\log k))

samples even under a constant rate of nasty noise. Our algorithm consists of two main ingredients: a cluster-based pruning scheme and a standard multiclass hinge loss minimization program. Even in the special case of binary setting, i.e.

k=2

, our result is strictly stronger than all prior works.

Rita Adhikari, Shiwei Zeng

Augusta University

Multiclass Classification Learnability Noise-Aware

May 16, 2026cs.CV

The Learnability Gap in Medical Latent Diffusion

Generative data augmentation with latent diffusion models is a promising strategy for addressing class imbalance in medical imaging, yet current approaches focus on perceptual fidelity and domain-specific autoencoder fine-tuning while neglecting a more fundamental bottleneck. We identify and formalize the learnability gap: large-scale pretrained autoencoders faithfully encode discriminative features for medical classification, as evidenced by near-lossless performance in reconstruction space, yet their latent representations are structured in ways that are difficult for classifiers to learn from. Across five autoencoder families and four medical benchmarks spanning chest radiography, dermatoscopy, computed tomography, and echocardiography, we show that this gap persists regardless of architecture, initialization strategy, or hyperparameter tuning, and that medical-domain fine-tuning of the autoencoder does not close it. To probe and partially narrow the gap, we develop noise-conditioned latent classifiers with FiLM layers and image-space distillation that offer 64x throughput and 120x memory gains over image-space models while serving as diagnostic tools for latent space quality. Our analysis provides a new framework for evaluating autoencoder latent spaces and identifies their structure, rather than their fidelity or domain specificity, as the primary obstacle to closing the performance gap between real and synthetic medical training data.

Mischa Dombrowski, Felix Nützel, Bernhard Kainz

FAU Erlangen-Nürnberg, Erlangen, DE · Department of Computing, Imperial College London, London, UK

Identity-Conditioned Latent Diffusion Models Latent Diffusion Model Conditional Variational Autoencoder Learnability

May 13, 2026stat.ML

What is Learnable in Valiant's Theory of the Learnable?

Valiant's 1984 paper is widely credited with introducing the PAC learning model, but it, in fact, introduced a different model: unlike PAC learning, the learner receives only positives, may issue membership queries, and must output a hypothesis with no false positives. Prior work characterized variants, including the case without queries. We revisit Valiant's original model and ask: Which classes are learnable in it? For every finite domain, including Valiant's Boolean-hypercube setting, we show that a class is learnable if and only if every realizable positive sample can be certified by a poly-size adaptive query-compression scheme. This is a new variant of sample compression where the learner certifies samples via a short interaction with the membership oracle. Our characterization shows that learnability in Valiant's model is strictly sandwiched between learnability in the PAC model and the variant of Valiant's model without membership queries. This is one of the rare cases where introducing membership queries changes the set of learnable classes, and not just the sample or computational complexity. Next, we study the natural extension of the model to arbitrary domains. While we do not obtain an exact characterization, our techniques readily generalize and show that the same strict sandwiching persists. Finally, we show that

d

-dimensional halfspaces, which are not learnable without queries, are learnable with queries: we give a

\mathrm{poly}(d) \tilde{O}(1/ε)

sample and

\mathrm{poly}(d) \mathrm{polylog}(1/ε)

query algorithm, and prove that at least

Ω(d)

samples or queries are necessary. To our knowledge, this is the first algorithm for halfspaces in Valiant's model. Together, these results uncover a surprisingly rich theory behind Valiant's original notion of learnability and introduce ideas that may be of independent interest in learning theory.

Steve Hanneke, Anay Mehrotra, Grigoris Velegkas +1

Purdue University · Stanford University · Google Research +1

Learnability O(\Bar{K}\Log N)$Singular Learning Theory Quantum Learning Theory

May 13, 2026cs.LG

Scale-Sensitive Shattering: Learnability and Evaluability at Optimal Scale

We study the optimal scale at which real-valued function classes exhibit uniform convergence and learnability. Our main result establishes a scale-sensitive generalization of the fundamental theorem of PAC learning: for every bounded real-valued class and every

γ>0

, uniform convergence at scale

γ

, agnostic learnability at scale

γ/2

, and finiteness of the fat-shattering dimension at every scale

γ'>γ

are equivalent. This resolves a question by Anthony and Bartlett (Cambridge Univ. Press 1999) on the precise scales governing learnability, refuting a conjecture attributed there to Phil Long that a multiplicative 2-factor gap is unavoidable, and improves the upper bounds of Bartlett and Long (JCSS 1998), which incur such a loss. The key technical ingredient is a direct bound on empirical

\ell_\infty

covering numbers, avoiding the standard detour through packing numbers. As a consequence, we obtain sharp asymptotic metric-entropy bounds in terms of the fat-shattering scale

γ

: an

O(\log^2 n)

bound holds already at scale

γ/2

, while an

O(\log n)

bound holds at scale

2γ

. We further show that the

O(\log^2 n)

bound is sometimes tight. These results resolve open questions by Alon et al. (JACM 1997) and Rudelson and Vershynin (Ann. of Math. 2006). As an application, we establish a sharp dichotomy for bounded integral probability metrics: every such IPM is either estimable or cannot be weakly evaluated within any multiplicative factor

c<3

, while

3

-weak evaluability always holds, resolving an open question from Aiyer et al. (ICML 2026). We also highlight several open questions on quantitative sample complexity and evaluability.

Shashaank Aiyer, Yishay Mansour, Shay Moran +2

University of Maryland · Tel Aviv University and Google Research · Technion and Google Research +1

Learnability Optimal Sample Complexity Convergence Sequential Scaling

May 13, 2026cs.LG

Strategic PAC Learnability via Geometric Definability

Strategic classification studies learning settings in which individuals can modify their features, at a cost, in order to influence the classifier's decision. A central question is how the sample complexity of the induced (strategic) hypothesis class depends on the complexities of the underlying hypothesis class and the cost structure governing feasible manipulations. Prior work has shown that in several natural settings, such as linear classifiers with norm costs, the induced complexity can be controlled. We begin by showing that such guarantees fail in general - even in simple cases: there exist hypothesis classes of VC dimension

1

on the real line such that, even under the simplest interval neighborhoods, the induced class has infinite VC dimension. Thus, strategic behavior can turn an easy learning problem into a non-learnable one. To overcome this, we introduce structure via a geometric definability assumption: both the hypothesis class and the cost-induced neighborhood relation can be defined by first-order formulas over

\mathbb{R}_{\mathtt{exp}}

. Intuitively, this means that hypotheses and costs can be described using arithmetic operations, exponentiation, logarithms, and comparisons. This captures a broad range of natural classes and cost functions, including

\ell_p

distances, Wasserstein distance, and information-theoretic divergences. Under this assumption, we prove that learnability is preserved, with sample complexity controlled by the complexity of the defining formulas.

Yuval Filmus, Shay Moran, Elizaveta Nesterova +2

Faculty of Mathematics, Technion – Israel Institute of Technology. · Faculty of Computer Science, Technion – Israel Institute of Technology. · Faculty of Data and Decision Sciences, Technion – Israel Institute of Technology. +1

Hypothesis Class Learnability Structural Complexity Voice Conversion

May 12, 2026cs.LG

High-arity Sample Compression

Recently, a series of works have started studying variations of concepts from learning theory for product spaces, which can be collected under the name high-arity learning theory. In this work, we consider a high-arity variant of sample compression schemes and we prove that the existence of a high-arity sample compression scheme of non-trivial quality implies high-arity PAC learnability.

Leonardo N. Coregliano, William Opich

Learnability Data Compression Methods

May 9, 2026cs.LG

A Complete Characterization of Learnability for Adversarial Noisy Bandits

We study adversarial noisy bandits given a known function class

\mathcal{F}

. In each round, the adversary selects a function

f \in \mathcal{F}

, the learner chooses an arm, and then observes a noisy reward determined by the chosen arm and the function

f

. The goal is to minimize the cumulative regret

R(T)

, defined as the difference between the learner's performance and that of the best fixed arm in hindsight over

T

rounds. We say that a function class

\mathcal{F}

is learnable if there exists an algorithm achieving sublinear regret. Our main result is a complete characterization of learnability for adversarial noisy bandits. The characterization is given in terms of a convexified variant of the generalized maximin volume introduced by Hanneke and Wang (2025): namely, the generalized maximin volume evaluated on the convex hull

\operatorname{co}(\mathcal F)

. We prove that

\mathcal F

is learnable if and only if this convexified generalized maximin volume is positive at every scale. This condition characterizes learnability against both oblivious and adaptive adversaries, showing in particular that these two notions of learnability are equivalent in the noisy bandit setting. Our analysis reveals that the key complexity measure is closely connected to two new combinatorial notions, hitting set and distribution covering number, which may be of independent interest. These results establish the first complete characterization of learnability for adversarial noisy bandits.

Steve Hanneke, Kun Wang

Department of Computer Science, Purdue University

Learnability Combinations Completion

May 8, 2026stat.ML

Learnability and Competition in High-Dimensional Multi-Component ICA

Independent Component Analysis (ICA) is a foundational tool for unsupervised representation learning, yet its high-dimensional theory remains largely limited to single-component recovery. We develop an asymptotically exact mean-field theory for multi-component online ICA, capturing the coupling induced by simultaneous learning and orthogonalization. In the high-dimensional limit, the joint empirical distribution of learned estimates and ground-truth components converges to a deterministic process, yielding a closed ODE system for the overlap matrix between learned directions and true components. This characterization reveals a genuinely multi-component, initialization-driven phase structure: a decoupled regime, where estimates align with distinct components and evolve nearly independently, and a competition regime, where overlapping initializations induce orthogonality-driven conflicts, slow reorientation, and delayed convergence. Our steady-state analysis gives explicit learnability boundaries and competition conditions linking step size, data moments, and initialization. These conditions show that larger higher-order moments and competition shrink the stable learning-rate window, increase convergence times, and predict a staircase phenomenon in which the number of recoverable components changes discretely with the learning rate. Experiments on synthetic data and hyperspectral remote sensing data validate the predicted trajectories and phase behavior.

Eser Ilke Genc, Samet Demir, Zafer Dogan

MLIP Research Group, KUIS AI Center, Koç University · Department of EEE, Koç University

Independent Component Analysis High-Dimensional Learnability Competition

May 8, 2026stat.ML

Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity

We show that, in a precise sense, a broad class of feedforward neural networks learn (have finite sample complexity) in the PAC model: every fixed finite feedforward architecture whose layers are definable in an o-minimal structure has finite sample complexity in the agnostic PAC setting, even with unbounded parameters. This covers standard fixed-size MLPs, CNNs, GNNs, and transformers with fixed sequence length, together with the operations and layers typically used in such architectures, including linear projections, residual connections, attention mechanisms, pooling layers, normalization layers, and admissible positional encodings. Hence, distribution-free learnability for modern non-recurrent architectures is not an exceptional property of particular activations or architecture-specific VC arguments, but a consequence of tame feedforward computation. Our results reposition finite-sample PAC learnability as a baseline rather than a differentiator: they shift the focus of architectural comparison toward inductive biases, symmetries and geometric priors, scalability, and optimization behaviour.

Anastasis Kratsios, Gregory Cousins, Haitz Sáez de Ocáriz Borde +2

Department of Mathematics & Statistics, McMaster University, Hamilton, ON, Canada · University of Cambridge, United Kingdom · Graduate School of Engineering, The University of Tokyo, Tokyo, Japan +1

Feed-Forward Learnability Multilayer Perceptrons Neural Network Optimal Sample Complexity

May 7, 2026cs.DS

Equivalence of Coarse and Fine-Grained Models for Learning with Distribution Shift

Recent work on provably efficient algorithms for learning with distribution shift has focused on two models: PQ learning (Goldwasser et al. (2020)) and TDS learning (Klivans et al. (2024)). Algorithms for TDS learning are allowed to reject a test set entirely if distribution shift is detected. In contrast, PQ learners may only reject points that are deemed out-of-distribution on an individual basis. Our main result is a surprising equivalence between these two models in the distribution-free setting. In particular, we give an efficient black-box reduction from PQ learning to TDS learning for any Boolean concept class. This equivalence implies the first hardness results for distribution-free TDS learning of basic classes such as halfspaces. The main technical contribution underlying our equivalence is a method for boosting, via branching programs, the weak distinguishing power of TDS learners that have rejected the target domain. We also show that giving a learner access to membership queries sidesteps these hardness results and allows for efficient, distribution-free PQ learnability of halfspaces. Our algorithm iteratively recovers large-margin separators obtained by applying successive Forster transforms on the training data.

Adam R. Klivans, Shyamal Patel, Konstantinos Stavropoulos +1

UT Austin · Columbia University

Learnability Distribution Shifts O(\Bar{K}\Log N)$Equivalence Temporal Difference Coarse

May 7, 2026cs.AI

Temporal Smoothness Doubly Robust Learning for Debiased Knowledge Tracing

Knowledge Tracing (KT) is fundamental to intelligent education systems, yet relies on educational logs that are selectively observed. The non-random nature of exercise recommendations and student choices inevitably induces severe selection bias. Most existing KT methods neglect this issue, training on observed logs using standard empirical risk, which yields biased mastery estimates and accumulates errors in subsequent recommendations. To address this, we introduce a doubly robust (DR) formulation for KT that integrates a propensity model with an error imputation model, theoretically guaranteeing unbiasedness if either model is accurate. Beyond unbiasedness, in the sequential setting of KT, we identify that the estimator's performance is compromised by variance-dependent stochastic deviations that accumulate over time, thereby causing training instability and limiting performance. To mitigate this, we derive a generalization bound that explicitly characterizes the impact of estimator variance and identifies temporal smoothness as a key factor in controlling it. Building on these theoretical insights, we propose the Temporal Smoothness Doubly Robust (TSDR) framework. TSDR jointly optimizes the KT predictor and the imputation model with a smoothness regularizer, effectively reducing variance while preserving the unbiasedness guarantee of DR. Experiments on multiple real-world benchmarks demonstrate that TSDR consistently enhances various state-of-the-art KT backbones, underscoring the vital role of principled bias correction in KT.

Peilin Zhan, Wei Chen, Weilin Chen +2

School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China

Learnability Selection Bias Smoothness Imputation

May 6, 2026cs.LG

Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learning

We compare in-context learning with fixed queries and agentic learning with adaptive queries for uniform approximation of task families. We consider two settings: an unrestricted regime, where querying and approximation are arbitrary functions, and a realizable regime, where we require these operations to be implemented by ReLU neural networks. In both settings, adaptivity never hinders approximation performance. However, this advantage can change when one passes from the unrestricted regime to the realizable regime. We identify four distinct approximation scenarios, each witnessed by an explicit task family: (a) no advantage of adaptivity; (b) an advantage in the unrestricted regime that persists under ReLU realizability; (c) an advantage that arises only under realizability; and (d) an advantage that disappears under realizability. This demonstrates that representational constraints interact profoundly with the effect of adaptivity.

Anastasis Kratsios, A. Martina Neuman, Philipp Petersen

McMaster University and the Vector Institute, Main St., Hamilton, L8S 4L8, Ontario, Canada · University of Vienna, Faculty of Mathematics, Kolingasse 14-16, 1090 Wien, Austria

Learnability In-Context Learning Agentic Learning Approximation In-Context Rectified Linear Unit+1

May 3, 2026cs.CV

Dual-branch Robust Unlearnable Examples

Unlearnable examples (UEs) aim to compromise model training by injecting imperceptible perturbations to clean samples. However, existing UE schemes exhibit limited robustness against advanced defenses due to their heuristic design or narrowly scoped domain perturbations. To address this, we propose \texttt{DUNE}, a \underline{\textbf{D}}ual-branch \underline{\textbf{UN}}learnable \underline{\textbf{E}}nsemble perturbation optimization approach. Specifically, \texttt{DUNE} separately optimizes perturbations in the spatial and color domains to establish the mapping between perturbations and shift-induced labels. This design extends the perturbation domain to increase noise intensity for improving robustness and drives the models to learn perturbation-oriented features with degraded generalization, thereby achieving unlearnability. To strengthen \texttt{DUNE}'s performance, we further propose an unlearnability-enhancing ensemble strategy that aggregates diverse pre-trained models during the dual-branch optimization. Extensive experiments on benchmark datasets CIFAR-10 and ImageNet verify that \texttt{DUNE}'s robustness outperforms 12 SOTA UE schemes under 7 mainstream defenses, yielding a lower average test accuracy of 14.95% to 50.82%.

Xianlong Wang, Hangtao Zhang, Wenbo Pan +4

Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · University of Pennsylvania, Philadelphia, USA · Huazhong University of Science and Technology, Wuhan, China +2

Unlearnable Examples Learnability Ensemble

Apr 30, 2026cs.AI

Robust Learning on Heterogeneous Graphs with Heterophily: A Graph Structure Learning Approach

Heterogeneous graphs with heterophily have emerged as a powerful abstraction for modeling complex real-world systems, where nodes of different types and labels interact in diverse and often non-homophilous ways. Despite recent advances, robust representation learning for such graphs remains largely unexplored, particularly in the presence of noisy or misleading connectivity. In this work, we investigate this problem and identify structural noise as a critical challenge that significantly degrades model performance. To address this issue, we propose a unified framework, Heterogeneous Graph Unified Learning (HGUL), which jointly handles heterophily and noisy graph structures. The framework consists of three complementary modules: a kNN-based graph construction module that recovers reliable local neighborhoods, a graph structure learning module that adaptively refines the adjacency by filtering noisy edges, and a heterogeneous affinity learning module that captures class-level relationships via an extended affinity matrix derived from a polynomial graph kernel. Extensive experiments on multiple datasets demonstrate that HGUL consistently outperforms existing methods on clean graphs and maintains strong robustness under varying levels of structural noise. The results further underscore the importance of jointly modeling heterophily and noise in heterogeneous graph learning.

Yihan Zhang, Ercan E. Kuruoglu

Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

Graph Representation Learning Text-Attributed Graph Learnability Kernel Method

Apr 29, 2026cs.LG

Near-Optimal Cryptographic Hardness of Learning With Homogeneous Halfspaces Under Gaussian Marginals

We study three problems that involve identifying homogeneous halfspaces under Gaussian distributions: agnostic learning, one-sided reliable learning, and fairness auditing. In each of these problems, we are given labeled examples

(\mathbf{x}, \mathrm{y})

drawn from an unknown distribution on

\mathbb{R}^d\times\{-1, +1\}

, whose marginal distribution on

\mathbf{x}

is standard Gaussian and on

\mathrm{y}

is arbitrary. The goal of each problem is to output a homogeneous halfspace that approaches the best-fitting homogeneous halfspace in terms of its corresponding loss measure. We prove near-optimal computational hardness results for these problems under the widely believed hardness assumption of the Learning With Errors (LWE) problem. Prior hardness results for these problems were mostly established for general halfspaces; our findings extend some of these hardness results to homogeneous halfspaces. Remarkably, our lower bound strictly generalizes over prior works and narrows the gap between the upper and lower bounds for agnostically learning homogeneous halfspaces under Gaussian marginals.

Jizhou Huang, Brendan Juba

Meta · Washington Universtiy in St. Louis

Learnability Np-Hard

Apr 28, 2026cs.LG

Barriers to Universal Reasoning With Transformers (And How to Overcome Them)

Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness. However, whether Transformers can learn to generalize to CoT traces longer than those seen during training is understudied. We use recent theoretical frameworks for Transformer length generalization and find that -- under standard positional encodings and a finite alphabet -- Transformers with CoT cannot solve problems beyond

TC^0

, i.e. the expressivity benefits do not hold under the stricter requirement of length-generalizable learnability. However, if we allow the vocabulary to grow with problem size, we attain a length-generalizable simulation of Turing machines where the CoT trace length is linear in the simulated runtime up to a constant. Our construction overcomes two core obstacles to reliable length generalization: repeated copying and last-occurrence retrieval. We assign each tape position a unique signpost token, and log only value changes to enable recovery of the current tape symbol through counts circumventing both barriers. Further, we empirically show that the use of such signpost tokens and value change encodings provide actionable guidance to improve length generalization on hard problems.

Oliver Kraus, Yash Sarrof, Yuekun Yao +2

1Saarland University · 2Ohio State University

Transformer Architectures Chain-of-Thought Reasoning Learnability Barriers

Apr 27, 2026cs.LG

Null Measurability at the Symmetrization Interface in VC Learning

Recent work revisiting measurability in the fundamental theorem of statistical learning imposes Borel measurability of ghost-gap suprema. We show that, at the one-sided ghost-gap interface actually used by the standard symmetrization proof, this requirement is stronger than necessary. For any Borel-parameterized concept class on a Polish domain, the bad event "there exists a hypothesis whose ghost empirical error exceeds its training empirical error by at least ε/2" is analytic. By Choquet capacitability, it is therefore measurable in the completion of every finite Borel measure. We then construct a concept class whose bad event is null-measurable but not Borel, giving a strict separation from the Borel supremum condition. Finally, we prove closure under patching, fixed and countable interpolation, and fiber-product amalgamation, showing that the weaker regularity level is stable under natural concept-class constructors. In the realizable setting, where targets belong to the class and are measurable, these results weaken the measurability hypothesis needed by the symmetrization route from finite VC dimension to PAC learnability. The main results and the descriptive-set-theoretic infrastructure used by them are formalized in Lean 4.

Dhruv Gupta

AI and Robotics Technology Park, Indian Institute of Science, Bengaluru

Probability Measures Learnability Pac-Bayesian Theory Measures Symmetry Voice Conversion

Apr 27, 2026cs.LG

Learning to Think from Multiple Thinkers

We study learning with Chain-of-Thought (CoT) supervision from multiple thinkers, all of whom provide correct but possibly systematically different solutions, e.g., step-by-step solutions to math problems written by different thinkers, or step-by-step execution traces of different programs solving the same problem. We consider classes that are computationally easy to learn using CoT supervision from a single thinker, but hard to learn with only end-result supervision, i.e., without CoT (Joshi et al. 2025). We establish that, under cryptographic assumptions, learning can be hard from CoT supervision provided by two or a few different thinkers, in passive data-collection settings. On the other hand, we provide a generic computationally efficient active learning algorithm that learns with a small amount of CoT data per thinker that is completely independent of the target accuracy

\varepsilon

, a moderate number of thinkers that scales as

\log \frac{1}{\varepsilon}\log \log \frac{1}{\varepsilon}

, and sufficient passive end-result data that scales as

\frac{1}{\varepsilon}\cdot poly\log\frac{1}{\varepsilon}

.

Nirmit Joshi, Roey Magen, Nathan Srebro +2

Toyota Technological Institute at Chicago. · Weizmann Institute of Science. · New York University.

Learnability Chain-of-Thought Reasoning Process-Level Supervision Pool-Based Active Learning Think

Apr 25, 2026math.NT

On (not) learning the Möbius function

We prove lower bounds on learning the Möbius or Liouville function with a variety of standard learning techniques, including kernel methods, noisy gradient methods, and correlational statistical query algorithms. These results follow from quantitative bounds on the correlation of Möbius with digital characters of various finite abelian groups, where the group is dictated by the type of input data the algorithm is given. Using residues mod

p

for many different primes corresponds to a cyclic group, and using the base

p

expansion for a fixed prime corresponds to an elementary abelian

p

-group. We also note that lower bounds of this form are closely related to certain types of digital prime number theorems.

Alexey Pozdnyakov

Finite Groups Learnability Lyapunov Function

Apr 23, 2026cs.LG

Graph Neural Network-Informed Predictive Flows for Faster Ford-Fulkerson and PAC-Learnability

We propose a learning-augmented framework for accelerating max-flow computation and image segmentation by integrating Graph Neural Networks (GNNs) with the Ford-Fulkerson algorithm. Rather than predicting initial flows, our method learns edge importance probabilities to guide augmenting path selection. We introduce a Message Passing GNN (MPGNN) that jointly learns node and edge embeddings through coupled updates, capturing both global structure and local flow dynamics such as residual capacity and bottlenecks. Given an input image, we propose a method to construct a grid-based flow network with source and sink nodes, extract features, and perform a single GNN inference to assign edge probabilities reflecting their likelihood of belonging to high-capacity cuts. These probabilities are stored in a priority queue and used to guide a modified Ford-Fulkerson procedure, prioritizing augmenting paths via an Edmonds-Karp-style search with bottleneck-aware tie-breaking. This avoids repeated inference over residual graphs while leveraging learned structure throughout optimization. We further introduce a bidirectional path construction strategy centered on high-probability edges and provide a theoretical framework relating prediction quality to efficiency via a weighted permutation distance metric. Our method preserves max-flow/min-cut optimality while reducing the number of augmentations in practice. We also outline a hybrid extension combining flow warm-starting with edge-priority prediction, establishing a foundation for learning-guided combinatorial optimization in image segmentation.

Eleanor Wiesler, Trace Baxley

1Harvard Department of Mathematics. · 2Harvard Department of Computer Science.

Graph Neural Networks Edge-Aware Neural Network Inference Learnability Importance Weighting Artificial Intelligence Search+1

Apr 20, 2026cs.CV

HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval

Composed Image Retrieval (CIR) is a flexible image retrieval paradigm that enables users to accurately locate the target image through a multimodal query composed of a reference image and modification text. Although this task has demonstrated promising applications in personalized search and recommendation systems, it encounters a severe challenge in practical scenarios known as the Noise Triplet Correspondence (NTC) problem. This issue primarily arises from the high cost and subjectivity involved in annotating triplet data. To address this problem, we identify two central challenges: the precise estimation of composed semantic discrepancy and the insufficient progressive adaptation to modification discrepancy. To tackle these challenges, we propose a cHrono-synergiA roBust progressIve learning framework for composed image reTrieval (HABIT), which consists of two core modules. First, the Mutual Knowledge Estimation Module quantifies sample cleanliness by calculating the Transition Rate of mutual information between the composed feature and the target image, thereby effectively identifying clean samples that align with the intended modification semantics. Second, the Dual-consistency Progressive Learning Module introduces a collaborative mechanism between the historical and current models, simulating human habit formation to retain good habits and calibrate bad habits, ultimately enabling robust learning under the presence of NTC. Extensive experiments conducted on two standard CIR datasets demonstrate that HABIT significantly outperforms most methods under various noise ratios, exhibiting superior robustness and retrieval performance. Codes are available at https://github.com/Lee-zixu/HABIT

Zixu Li, Yupeng Hu, Zhiwei Chen +4

School of Software, Shandong University

Composed Image Retrieval Multimodal Retrieval Learnability Chronos

Apr 19, 2026cs.LG

Diverse Dictionary Learning

Given only observational data

X = g(Z)

, where both the latent variables

Z

and the generating process

g

are unknown, recovering

Z

is ill-posed without additional assumptions. Existing methods often assume linearity or rely on auxiliary supervision and functional constraints. However, such assumptions are rarely verifiable in practice, and most theoretical guarantees break down under even mild violations, leaving uncertainty about how to reliably understand the hidden world. To make identifiability actionable in the real-world scenarios, we take a complementary view: in the general settings where full identifiability is unattainable, what can still be recovered with guarantees, and what biases could be universally adopted? We introduce the problem of diverse dictionary learning to formalize this view. Specifically, we show that intersections, complements, and symmetric differences of latent variables linked to arbitrary observations, along with the latent-to-observed dependency structure, are still identifiable up to appropriate indeterminacies even without strong assumptions. These set-theoretic results can be composed using set algebra to construct structured and essential views of the hidden world, such as genus-differentia definitions. When sufficient structural diversity is present, they further imply full identifiability of all latent variables. Notably, all identifiability benefits follow from a simple inductive bias during estimation that can be readily integrated into most models. We validate the theory and demonstrate the benefits of the bias on both synthetic and real-world data.

Yujia Zheng, Zijian Li, Shunxing Fan +2

CMU · MBZUAI · NYU

Unsupervised Dictionary Learning Identifiability Latent Variable Learnability Inductive Bias

Apr 16, 2026cs.DS

Tight Bounds for Learning Polyhedra with a Margin

We give an algorithm for PAC learning intersections of

k

halfspaces with a

ρ

margin to within error

\varepsilon

that runs in time

\textsf{poly}(k, \varepsilon^{-1}, ρ^{-1}) \cdot \exp \left(O(\sqrt{n \log(1/ρ) \log k})\right)

. Notably, this improves on prior work which had an exponential dependence on either

k

or

ρ^{-1}

and matches known cryptographic and Statistical Query lower bounds up to the logarithmic factors in

k

and

ρ

in the exponent. Our learning algorithm extends to the more general setting when we are only promised that most points have distance at least

ρ

from the boundary of the polyhedron, making it applicable to continuous distributions as well.

Shyamal Patel, Santosh Vempala

Columbia University · Georgia Tech

O(\Bar{K}\Log N)$Polytopes Learnability Lower Bounds Polynomials

Apr 13, 2026cs.LG

THEIA: Learning Complete Kleene Three-Valued Logic in a Pure-Neural Modular Architecture

We present THEIA, a 2.75M-parameter modular neural architecture that learns the complete Kleene three-valued logic (K3) truth table from task data without external symbolic inference or hand-encoded K3 gate primitives. Across 5 seeds it passes all 39 K3 rules at >99% per-rule accuracy. K3 learnability is not the central finding: Transformer baselines also pass all 39 rules, and flat MLPs match THEIA on Phase-1 accuracy within 0.04pp. The contributions are two properties of the learned system. (1) Uncertainty-verdict asymmetric propagation. THEIA preserves Has-Unknown at every upstream boundary (80.0/91.1/90.8/99.7% across Arith/Order/Set/Logic vs. ~52% majority) while final-verdict decodability stays at or below a 73.4% U-vs-non-U oracle reference under linear and nonlinear probes. Activation patching on non-absorbent T->U cases flips 4,898/4,898 OR and 4,719/4,719 AND pairs across 5 seeds, ruling out residual shortcuts. (2) Reliability spectrum under discretized end-to-end training, on tasks decomposable along the engine boundaries. A mod-3 sequential composition task generalizes from 5- to 500-step evaluation at 99.96+-0.04% (5 seeds). Under identical Gumbel-softmax training, flat MLPs collapse to chance by 50 steps; a 2x2 ResMLP grid reaches >=99% on only 3/20 (config, seed) trials; a pre-LN Transformer reaches 99.24+-0.34%. Straight-through discretization prevents 0.999^500 compounding; the architectural separator is sustaining Phase-1 accuracy under Phase-3 training, where flat MLPs fail. Auxiliary: under per-architecture development defaults (not optimizer-controlled), THEIA reaches 12/12 Kleene coverage 6.5x faster than a parameter-comparable 8L Transformer; this narrows to ~3.6x with Transformer-standard tuning and 4.93x with the same recipe on both. Ratios are config-specific, not asymptotic.

Augustus Haoyang Li

Irvine Valley College Irvine, CA, USA

Logic Multilayer Perceptrons Natural Language Inference Learnability Transformer Architectures

Mar 7, 2026cs.LG

Margin in Abstract Spaces

Margin-based learning, exemplified by linear and kernel methods, is one of the few classical settings where generalization guarantees are independent of the number of parameters. This makes it a central case study in modern highly over-parameterized learning. We ask what minimal mathematical structure underlies this phenomenon. We begin with a simple margin-based problem in arbitrary metric spaces: concepts are defined by a center point and classify points according to whether their distance lies below

r

or above

R

. We show that whenever

R>3r

, this class is learnable in \emph{any} metric space. Thus, sufficiently large margins make learnability rely only on the triangle inequality, without any linear or analytic structure being necessary. Our first main result extends this phenomenon to concepts defined by bounded linear combinations of distance functions, and reveals a sharp threshold: there exists a universal constant such that whenever the margin is larger than this constant, the class is learnable in every metric space, while below it there exist metric spaces where it is not learnable at all. We then ask whether margin-based learnability can always be explained via an embedding into a linear space -- that is, reduced to linear classification in some Banach space through a kernel-type construction. We answer this negatively by demonstrating a margin learnable class that cannot be embedded into any Banach space in which linear classification with margins is learnable.

Yair Ashlagi, Roi Livni, Shay Moran +1

School of Electrical and Computer Engineering, Tel Aviv University. · Departments of Mathematics, Computer Science, and Data and Decision Sciences, Technion, and Google Research. · Departments of Mathematics, Technion.

Metric Spaces Learnability Overparameterization

Jan 27, 2026cs.LG

Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining

The empirical success of deep learning is often attributed to deep networks' ability to exploit hierarchical structure in data, constructing increasingly complex features across layers. Yet despite substantial progress in deep learning theory, most optimization results sill focus on networks with only two or three layers, leaving the theoretical understanding of hierarchical learning in genuinely deep models limited. This leads to a natural question: can we prove that deep networks, trained with gradient-based methods and standard input-label pairs, can efficiently exploit hierarchical structure? In this work, we consider Random Hierarchy Models -- a hierarchical context-free grammar introduced by arXiv:2307.02129 and conjectured to separate deep and shallow networks. We prove that, under mild conditions, a deep convolutional network can be efficiently trained to learn this function class. Our proof builds on a general observation: if intermediate layers can receive clean signal from the labels and the relevant features are weakly identifiable, then layerwise training each individual layer suffices to hierarchically learn the target function.

Yunwei Ren, Yatin Dandi, Florent Krzakala +1

Princeton University · École Polytechnique Fédérale de Lausanne · University of California, Berkeley

Deep Network Learnability Hierarchical Circuit Depth

Dec 5, 2025cs.LG

Learnability Window in Gated Recurrent Neural Networks

We develop a statistical theory of temporal learnability in recurrent neural networks, quantifying the maximal temporal horizon

\mathcal{H}_N

over which gradient-based learning can recover lag-dependent structure at finite sample size

N

. The theory is built on the effective learning rate envelope

f(\ell)

, a function that captures how gating mechanisms and adaptive optimizers jointly shape the coupling between state-space dynamics and parameter updates during Backpropagation Through Time. Under heavy-tailed (

α

-stable) fluctuations, where empirical averages concentrate at rate

N^{-1/κ_α}

with

κ_α= α/(α-1)

, the interplay between envelope decay and statistical concentration yields explicit scaling laws for the growth of

\mathcal{H}_N

: logarithmic, polynomial, and exponential temporal learning regimes emerge according to the decay law of

f(\ell)

. These results identify envelope decay as the key determinant of temporal learnability. Slower attenuation of

f(\ell)

enlarges

\mathcal{H}_N

, while heavy-tailed fluctuations compress it by weakening statistical concentration. Moreover, envelope geometry outweighs dataset size: slowing the envelope's decay enlarges

\mathcal{H}_N

more than adding data, so more complex architectures that realize slower-decaying envelopes can be more data-efficient than simpler ones. Experiments across multiple gated architectures and optimizers corroborate these structural predictions.

Lorenzo Livi

OPIT – Open Institute of Technology

Recurrent Neural Networks Learnability Batch Temporal Difference

Nov 27, 2025cs.CV

Anchor to Expand: Semantic Anchoring for Personalized Text-to-Image Diffusion Models

Personalizing text-to-image diffusion models extends pretrained models to represent novel user-specific concepts from only a few reference images. However, learning a new concept while building on the prior knowledge of the pretrained model remains a key challenge. When personalization focuses on learning the target concept, the model tends to overfit the reference examples and degrade its general capability. In contrast, emphasizing prior preservation can hinder capturing distinctive personalized attributes. In this paper, we address this challenge by viewing a personalized concept as an underrepresented concept whose semantic counterpart is well represented in the pretrained model. Rather than treating the learning of a new concept and prior preservation as separate objectives, we reformulate them as a single anchored learning problem. We therefore introduce Semantic Anchoring Personalization (SAP), which keeps concept learning grounded in the pretrained semantic structure while capturing subject-specific attributes. The proposed objective offers a simple yet effective formulation that can be applied across different model backbones without architectural modifications or auxiliary networks. Extensive experiments across various settings demonstrate that SAP achieves a better balance between subject fidelity and text-image alignment than baseline methods. Further ablation studies validate the contribution of semantic anchoring to personalization.

Seoyun Yang, Gihoon Kim, Taesup Kim

Graduate School of Data Science, Seoul National University

Text-To-Image Diffusion Models Image-Text Alignment Diffusion Models Semantic Anchor Personalization Fidelity+1

Nov 25, 2025cs.LG

Pre-train to Gain: Robust Learning Without Clean Labels

Training deep networks with noisy labels leads to poor generalization and degraded accuracy due to overfitting to label noise. Existing approaches for learning with noisy labels often rely on the availability of a clean subset of data. By pre-training a feature extractor on the target dataset without labels using in-domain self-supervised learning (SSL), followed by standard supervised training on the same noisy dataset, we can train a more noise robust model without requiring a subset with clean labels. We evaluate both contrastive and non-contrastive SSL pre-training methods across datasets with synthetic and real-world label noise, demonstrating the broad applicability of our approach across large-scale datasets, diverse downstream tasks, and model architectures. Across all noise rates, in-domain self-supervised pre-training consistently improves classification accuracy and downstream label-error detection (F1 and Balanced Accuracy) compared with supervised training from scratch. The performance gap widens as the noise rate increases, demonstrating improved robustness. Notably, our approach achieves comparable results to ImageNet and DinoV2 pre-trained models at low noise levels, while substantially outperforming them under high noise conditions.

David Szczecina, Nicholas Pellegrino, Paul Fieguth

Systems Design Engineering University of Waterloo Waterloo, Ontario, Canada

Noisy Labels Self-Supervised Learning Pretraining Learnability Contrastive Learning Deep Network

Nov 14, 2025cs.DS

Learning and Testing Convex Functions

We consider the problems of \emph{learning} and \emph{testing} real-valued convex functions over Gaussian space. Despite the extensive study of function convexity across mathematics, statistics, and computer science, its learnability and testability have largely been examined only in discrete or restricted settings -- typically with respect to the Hamming distance, which is ill-suited for real-valued functions. In contrast, we study these problems in high dimensions under the standard Gaussian measure, assuming sample access to the function and a mild smoothness condition, namely Lipschitzness. A smoothness assumption is natural and, in fact, necessary even in one dimension: without it, convexity cannot be inferred from finitely many samples. As our main results, we give: - Learning Convex Functions: An agnostic proper learning algorithm for Lipschitz convex functions that achieves error

\varepsilon

using

n^{O(1/\varepsilon^2)}

samples, together with a complementary lower bound of

n^{\mathrm{poly}(1/\varepsilon)}

samples in the \emph{correlational statistical query (CSQ)} model. - Testing Convex Functions: A tolerant (two-sided) tester for convexity of Lipschitz functions with the same sample complexity (as a corollary of our learning result), and a one-sided tester (which never rejects convex functions) using

O(\sqrt{n}/\varepsilon)^n

samples.

Renato Ferreira Pinto, Cassandra Marcussen, Elchanan Mossel +1

Columbia University · Harvard University · MIT

Nonconvex Optimal Sample Complexity Sample Complexity Learnability Gaussian Primitives

Nov 4, 2025cs.LG

Recursively Enumerably Representable Classes and Computable Versions of the Fundamental Theorem of Statistical Learning

We study computable probably approximately correct (CPAC) learning, where learners are required to be computable functions. It had been previously observed that the Fundamental Theorem of Statistical Learning, which characterizes PAC learnability by finiteness of the Vapnik-Chervonenkis (VC-)dimension, no longer holds in this framework. Recent works recovered analogs of the Fundamental Theorem in the computable setting, for instance by introducing an effective VC-dimension. Guided by this, we investigate the connection between CPAC learning and recursively enumerable representable (RER) classes, whose members can be algorithmically listed. Our results show that the effective VC-dimensions can take arbitrary values above the traditional one, even for RER classes, which creates a whole family of (non-)examples for various notions of CPAC learning. Yet the two dimensions coincide for classes satisfying sufficiently strong notions of CPAC learning. We then observe that CPAC learnability can also be characterized via containment of RER classes that realize the same samples. Furthermore, it is shown that CPAC learnable classes satisfying a unique identification property are necessarily RER. Finally, we establish that agnostic learnability can be guaranteed for RER classes, by considering the relaxed notion of nonuniform CPAC learning.

David Kattermann, Lothar Sebastian Krapp

Institute for Symbolic Artificial Intelligence, Johannes Kepler University Linz, Austria. · Department of Mathematics and Statistics, University of Konstanz, Germany. · Institute for the Interdisciplinary Study of Language Evolution, University of Zurich, Switzerland.

Learnability Tractability Intrinsic Dimensionality

Jun 22, 2025cs.LG

Tight Stability Bounds for Robust Distributed Learning: Byzantine Failures Hurt Generalization More than Data Poisoning

Robust distributed learning algorithms aim to maintain reliable performance despite the presence of misbehaving workers. Such misbehaviors are commonly modeled as \textit{Byzantine failures}, allowing arbitrarily corrupted communication, or as \textit{data poisoning}, a weaker form of corruption restricted to local training data. While prior work shows similar optimization guarantees for both models, an important question remains: \textit{How do these threat models impact generalization?} We show, for the first time, a fundamental gap in generalization guarantees between the two threat models: Byzantine failures yield strictly worse rates than those achievable under data poisoning. Our findings are based upon a tight algorithmic stability analysis of robust distributed learning. Specifically, with

f

out of

n

workers misbehaving, we prove that: \textit{(i)} under data poisoning, the uniform algorithmic stability of a robust distributed learning algorithm

Thomas Boudou, Batiste Le Bars, Nirupam Gupta +1

PreMeDICaL team, Inria, Idesp, Inserm, Université de Montpellier, Montpellier, France · Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189, CRIStAL, F-59000 Lille, France · Department of Computer Science, University of Copenhagen, Copenhagen, Danemark

Byzantine Attacks Learnability Distributionally-Robust Optimization Log-Ratio-Based Gaussian Trust Weight Poisoning

Jun 4, 2025cs.LG

Temporal horizons in forecasting: a performance-learnability trade-off

When training autoregressive models to forecast dynamical systems, a critical question arises: how far into the future should the model be trained to predict for optimal performance? In this work, we address this question by analyzing the relationship between the geometry of the loss landscape and the training time horizon. Using dynamical systems theory, we prove that loss minima for long horizons generalize well to short-term forecasts, whereas minima found on short horizons result in worse long-term predictions. However, we also prove that the loss landscape becomes rougher as the training horizon grows, making long-horizon training inherently challenging. We validate our theory through numerical experiments and discuss practical implications for selecting training horizons. Our results provide a principled foundation for hyperparameter optimization in autoregressive forecasting models.

Pau Vilimelis Aceituno, Jack William Miller, Noah Marti +2

Institute of Neuroinformatics, ETH Zürich and University of Zürich, Winterhurerstrasse 190, Zürich 8057, Switzerland · School of Computing, National Australian University, 108 North Rd, Acton ACT 2601, Australia · Unit of Land Change Science, Swiss Federal Research Institute for Forest, Snow and Landscape Zürcherstrasse 111, Birmensdorf 8903, Switzerland

Autoregressive Model Dynamical Systems Learnability Flat Minima

Feb 8, 2025cs.LG

Machine Unlearning via Information Theoretic Regularization

How can we effectively remove or ``unlearn'' undesirable information, such as specific features or the influence of individual data points, from a learning outcome while minimizing utility loss and ensuring rigorous guarantees? We introduce a unified mathematical framework based on information-theoretic regularization to address both data-point unlearning and feature unlearning. For data-point unlearning, we introduce the \emph{Marginal Unlearning Principle}, an auditable and provable framework. Moreover, we provide an information-theoretic unlearning definition based on the proposed principle and provable guarantees on sufficiency and necessity of marginal unlearning. We then show that the proposed framework provides a natural solution to the marginal unlearning problem and yields auditable high-probability marginal-unlearning guarantees. For feature unlearning, the framework applies to deep learning with flexible training objectives. By combining flexibility in learning objectives with simplicity in regularization design, our approach is highly adaptable and practical for a wide range of machine learning and AI applications. From a mathematical perspective, we provide a unified analytic solution to the optimal feature unlearning problem with a variety of information-theoretic training objectives. Our theoretical analysis reveals intriguing connections between machine unlearning, information theory, optimal transport, and extremal sigma algebras. Numerical simulations support our theoretical findings.

Shizhou Xu, Thomas Strohmer

Department of Mathematics University of California Davis Davis, CA 95616-5270, USA · Department of Mathematics Center of Data Science and Artificial Intelligence Research University of California Davis Davis, CA 95616-5270, USA

Machine Unlearning Exact Unlearning Learnability Kullback-Leibler Regularization