Classification

Follow this topic Ask about this topic

Momentum

45 papers in the last four weeks, up 137% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 254

Jun 27, 2026cs.LG

A Kernel Fisher Discriminant Analysis-Based Tree Ensemble Classifier: KFDA Forest

In general, an ensemble classifier is more accurate than a single classifier. In this study, we propose an ensemble classifier called the kernel Fisher discriminant analysis forest (KFDA Forest), which is a tree-based ensemble method that applies KFDA. To promote diversity, bootstrap is used, and variable sets are randomly divided into K subsets. KFDA is performed on each subset to increase classification accuracy. KFDA maximizes the distance between classes while minimizing the distance within classes. KFDA can also be applied to classification problems in a nonlinear data structure using the kernel trick because it can transform the input space into a kernel feature space, commonly named a rotation, rather than performing a dimensionality reduction. Because new feature axes and KFDA projections are parallel, decision trees are used as a base classifier. To compare the proposed method with existing ensemble methods, we apply these to real datasets from the UCI and KEEL repositories.

Donghwan Kim, Seung Hwan Park, Jun-Geol Baek

School of Industrial Management Engineering, Korea University Seoul, Korea · School of Mechanical Engineering, Chungnam National University Daejeon, Korea

Ensemble Learning Classification Kernel Methods Random Forest

Jun 27, 2026cs.DL

Categorizing Mathematical Concepts with LLM Voting Ensembles in Mathswitch

Mathswitch is an open-source project that imports mathematical concept records from sources such as Wikidata, Wikipedia, MathWorld, Encyclopedia of Mathematics, nLab, ProofWiki, and Agda-Unimath, and links records that refer to the same concept. It does not reorganize or redefine the imported content; each source retains its own structure. The current focus is on importing concept data from Wikidata and the resources it links to, with plans to expand to further sources and better concept linking. Because the concept set is approximated through queries over Wikidata's collaboratively edited graph, the imported data is noisy: some items are non-mathematical, while others are ambiguous. In this paper, we test whether a voting ensemble of LLM judges can filter this noise. We evaluate it on Wikidata items with known MathWorld identifiers as a positive control, and examine how classification changes when database identifiers are removed from context. We then inspect the cases where the judges disagree with MathWorld and group these disagreements into three categories (degenerate descriptions, narrow scope bias, and editorial-scope mismatches) that suggest different remediation strategies.

Katja Berčič, Slobodan Stanojevikj

Faculty of Mathematics and Physics, University of Ljubljana, Ljubljana, Slovenia · Institute of Mathematics, Physics and Mechanics, Ljubljana, Slovenia

LLM-as-a-Judge Classification Language Model Ensembles

Jun 26, 2026stat.ML

Surprises in Proper Positive-Only Learning

Binary classification from positive-only samples is a variant of PAC learning in which the learner receives i.i.d. samples from the positive region of an unknown target concept, but is evaluated under the original distribution (which places mass on both positive and negative regions). This model dates back to Natarajan [1987, STOC], and the characterization of improper learning is well-known -- it even appears in textbooks. The characterization of proper positive-only learning, however, has long remained open. In this work, we revisit and settle this question: a concept class is properly learnable from positive-only samples if and only if it has finite VC dimension and satisfies a new combinatorial condition, which we call uniform exterior separability. Together with several separation results, this characterization reveals a surprisingly rich landscape that differs sharply from standard PAC learning: proper and improper learning are separated, randomized and deterministic proper learning are separated, there are classes for which no ERM is a learner, and finite VC dimension does not suffice even for non-uniform learning. Along the way, we introduce new combinatorial dimensions that we believe can be of broader interest in learning theory.

Shai Ben-David, Farnam Mansouri, Anay Mehrotra +1

University of Waterloo and Vector Institute · Stanford University · Yale University

Binary Classification Classification PAC Learning Statistical Learning Theory

Jun 26, 2026cs.GT

Non-Linear Strategic Classification Made Practical

Algorithmic developments in Strategic Classification have been mostly limited to linear classifiers in settings where the best response has a closed-form solution or can be easily approximated. While some work has explored the role of non-linear classifiers in strategic settings, progress in this direction is impeded by the computational intractability of the strategic behaviour. Addressing this, we present a novel method for approximating the best response by exploiting Lagrangian duality. By reformulating the strategic response as a constrained optimisation problem, we can construct a Lagrangian that is amenable to first order optimisation methods. This approach reproduces closed-form strategic behaviour in linear settings and can be straight-forwardly applied to non-linear settings. We show how the Implicit Function Theorem can be used in conjunction with our proposed response formulation during classifier learning to compute the total gradient of the loss. This connects the classifier parameters directly to the consequent strategic behaviour, yielding a novel training algorithm that can exploit this relationship. Experimental evaluation shows that the resulting models achieve improved strategic accuracy on common machine learning datasets.

Jack Geary, Boyan Gao, Henry Gouk

School of Informatics, University of Edinburgh, Edinburgh, United Kingdom · Department of Engineering Science, University of Oxford, Oxford, United Kingdom

Strategic Classification Classification

Jun 22, 2026cs.LG

MGI: Member vs Generated Inference

As generative models increasingly produce samples that are indistinguishable from human-created content, it becomes difficult to determine whether a given data point was part of a model's natural training set or was generated by the model itself, especially when models memorize and reproduce training data. We formalize this challenge as Member vs Generated Inference (MGI): given a sample and a target generative model, infer whether the sample is a true training member or a generated output of that model. Focusing on image generation, we show that existing membership inference methods systematically misclassify generated samples as training members, while attribution-based methods often misclassify true members as generated. This failure arises because both approaches rely on likelihood-related signals that are similarly elevated for training examples and for the model's own outputs. To address MGI, we propose Data Circuit Breaker (DCB), a three-stage method that combines complementary signals from a generative model's autoencoder and latent generator to distinguish training members from generated samples. Across multiple generative models, including image autoregressive and diffusion models, DCB consistently addresses the shortcomings of membership inference and attribution methods, remains effective even when models reproduce near-duplicates of training samples, and generalizes to challenging model derivative settings in which new models are trained on generated data.

Bihe Zhao, Michel Meintz, Juangui Xu +2

CISPA Helmholtz Center for Information Security

Image Generation Memorization in Generative Models Classification

Jun 22, 2026cs.LG

Discovering Latent Groups for Robust Classification

Machine learning models exploit spurious correlations, achieving high average accuracy but failing disproportionately on underrepresented subgroups. Existing methods address this by adjusting network parameters, guided either by subgroup annotations or inferred pseudo-group labels. Yet at inference, these methods produce only a class prediction, with no insight into a sample's latent subgroup. We propose neural classification trees (NCT), a framework that achieves robustness by encoding subgroup structure in its tree-shaped architecture. By routing each sample to an "easy" or "hard" node of this tree -- based on prediction correctness -- and reusing these routes as pseudo-labels for the next iteration, NCT disentangles conflicting subgroups, without requiring subgroup supervision. We evaluate NCT on five benchmarks spanning binary and multi-class spurious correlations. Our experiments show that the learned tree topology provides strong interpretability by consistently isolating minority subgroups, which provides a transparent mapping between the model architecture and the data's latent group structure, while yielding competitive robustness with state-of-the-art methods.

Ankur Garg, Ulrich Aïvodji, Samira Ebrahimi Kahou +1

Interpretable ML Neural Network Robustness Classification Spurious Correlation Robustness

Jun 22, 2026cond-mat.dis-nn

Approximating velocity fields with planted attractors via Neural-ODEs for classification purposes

In this work, Neural ODEs equipped with a curated collection of equilibrium points have been successfully employed for classification tasks. The planted attractors serve as indicators for the target classes, while the velocity field leveraging the universal approximation capabilities of the architecture shapes the dynamical landscape. This process defines the basins of attraction of the trained model, effectively directing each input (provided as an initial condition) toward its corresponding destination target.

Feliciano Giuseppe Pacifico, Duccio Fanelli, Lorenzo Buffoni +3

Department of Informatics and Computer Science, University of Pisa, Italy · Department of Physics and Astronomy, University of Florence, Sesto Fiorentino, Italy · INFN, Italy

Dynamic Neural Networks Classification Neural ODEs

Jun 21, 2026cs.CR

Black-Box Forensics for Conversational LLM Agents

As LLM-powered scams proliferate, black-box forensics for conversational LLM agents offers a path to accountability for systems hidden behind anonymous endpoints. Identifying the base model behind a chatbot endpoint (attribution), without model parameter access or knowledge of the hidden system prompt, would let investigators trace AI-enabled scams back to the providers whose models power them. Detecting when two endpoints run the exact same system prompt (fingerprinting), even one novel and unseen, would link individual scams into criminal networks and expose silent API changes. We conduct an empirical investigation of both capabilities. Our attribution classifiers identify the base model behind an agent with 98% accuracy from a few turns of non-adversarial conversation. Attribution of system prompts, while possible, requires retraining on a large amount of data for each prompt; system prompts in the wild are unbounded and ever-changing, making this approach costly. To tackle this more open-ended setting, our cross-encoder fingerprinting method achieves an AUC of 0.768 and an F1 of 0.703 on entirely unseen system prompts, and aggregating 50 interaction conversations from each target agent boosts AUC to 0.943. Conversational agents with unseen system prompts can thus be fingerprinted with robust accuracy from a few turns of ordinary conversation.

Isadora White, Yasaman Jafari, Taylor Berg-Kirkpatrick

University of California, San Diego

Language Model Fingerprinting LLM Agent Security Classification Language Model Auditing

Jun 21, 2026cs.LG

QeHDC: Hyperdimensional Computing based on Quantum-enhanced binding and SuperClass Construction

Hyperdimensional Computing (HDC) is a robust computational framework inspired by human cognition characterized by simple and efficient operations within high-dimensional vector spaces. Quantum-enhanced Hyperdimensional Computing (QeHDC) extends classical HDC by leveraging quantum mechanical properties to enhance computational efficiency. In this paper, we propose a novel Quantum HDC framework featuring a one-pass training method, leveraging sinusoidal and quantum encoding to project classical data into quantum amplitude states efficiently. Our framework introduces an innovative reference-state-based quantum binding operation realized via quantum circuits. Furthermore, we propose a density-matrix-based superclass generation strategy employing eigenvalue decomposition to extract critical quantum state features effectively, enabling a more accurate and robust class representation. Experimental evaluations conducted on standard benchmark datasets demonstrate our approach's superior performance, robustness to noise, and computational feasibility compared to traditional classical and existing quantum-enhanced approaches. The results highlight the practical benefits and potential of Quantum HDC for quantum-enhanced classification tasks and pave the way for future advancements in quantum-inspired computational paradigms.

Yangjie Xu, Hui Huang, Li Ning +1

University of Luxembourg Kirchberg, Luxembourg · Stellaris AI Limited Hong Kong, China

Hybrid Quantum-Classical ML Hyperdimensional Computing Classification

Jun 19, 2026cs.LG

RocketPFN: Accurate Time Series Classification via In-Context Learning

We introduce RocketPFN, a training-free pipeline for time series classification that combines random convolutional feature extraction (Rocket) with in-context classification via a pretrained tabular foundation model (TabPFN v2.5). On 92 UCR datasets (30-resample protocol), RocketPFN matches HC2, the strongest published method on the archive, in mean accuracy (both 0.900, Wilcoxon p=0.50), with no training on the target data and a median inference time of 30 seconds per fold. It also significantly outperforms every individual classifier in the HC2 ensemble. On UEA (20 datasets) the difference is likewise not statistically significant. A separate comparison concerns TSC foundation models: when paired with the same downstream classifier, MOMENT, Mantis, and MantisV2 are all significantly outperformed by RocketPFN using fewer extracted features and no learned parameters (p<0.001 in each case). This holds even when the encoders were pretrained on corpora that include the UCR training samples. We propose this two-stage pipeline as a reference point for evaluating zero-shot TSC foundation models.

Franco Martino O'Rourke, Ana Trisovic, Dimitris Bertsimas

Time Series Classification Classification Time-Series Foundation Models

Jun 19, 2026cs.CL

Clinical Term Extraction using Open-Source Small Language Models

Clinical information for amyotrophic lateral sclerosis (ALS) care documented in unstructured clinical notes limits downstream analysis without extraction into structured formats. Open-source small language models with few-shot prompting for detecting the presence of ALS-relevant clinical terms in patient documentation were evaluated without task-specific training data. The detection task targeted 17 categories spanning functional scores, respiratory measures, medications, and related clinical and non-clinical attributes. Clinical note content was normalized from JSON-encoded discharge summaries and processed with a prompt template having structured JSON outputs. We compared 26 open-source models using aggregate, label-level, and manual-validation multilabel classification metrics. Manual validation showed that a regex rule baseline had higher overall micro-F1 and lower Hamming loss than any single SLM or TF-IDF baseline, while Qwen3-4B-Instruct-2507 was the highest-performing SLM by micro-F1. Model rankings varied by metric and label category, with the TF-IDF baseline showing high recall but low precision, some SLMs showing higher precision but lower recall, and Hammer2.1-7b showing strong performance for ALSFRS-R subscore detection. These findings support targeted hybrid extraction workflows rather than replacement of existing rule-based methods.

Noah Marchal, William E. Janes, Mihail Popescu +1

University of Missouri, Columbia, Missouri, USA · Department of Occupational Therapy; University of Missouri, Columbia, Missouri, USA

Small Language Models Clinical Information Extraction Classification Clinical NLP

Jun 18, 2026cs.CL

Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability

Pre-trained language models such as BERT achieve strong text classification performance but lack transparency, limiting their use in high-stakes settings. The Tsetlin Machine (TM) offers fully interpretable, clause-based reasoning but captures little semantic information, and prior attempts to bridge the two rely on static word embeddings that miss contextual meaning. We propose a semantic pre-training framework that transfers knowledge from a pre-trained language model into a TM without using embeddings. Text samples are grouped into semantically coherent clusters with K-means or Top2Vec, and the resulting cluster-sample pairs pre-train a non-negated TM with enhanced Type I feedback. The TM thereby learns interpretable semantic keywords that are fine-tuned on downstream tasks. Across five datasets, our method substantially outperforms vanilla and embedding-based TMs and reaches performance competitive with BERT while remaining interpretable.

Jiechao Gao, Rohan Kumar Yadav, Yuangang Li +4

Stanford University · Independent Researcher · University of California, Irvine +1

Interpretable ML Text Classification Classification

Jun 18, 2026cs.LG

Efficient Neural Network Model Selection for Few-Class Application Datasets

While much effort has focused on developing and benchmarking high-performance neural networks, less attention has been given to how dataset properties, known to practitioners, can guide efficient model selection. Neural models are typically evaluated on datasets with thousands of classes, yet many real-world applications involve fewer than ten. To address this understudied but common setting, we develop a measure of classification difficulty based on data-side properties and show how it enables more efficient model selection for few-class datasets, where traditional approaches are less effective. We term this phenomenon "few-class distinctiveness". Our metric allows comparison of models and datasets 6 to 29

\times

faster than repeated training and testing. Leveraging this insight, we extend scaled model families below the smallest published models, achieving greater efficiency at similar accuracy, for example models up to 42% smaller than YOLOv5-nano for a mobile robot task. Targeting resource-constrained applications, we demonstrate few-class model selection across mobile robot, drone, and IoT scenarios, highlighting practical gains in efficiency without sacrificing performance.

Bryan Bo Cao, Abhinav Sharma, Lawrence O'Gorman +2

Department of Computer Science Stony Brook University Stony Brook, NY 11790, USA · Nokia Bell Labs Murray Hill, NJ 07974, USA

Model Selection Classification Edge ML

Jun 17, 2026cs.LG

Zero-Shot Active Feature Acquisition via LLM-Elicitation

Active feature acquisition (AFA) sequentially selects which features to observe to reach a classification or ranking decision. Its central limitation is reliance on large amount of labeled data to fit probabilistic models guiding acquisition. Large language models (LLMs) supply unsupervised domain knowledge, but are poor sequential planners. Asking one to both know and decide conflates capabilities best kept separate. Here, we develop a framework for zero-shot AFA through disciplined elicitation: asking the LLM only for what it can be trusted to return, the unary deviations and pairwise co-variations that are the sufficient statistics of a Markov random field (MRF). We apply our framework to two settings: binary classification and top-

k

identification. In practice, the LLM reliably returns only discriminative statistics, what distinguishes the classes rather than each class in isolation, which precludes classical AFA. We apply a maximum-entropy closure that resolves this gauge ambiguity. We evaluate on a cohort of Inflammatory Bowel Disease (IBD) patients, an active clinical setting where diagnostic ambiguity and patient heterogeneity obstruct stable treatment strategies. Our framework outperforms the LLM both on real labels and on its own extracted beliefs. Where it matters most, on the hardest patients, our top-

k

acquisition policy markedly outperforms all existing methods.

Binyamin Perets, Natalie Mendelson, Shiran Vainberg +3

Faculty of Medicine, Technion. · NVIDIA

Zero-Shot Learning Markov Random Fields Active Learning Active Feature Acquisition Classification

Jun 16, 2026cs.LG

Meta-classification of one-class classification models using ranking correlation and nearest neighbor

Machine Learning (ML) techniques have been applied to various problems. However, applying ML to ML models is an unexplored direction. For this purpose, this paper considers a meta-classification of one-class classification (OCC) models, because all ML models could be approximated as OCC models. The proposal represents OCC models as normality rankings and classifies them using nearest-neighbor and ranking-correlation metrics. The experiment classifies OCC models, where classes correspond to training datasets, algorithms, and hyperparameters. The proposal achieves high accuracy when class labels are datasets. Moreover, it can classify algorithms when the training datasets contain the same class. In addition, the discussion highlights that the classification of OCC models is essentially the classification of datasets that treats multiple samples as a single input. The experiment demonstrates the classification of datasets using sleeping records. The proposed method can provide a unified solution for classifying OCC models, datasets, and rankings. Source code is uploaded to the public repository https://github.com/ToshiHayashi/ClassOCC.

Toshitaka Hayashi, Hamido Fujita, Dalibor Cimr +2

aFaculty of Science, University of Hradec Kralove, Hradecká 1285, Hradec Kralove, 50003, Czech Republic · bMalaysia-Japan International Institute of Technology (MJIIT), Universiti Teknologi Malaysia, Jalan Sultan Yahya Petra, Kampung Datuk Keramat, Kuala Lumpur, 54100, Malaysia · cRegional Research Center, Iwate Prefectural University, Sugo 152-52, Takizawa, 020-0693, Iwate, Japan

Classification One-Class Classification

Jun 15, 2026cs.LG

Neural Bayesian Anomaly Mitigation: A Robust Loss that Doubles as an Unsupervised Contamination Classifier

Engineered robust losses such as Huber, Student-

t

, and generalised cross-entropy make supervised models tolerant of contamination but cannot answer which observations are corrupted. We introduce Neural Bayesian Anomaly Mitigation (NBAM), a general-purpose drop-in loss derived from a Bayesian latent-switch mixture model: the marginal likelihood defines a robust supervised loss, and the associated posterior defines an unsupervised contamination classifier. Like Huber or Student-

t

, NBAM can replace the standard training loss in any supervised pipeline; unlike them, it additionally learns a structured contamination model and returns a calibrated per-sample contamination posterior. A learned input-dependent prior

π_φ(x)

captures the spatial locality of contamination, so that samples near known corruptions are more likely to be flagged, while an Occam penalty emerges automatically and regularises against over-flagging. On CIFAR-10 with asymmetric label contamination, NBAM recovers the structure of the corruption process without supervision: the contamination posterior separates clean from corrupted samples, and the learned anomaly head identifies the direction of every label-flip pair. Alongside these capabilities, NBAM outperforms the four robust-loss baselines considered here at contamination rates 0.2-0.6.

S. A. K. Leeney, W. J. Handley, H. T. J. Bevins +1

Astrophysics Group, Cavendish Laboratory University of Cambridge · Institute of Astronomy University of Cambridge

Neural Network Robustness Noisy-Label Learning Robust Loss Functions Corruption Robustness Unsupervised Anomaly Detection Classification+1

Jun 14, 2026cs.LG

Inference-Time Decision Calibration for Temporal Classification

Temporal classification errors are often treated as representation failures, but they can also arise from how available evidence is converted into decisions. This paper proposes a representation--calibration decomposition for temporal classification. We keep a trained native classifier frozen and separate two inference-time interventions: a conservative residual multi-scale branch that adds auxiliary logits to the native prediction, and a post-hoc branch-aware calibrator that recombines native and residual evidence at decision time. This design distinguishes missing temporal evidence from underused decision-level evidence without retraining the backbone. Across FI-2010, PTB-XL, UCI-HAR, MHEALTH, and HARTH, we find that gains are strongly regime-dependent. Residual multi-scale evidence is most useful in noisy or representation-limited settings, especially short-horizon FI-2010 and weaker recurrent backbones, while branch-aware calibration helps when native and auxiliary logits contain complementary evidence not fully exploited by the raw decision rule. Near-saturated settings show limited gains from either intervention. These results suggest that temporal classification should be understood not only as representation learning, but also as the problem of trusting, combining, and calibrating evidence from multiple views.

Arthur Chagas, Arthur Buzelin, Yan Aquino +4

Department of Computer Science (DCC), Universidade Federal de Minas Gerais (UFMG)

Post-Hoc Calibration Time Series Classification Classification Inference-Time Intervention

Jun 14, 2026cs.LG

Decomposing one-class support vector machine into an ensemble of one-data support vector machines

One-class classification (OCC) is a classification problem in which the training data contains only one class. The one-class support vector machine (OCSVM) is one of the most competitive OCC algorithms. However, OCSVM has scalability issues with large-scale datasets. This paper proposes the acceleration strategy of OCSVM. The idea is to decompose the dataset into samples and train OCSVM models for single data points. Subsequently, ensemble learning is applied to combine all models to compute the OCSVM model for the dataset. In addition, further acceleration is achieved through a data-reduction strategy with an OCSVM model trained on the average of the training samples. The experiment compared the proposal and traditional OCSVM using the Python package. The proposed strategy is faster than traditional OCSVM, while achieving similar classification results. Moreover, the proposed strategy can create one-to-one correspondence between samples and models. Source code is uploaded at https://github.com/ToshiHayashi/ODSVM

Toshitaka Hayashi, Dalibor Cimr, Hamido Fujita +1

Faculty of Science, University of Hradec Kralove, Hradecká 1285, Hradec Kralove, 50003, Czech Republic · Malaysia-Japan International Institute of Technology (MJIIT), Universiti Teknologi Malaysia, Jalan Sultan Yahya Petra, Kampung Datuk Keramat, Kuala Lumpur, 54100, Malaysia · Regional Research Center, Iwate Prefectural University, Sugo 152-52, Takizawa, 020-0693, Iwate, Japan

Support Vector Machines Classification One-Class Classification

Jun 14, 2026cs.CV

The Circumplex Degeneracy Behind the Rare-Class Limit in Affect Recognition

In-the-wild expression recognition persistently fails on a few rare emotions, and the standard explanation is class imbalance. Through a controlled multi-task study on two benchmarks, we show the failure is instead a property of affect geometry: the rare classes are degenerate on Russell's circumplex, and that degeneracy bounds what any loss or cost can achieve. Our instrument is a circumplex-cost optimal-transport term that prices expression confusions by their valence-arousal distance. The term improves the official score and expression macro-F1, but a control most studies omit shows the gain is not geometric: a uniform cost, equivalent to a generic confidence penalty, matches it on Aff-Wild2 (p=0.625) and significantly exceeds it on AffectNet (+0.057 over base, larger than the circumplex). What the geometry reshapes is the structure of the errors, making them affectively nearer the truth on Aff-Wild2 (p=0.031 against the uniform control), an effect that does not survive on AffectNet, where a visual confound at the far corner of the circumplex overwhelms it. The rare-class failure, by contrast, is stable across both datasets we examine: the degenerate pairs (anger-fear on Aff-Wild2, anger-contempt on AffectNet) resist frequency-based interventions, the transport term, and an action-unit-augmented cost built specifically to separate them. We conclude that progress on rare expressions requires representations that distinguish the classes, not supervision that reprices their confusions, and we provide the controls and metrics needed to tell the two apart.

Van Thong Huynh, Hong Hai Nguyen, Soo-Hyung Kim

Faculty of CSE, Ho Chi Minh City University of Technology (HCMUT), VNUHCM, Vietnam · Dept. of AI, FPT University, Vietnam · Dept. of AI Convergence, Chonnam National University, South Korea

Class-Imbalanced Learning Facial Expression Recognition Classification Affective Computing

Jun 13, 2026cs.LG

Transformers Learn the Mestre-Nagao Heuristic

We train a two-layer transformer encoder to classify rational elliptic curves

E/\mathbb{Q}

of conductor

\leq 10000

as either rank 0 or rank 1 from the first 128 normalized Frobenius traces. We achieve >99% accuracy on both classes, and accuracy is essentially unchanged on test curves with no isogeny or quadratic-twist relative in the training set. We then apply techniques from mechanistic interpretability such as attention analysis, linear probing, activation patching, logit attribution, and neuron-level circuit analysis to reverse-engineer the algorithm the (centroid in function space) model learned. We find that a sparse circuit of 20 out of 512 layer-1 MLP neurons is sufficient for rank prediction under a linear probe with an AUROC of 0.992 at plateau, implementing a push-pull detector architecture of rank-0 and rank-1 detectors with a one-sided readout. However, we notice that the model has sub-optimal readout problems indicating a mismatch in rank-order between the readout pathway and the discriminative circuit. Critically, the learned input weights of the top discriminating neuron match the Mestre-Nagao sum heuristic weights

\log(p)/(p\cdot \log{B})

with a Spearman coefficient

r = 0.997

and Pearson coefficient

r = 0.952

: the model has learnt a result from analytic number theory from the Frobenius trace data alone. We additionally find that all 50 independently trained models concentrate CLS attention on prime positions at 2-50

\times

the rate of composite positions. The CLS embedding encodes

\log{L(E,1)}

with

R^2 = 0.962\pm 0.011

across the 50 models (after controlling for the conductor). Activation patching analysis reveals that attention weights are dissociated from causal information flow. Additionally, the 50 solutions from training are near-identical in function space (with pairwise agreement

>

98.8%) despite large weight space barriers.

Pranav Venkata Konda

Transformer Interpretability Transformer Classification

Jun 12, 2026cs.LG

The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classification

Few-shot label acquisition lacks a label-free signal for when additional labels cease to improve accuracy: existing stopping criteria either require a held-out validation set (violating the few-shot premise) or rely on theoretically ungrounded heuristics, so we introduce the spectral saturation index

S(K)=\mathrm{erank}(\hatΣ_W^{(K)})/K

, the exponential spectral entropy of the pooled within-class covariance normalized by per-class support size

K

, which measures the exploration rate per label and falls below a fixed threshold

τ=0.02

once the explored spectral subspace saturates and marginal accuracy gains vanish; across 49 real tasks (binary, 5-way, 10-way) and three frozen backbones (PCA-50, CLIP ViT-B/32, DINOv2 ViT-S/14),

S(K)

correlates strongly with the marginal gain on doubling the support set (

ρ_{\text{pool}}=0.6366

,

p=2.9\times10^{-57}

, cluster-bootstrap 95% CI

[0.551,0.720]

), a fixed

τ=0.02

classifies stop/continue decisions with cluster-bootstrap

\mathrm{AUC}=0.787

(95% CI

[0.713,0.860]

) with high recall on meaningful gains (

ΔA>1\%

), and a partial correlation controlling for

\log K

yields

ρ_{\text{partial}}=0.324

(

p=1.65\times10^{-13}

), confirming

S(K)

carries spectral information beyond shared

K

-dependence; theory predicts this from first principles, since the population effective rank sets the saturation scale

K_{\text{sat}}\approx\mathrm{erank}(Σ_W)/τ

,

τ=0.02

sits at the boundary between the first and second descent (Nakkiran et al., 2021), and

O(1/K)

bias in the sample effective rank explains the small-

K

hump in

S(K)

; for unregularized linear probes (

C=\infty

), practitioners should halt when

S(K)<0.02

(PCA-50, hard stop) or monitor

S(K)

dropping from

\sim0.3\to0.05

(foundation models, diminishing-returns signal), with computation costing

\sim1

ms at

d=50

.

Arnav Gupta

Independent Researcher, Nepal

Early Stopping Few-Shot Learning Classification Statistical Learning Theory

Jun 12, 2026cs.CV

Rethinking Global Average Pooling: Your Classifier Is Secretly a Multi-Instance Learner

Modern image classifiers widely adopt global average pooling (GAP) followed by a linear classification head. This linearity ensures that the image-level logits equal the average of logits obtained by applying the classification head pointwise to the feature grid prior to GAP. Consequently, standard classifiers may inherently retain spatial class evidence that remains recoverable even when the image-level prediction is incorrect. This structure naturally suggests a multiple-instance learning (MIL) interpretation, where an image is viewed as a bag of spatial instances. Within this formulation, we demonstrate that standard classifiers trained with a single label per image can still learn the intended classification task in multi-object scenes. We further exploit this property to decompose image-level logits into a prediction grid, providing a post-hoc diagnostic to extract spatial class evidence that GAP otherwise obscures. Our systematic evaluation reveals that off-the-shelf models consistently recover the ground-truth class within foreground regions. The MIL interpretation further suggests that common classifier failures reflect known limitations of mean aggregation.

Aray Karjauv

indigo bunting

Feature Attribution Classification Multiple Instance Learning Image Classification

Jun 12, 2026stat.ML

A Bregman Perspective on Classification and Regression Trees

Classification and Regression Trees (CART) constitute one of the most influential paradigms in statistical learning. Although a variety of impurity measures have been proposed for different statistical models, these criteria are typically introduced on a case-by-case basis and analyzed separately. In this paper, we study CART through the lens of Bregman divergences. This perspective places the classical least-squares criterion, Poisson deviance, Kullback-Leibler-type losses, and other impurity measures associated with exponential-family models within a common framework. As a result, key ingredients of the CART methodology -- including node representatives, impurity measures, and split selection rules -- can be expressed and analyzed through general properties of convex functions rather than through separate model-specific constructions. Beyond the algorithmic formulation, we investigate theoretical properties of Bregman-based CART procedures. In particular, we analyze how geometric properties of the generating convex function influence impurity reductions and stability of recursive partitions. We also establish consistency results within the proposed framework, providing a unified theoretical treatment for a broad family of CART type procedures. Our results provide a geometric interpretation of impurity-based tree construction and show that many classical CART impurity criteria admit a common interpretation within a Bregman framework.

Mathias Bourel

IESTA, Facultad de Ciencias Económicas y de Administración, Universidad de la República, Uruguay · IRL-2030, Instituto Franco-Uruguayo de Matemática e Interacciones (IFUMI)

Functional Bregman Divergences Decision Tree Learning Classification Statistical Learning Theory

Jun 11, 2026stat.ML

Majority-of-Three is Optimal

We give a short proof that the majority vote of three independent consistent classifiers is an optimal learner in the realizable PAC setting. This proves optimality for the simplest voting scheme, while simplifying both the algorithmic structure and the probabilistic analysis of previous voting learners, including the algorithm of S. Hanneke and the analysis of bagging by K. Green Larsen.

Divit Rawal, Nikita Zhivotovskiy

Department of Statistics, University of California, Berkeley.

Ensemble Learning Classification PAC Learning Majority Voting

Jun 11, 2026cs.LG

Decoding Insect Song: A Multitask Semisupervised Orthoptera Bioacoustic Classifier

Passive acoustic monitoring holds great promise for ecological inference, yet existing automated tools are typically narrowly trained and non-transferable. We address these limitations with PULSE, a semi-supervised, multi-task framework for Orthoptera bioacoustics, combining weakly-supervised species classification, self-supervised learning on unlabelled field audio, and knowledge distillation from a general-purpose bioacoustic model. Our domain-adapted specialist model outperforms a state-of-the-art general model across all metrics (macro F1: 0.21 vs. 0.07; AUC: 0.74 vs. 0.45; AP: 0.32 vs. 0.19), with active learning further raising F1 to 0.34 and AUC to 0.84. Beyond classification, the learned embeddings encode ecologically meaningful structure, exposed through an interactive visualisation tool for ecological discovery.

Olga Isupova, Danil Kuzin, Ella Browning +2

Leverhulme Centre for Nature Recovery, School of Geography and the Environment, University of Oxford, Oxford, UK

Audio Representation Learning Audio Understanding Classification Bioacoustics Passive Acoustic Monitoring Semi-Supervised Learning+2

Jun 11, 2026cs.LG

Distributional Loss for Robust Classification

This paper proposes a novel loss concept for supervised classification tasks. Rather than enforcing a direct mapping from each input sample to a single assigned label, we define an optimization objective over all classifier outputs as a bimodal Gaussian distribution. This softer target formulation implicitly captures class ambiguity, mitigates overfitting, and encourages the learning of more robust decision boundaries, all without requiring additional label information. Experimental results demonstrate consistent improvements in robustness, with particularly pronounced gains in low-data regimes, while requiring only minimal modifications to standard training pipelines.

Kathleen Anderson, Thomas Martinetz

Institute for Neuro- and Bioinformatics, University of Lübeck, Germany

Robust Loss Functions Classification

Jun 9, 2026cs.LG

Learning from almost nothing: How neural networks survive heavy input corruption

Learning from imperfect data is a central theme in machine learning, connecting practical questions of robustness to fundamental questions of learnability. Here we examine attribute noise: learning from corrupted inputs while keeping the labels intact, a setting that has received considerably less analytical attention than its label-noise counterpart. We consider two types of corruption models: additive noise and replacement noise. Through experiments with multi-layer perceptrons (MLPs) on corrupted classification datasets, we find that neural networks remain robust, maintaining well-above-chance accuracy even when inputs are >90% corrupted -- far beyond human recognition. To understand this robustness, we analyze infinite-width networks in the heavy-corruption regime using a mean-field-inspired approach and derive a leading-order decision rule for the classification outcome: the network implements a prototype rule, the nearest-class-mean, assigning each test point to the class whose training-set average it most closely resembles. This leading-order decision rule is universal across a broad range of MLP architectures, holding for any depth, as well as a wide class of activation functions and noise distributions. The same centroid mechanism closely matches finite-width network behavior in our experiments and provides an interpretable and analytically tractable account of why learning can succeed even when individual training examples carry almost no signal.

Justin Tahmassebpur, Asadullah Bhuiyan, Hyejin Kim +1

Department of Applied and Engineering Physics, Cornell University, Ithaca, NY 14853, USA · Department of Physics, Cornell University, Ithaca, NY 14853, USA

Neural Network Generalization Multilayer Perceptrons Prototype-Based Classification Neural Network Robustness Corruption Robustness Classification+1

Jun 9, 2026stat.ML

Near-Exponential Convergence Rates for kNN Classification based on Boltzmann Margin

Convergence-rate analysis for classifiers is often conducted under either Tsybakov margin or Massart margin. The former is a relatively weak condition that typically yields polynomial rates, while the latter is substantially stronger but can guarantee exponential rates. In this paper, we introduce a new condition, called Boltzmann margin, that bridges the gap between these two regimes. It is weaker than Massart margin, generally stronger than Tsybakov margin, and can imply many of their properties under suitable conditions. We apply Boltzmann margin to the analysis of kNN classifiers and establish the first near-exponential convergence rates for kNN classification. We also present extensions of the main results and provide numerical evidence supporting the main theoretical implications.

Luyuan Yang, Shayan Shafaei, Chao Lan

School of Computer Science, University of Oklahoma, Norman, Oklahoma, USA

Classification k-Nearest Neighbors Statistical Learning Theory

Jun 8, 2026cs.LG

Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic

While neural collapse (NC) predicts that a

K

-class-balanced classifier should organize terminal representations as a

(K-1)

-dimensional simplex equiangular tight frame (ETF), modular addition consistently enters a different regime: networks compress to a two-dimensional cyclic geometry in which both classifier weights and token embeddings lie on circles. We refine the explanation of this phenomenon in three directions. First, we formalize a layerwise non-uniform training mechanism: downstream classifier weights are driven by dense cross-entropy gradients into a rank-2 equiangular configuration before upstream embeddings fully reorganize, and once this classifier plane forms, backpropagated feature gradients constrain embedding motion to the same plane while weight decay suppresses orthogonal components. Second, after this subspace locking, the induced in-plane dynamics admit an entropy-regularized transport interpretation on

S^1

; combined with modular-addition labels, this reduces embedding formation to phase alignment, whose minimizers are single-frequency characters of

\mathbb{Z}/P\mathbb{Z}

and hence equal-angle points on a circle. Third, we quantify why this solution prevails over NC: a simplex ETF gains only an

O(1)

advantage in cross-entropy, whereas the cyclic rank-2 solution enjoys a

Θ(K)

advantage under Schatten or weight-decay surrogates, yielding a critical threshold

λ_{\mathrm{crit}} = Θ(1/K)

. Our results explain both why classifier weights move first and why embeddings subsequently align with them, showing that grokking on modular arithmetic is governed not by maximal separation alone but by a task-structured trade-off between separation, symmetry, and complexity.

Hu Tan, Kuo Gai, Shihua Zhang

Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; and School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China · Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China; School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China; and Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China

Neural Representation Geometry Classification Neural Collapse Neural Network Training Dynamics Modular Arithmetic Grokking

Jun 5, 2026cs.CL

Style or Content? Evaluating Style Classifiers with Controlled Content Overlap

Style classifiers can use content cues that correlate with style labels in naturally collected data, yet we lack a systematic way to measure this reliance. We study this problem with a controlled content overlap setup built on parallel Bible translations. Specifically, we define the overlap parameter

α

as the normalized residual of mutual information between content identity and style label, so that it measures how much content is shared across style classes: from no shared content (

α=0

) to fully shared content (

α=1

). Cross-overlap evaluation of RoBERTa-based classifiers shows that low-overlap models degrade when content cues are removed, while high-overlap models transfer more robustly. A cross-style content retrieval probe further shows that content becomes less recoverable as

α

increases, with training dynamics showing this removal occurs gradually. Together, these results suggest that controlled overlap provides a simple diagnostic for separating style learning from content shortcuts.

Zhuo Liu, Haozheng Du, Xiangxiang Xu +1

University of Rochester

Text Classification Classification Shortcut Learning