Equivariant Learning

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 53

Sep 30, 2026eess.SP

PI-AMFM: Permutation-Invariant Learning for Variable-Cardinality AM-FM Mode Decomposition in Biomedical Signal Analysis

Physiological recordings often contain nonstationary oscillatory components whose number and dynamics vary across signals. Amplitude- and frequency-modulated (AM-FM) representations are well suited to characterizing such dynamics and have shown broad utility in biomedical signal analysis. Recent approaches have incorporated neural networks to learn mode decomposition patterns from data, but component cardinality is often predefined or determined through separate stopping or selection mechanisms. We propose a permutation-invariant neural framework for variable-cardinality AM-FM mode decomposition (PI-AMFM). PI-AMFM combines a multiscale temporal encoder, Mamba backbone, and component-presence estimation, with permutation-invariant Hungarian matching during training. On synthetic AM-FM signals, PI-AMFM achieved lower decomposition, instantaneous-frequency, reconstruction, and mode-count errors than the compared methods while preserving the overall trajectory pattern in a crossing-chirp example. On photoplethysmographic recordings, recovered modes captured cardiac and respiratory dynamics despite training only on synthetic signals. These results support the feasibility of PI-AMFM for variable-cardinality decomposition of nonstationary biomedical signals.

Youngsun Kong, Ki H. Chon

Biomedical Engineering, University of Connecticut, Storrs, CT 06029, USA

Dynamic Mode Decomposition Physiological Signals Time-Frequency Frequency Neural Recordings Equivariant Learning+1

Sep 30, 2026cs.LG

WinoTS: Wavelet-based Self-Distillation for Time Series Models

Self-supervised pre-training of time series models is currently dominated by next-token prediction and reconstruction objectives. In continuous-valued domains, these paradigms often waste model capacity on high-frequency, point-wise noise at the expense of learning invariant structure. While invariance-based self-distillation has proven highly effective in computer vision, its application to temporal data remains largely underexplored. Effectively adapting such methods to time series requires carefully designed augmentations: spatial operations like cropping can shift the timing of repeating cycles or distort the signal, while basic jittering may provide limited variation. We introduce Wavelet-based self-distillation for time series (WinoTS), an invariance-based pre-training paradigm designed specifically for temporal signals. At its core, WinoTS leverages time-frequency augmentations to construct multi-scale structural views without distorting underlying signal dynamics. Across extensive evaluations, WinoTS outperforms state-of-the-art baselines in long-term forecasting, cross-domain zero-shot transfer, and unsupervised anomaly detection. Notably, linear probing on frozen WinoTS representations frequently surpasses fully supervised models trained from scratch. Systematic ablations demonstrate that WinoTS is a flexible, architecture-agnostic framework yielding gains across time series backbones, and establish that time-frequency transformations provide a principled alternative to vision-style spatial augmentations.

Noam Major, Kathy Razmadze, Yoli Shavit

Bar-Ilan University, Ramat-Gan, Israel

Time Series Wavelets Equivariant Learning Data Augmentation

Sep 29, 2026cs.LG

Longer Records, Broader Invariance: The Hidden Scaling Problem in Longitudinal Contrastive Learning

Longitudinal data are valuable because people change. Yet the objectives used to learn from these data can inadvertently erase that change. In person-level contrastive learning, observations from the same person are treated as positives; as records grow, those positives can span increasingly distant---and increasingly different---behavioral states. More history can therefore produce not only more data, but broader invariance. We show that this distinction is fundamental. We separate \emph{record span}, how much history the learner sees, from \emph{supervision span}, how far across that history positive-pair supervision reaches. Across in-home sensing records spanning up to 2.7 years, broader supervision systematically suppresses recoverable changing-state information, even when the available history is held fixed. At the broadest span, less than 10% of the information recoverable from an untrained encoder remains. Yet keeping positives local is not sufficient: as records grow, even distant states that are never paired become increasingly similar. Explicitly contrasting other observations from the same person reverses this loss without shortening the record, revealing a second route by which longitudinal scale can broaden invariance. Finally, we prospectively reproduce the supervision-span effect in 199 GLOBEM participants. Longitudinal scale therefore presents a choice: more history need not mean more invariance. By controlling what is held invariant as records grow, we can preserve the change that made the longitudinal data valuable in the first place.

Rameen Mahmood, Xuhai "Orson" Xu, Zachary Beattie +2

New York University · Columbia University · Oregon Health & Science University

Longitudinal Data Contrastive Learning Equivariant Learning Sequential Scaling

Sep 28, 2026cs.CV

Transform-Aligned Learned Features for Lossy Point Cloud Attribute Compression

Transform-based methods provide an effective framework for point cloud attribute compression by representing attributes as transform coefficients. Introducing learned spatial context into this framework requires mapping spatial representations to the transform domain, but this known basis change is often left for the network to learn implicitly. We propose Transform-Aligned Learned Features (TALF) by applying the attribute transform to learned spatial representations, explicitly aligning them with the coding targets. Our analysis shows that the resulting features exactly represent the first-order prediction term of a smooth nonlinear model, with a bounded Taylor remainder. We integrate TALF into a transform-based attribute codec with explicit coefficient prediction and conditional residual entropy modeling under a unified coefficient-domain rate--distortion objective, while retaining explicit quantization-step control. Extensive experiments across three benchmark datasets and multiple transform bases demonstrate that TALF improves rate--distortion performance over conventional and learned baselines.

Yueru Chen, Pengpeng Yu, Dingquan Li +3

Pengcheng Laboratory, Shenzhen, China · Sun Yat-sen University, Shenzhen, China · Peking University, Shenzhen, China +1

Point Clouds Rate-Distortion Theory Data Compression Methods Equivariant Learning

Sep 21, 2026cs.LG

PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatorics

Diverse pretraining has been shown to be an effective method for learning reusable, domain-aware representations that provide a starting point for fine-tuning on downstream tasks. While much of the excitement in AI for math has been concentrated in the use of frontier reasoning models to solve well-specified problems through the medium of language, narrow, specialized models remain an important component of the AI for math ecosystem. In contrast to large language models, specialized models are usually trained directly on the mathematical objects themselves (e.g., graphs, sequences of numbers) rather than the textual descriptions that characterize these objects. However, the common practice of training specialists from scratch may limit their ability to develop domain-aware representations that capture the multifaceted nature of mathematics. In this paper, we describe an approach to pretraining for permutation-focused tasks in algebraic combinatorics. We introduce PermuFormer, an autoregressive transformer trained on a 2.8 billion token multi-task, multi-encoding corpus. We show that PermuFormer is an effective starting point for fine-tuning on basic tasks unseen during pretraining and more complex research-level tasks, frequently outperforming the same architecture trained from scratch, baseline MLPs, and a fine-tuned generic language model of comparable size. We also analyze some of the internal mechanisms by which PermuFormer learns to solve training tasks. For example, we show that while some tasks can be linearly decoded directly from the internal representation of the prompt, other tasks require multiple rounds of generation before the answer can be decoded.

Henry Kvinge

Pacific Northwest National Laboratory · University of Washington

Pretraining Permutation Math Problems Combinations Equivariant Learning Domain Invariance+2

Sep 14, 2026cs.LG

$\mathbb{SL}(n)$ Representation Learning: An Intrinsic Mixed-Curvature Space with Higher Curvature Capacities and Deeper Order-Aware Composition

Mixed-curvature representation learning seeks to capture rich geometric structures that cannot be adequately modeled by a single curvature regime. Existing approaches largely rely on product manifolds, which require manually specifying how different curvature spaces are combined and separate their curvature contributions across factors. We introduce the

\mathbb{SL}(n)

space, a representation geometry defined by the simple

\det(A)=1

constraint and a left invariant Schatten-

p

Finsler structure. Despite this minimal construction,

\mathbb{SL}(n)

exhibits pointwise negative, zero, and positive flag curvature around a common flagpole, while its mixed-curvature and curvature-coupling capacities are asymptotically maximal relative to the intrinsic geometric upper bound. Beyond geometry, its noncommutative group structure provides inherent order sensitivity, and its non-nilpotent Lie algebra admits nonzero nested Lie brackets at arbitrary depth, enabling deep order-aware composition. Empirically,

\mathbb{SL}(n)

consistently outperforms a broad range of representation manifold baselines across graph benchmarks at different scales. It reduces average distortion over the strongest baselines by

44.3\%

on KEGG and

40.5\%

on HumanCyc, and improves Hits@20 by

42.8\%

on OGBL-PPA. Experiments on Flickr30k-Order further support its ability to capture higher order dependencies from ordered composition. Together, these results show how a seemingly simple structural constraint can yield unexpectedly rich geometry, capacity, and composition within a unified representation space.

Xingrun Li, Yusuke Mukuta, Xin Yang +2

The University of Tokyo · Stanford University

Representation Space Hyperbolic Learning Equivariant Learning Local Curvature Riemannian Manifolds Lie Group

Sep 9, 2026cs.LG

Hierarchical and Permutation-Invariant Feature Transformation Learning via Policy-Guided Embedding Search

Feature transformation improves predictive performance on tabular data by constructing informative abstractions from raw features. Recent generative approaches encode transformation knowledge into continuous embedding spaces for efficient exploration of candidate strategies, but face three key limitations: (1) overlooking hierarchical relationships between low-level features, operations, and high-level abstractions; (2) enforcing order-sensitive embeddings on inherently permutation-invariant transformation sequences, thereby introducing systematic bias; and (3) relying on gradient-based search, which is ill-suited to non-convex transformation spaces. We propose a framework with two complementary components. First, a permutation-invariant hierarchical module captures interactions across features, operations, and abstraction levels, with a self-attention pooling mechanism that maps semantically equivalent structures to consistent embeddings aligned with downstream performance. Second, a policy-guided multi-objective reinforcement learning strategy initializes the search from empirically strong seeds and jointly optimizes predictive accuracy and transformation efficiency. Extensive experiments on diverse tabular benchmarks demonstrate the effectiveness and robustness of our framework against strong baselines. Our code and data are publicly available at: https://github.com/RayLiu1103/PHER.

Rui Liu, Tao Zhe, Yanyong Huang +5

University of Kansas Lawrence, United States · Southwest University of Finance and Economics Chengdu, China · University of Wisconsin–Madison Madison, United States +3

Tabular Learning Equivariant Learning

Aug 14, 2026cs.AI

Attributing Preprocessing Invariance in Spectral Foundation Models

A spectral foundation model should remain useful when laboratories preprocess spectra differently. The standard test trains a classifier under one pipeline and evaluates under another, taking preserved accuracy as evidence of learned invariance. However, these models normalize each input before any learned parameter is applied. When normalization maps differently preprocessed spectra to the same vector, the encoder receives identical inputs and the measured invariance cannot be attributed to learning. We propose a normalization-only attribution control: compare the encoder against its normalization before interpreting transfer as learned invariance. On six Raman datasets the encoder does not measurably improve transfer over its normalization. A controlled experiment confirms invariance develops only when variation reaches the encoder past normalization. Across three systems, no encoder improves relative retention over its normalization. An audit of eighteen configurations across five modalities confirms the issue is widespread: the normalization-only control should be reported before crediting transfer to the encoder.

Dongjun Wei, Hongyi Wu

ESCP Business School, Paris · OpluxCare Tech Group Ltd. · The University of Hong Kong, Hong Kong

Equivariant Learning Raman Wireless Foundation Models Hyperspectral Image Encoders

Aug 13, 2026cond-mat.stat-mech

Equivariant learning of a transferable three-dimensional classical density functional

Liquids exhibit collective behavior that depends sensitively on thermodynamic conditions, interfaces and confinement, yet predicting each new state commonly requires a separate atomistic simulation. Classical density functional theory offers a reusable variational description, but its central excess free-energy functional is generally unknown, and learned approximations have largely remained restricted to planar or lower-dimensional settings. Here we show that this functional can be learned directly from fully three-dimensional equilibrium density fields while preserving spatial symmetry and variational consistency, without free-energy or chemical-potential labels. A single learned functional transfers across temperatures, system sizes and statistical ensembles, and recovers structure factors, the equation of state, liquid--vapor coexistence and interfacial broadening, none of which are used as training targets. Applied to complex three-dimensional geometries, it predicts the non-monotonic force associated with formation and rupture of a solvent-depleted bridge between colloids and adsorption in an interconnected gyroid pore. These results demonstrate that equilibrium density data can be converted into a transferable thermodynamic generator connecting microscopic liquid structure to response, phase behavior and collective phenomena.

Bingqing Cheng

Department of Chemistry, UC Berkeley, California 94720, United States · Chemical Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, California, 94720, United States · Bakar Institute of Digital Materials for the Planet, UC Berkeley, California 94720, United States

Density Functional Theory Statistical Physics Free Energy Principle Equivariant Learning Collective Behaviors

Aug 13, 2026cs.CV

Dual-Manifold Geometry Guided Representation Learning: Adaptive Coupling between Kernel and Data Spaces

Deep representation learning has primarily focused on how features evolve across network layers, while largely overlooking the structured geometry embedded in network parameters. We introduce a dual-manifold perspective in which each convolutional layer contains two coupled geometric spaces: a Kernel Manifold induced by convolutional filters and a Data Manifold characterized by intermediate feature representations. Because these manifolds share the same channel space, parameter geometry can provide complementary structural information to guide feature evolution. Based on this insight, we propose Kernel-Guided Feature Transform (KGFT), a lightweight module that derives a geometric guidance matrix from the kernel Gram matrix and uses it to transform the covariance structure of feature representations. Unlike conventional attention mechanisms that reweight feature responses, KGFT explicitly reshapes feature relationships by transferring geometric information from the kernel manifold to the data manifold. To accommodate network hierarchy, we further introduce Exploit and Explore modes with a depth-aware scheduling strategy and a learnable guidance strength that adaptively controls the contribution of geometric transformation. This design promotes geometric alignment in shallow layers while encouraging feature diversity in deeper layers, without imposing excessive constraints on representation learning. Theoretical analysis establishes the validity of the proposed transformation and characterizes its effect on feature covariance. Extensive experiments across CNN- and Transformer-based architectures, including ResNet, ViT, and LLaMA-7B, demonstrate consistent improvements on image classification and arithmetic reasoning tasks, validating the generality and effectiveness of kernel-guided dual-manifold representation learning. Code will be publicly available.

Wencong Zhang, Yue Zhang, Meiyan Huang +2

School of Biomedical Engineering, Southern Medical University, China, Guangzhou 510515

Representation Learning Equivariant Learning Convolutional Covariance

Aug 11, 2026stat.ML

Iterative Erasure Count Is Not an Affine-Invariant Concept Dimension

How many directions does a neural representation use to encode a concept? A common answer repeatedly erases probe directions and reports the stopping count or cumulative removed rank. We show that both quantities can change under an information-preserving invertible reparameterization, so neither is intrinsically a concept dimension. We distinguish model-defined population quantities (generating dimension, sufficient linear dimension, and minimum guarding rank) from procedure-defined quantities such as stopping count and cumulative edit rank. In a population Gaussian construction, an invertible shear preserves the prediction problem and all three quantities, yet changes the cumulative Euclidean erasure count from one to two. The separation holds for Moore--Penrose ordinary least squares and every finite nonnegative ridge weight. For a two-output full-QR procedure matching our motivating video analysis, cumulative edit rank similarly changes from two to the ambient dimension four. Conversely, the complete cumulative metric-QR trajectory is affine-equivariant when its positive-definite metric, probe, regularizer, and tie-breaking are transported consistently; exact covariance is one corollary, not a canonical semantic metric. In a known-rank finite-sample Adam/QR calibration, identity mixing stops after one accepted update in all 20 large-sample runs, whereas each tested shear

a\in\{.5,.75,1,1.25,2\}

accepts at least two updates in all 20 runs. Controlled reparameterizations of frozen V-JEPA2 features preserve rank-zero predictions yet alter later Euclidean trajectories under practical optimization. These visual contact experiments are stress tests, not estimates of contact dimension. Iterative erasure therefore returns a procedure-relative estimand jointly determined by representation geometry and the full measurement procedure, not a semantic dimension by itself.

Tingan Jin, Shuhang Dong, Haosong Li +1

UCLA · Independent Researcher · Cal Poly Pomona

Intrinsic Dimensionality Concept Erasure Neural Network Representations Equivariant Learning Reparameterization Covariance+2

Aug 9, 2026cs.CV

UPolarSQ: Polar Representation Learning for Optic Disc and Peripapillary Atrophy Segmentation and Quantification in Fundus Photographs

Myopia-induced posterior-pole remodeling is frequently accompanied by Optic Disc (OD) deformation and Peripapillary Atrophy (PPA), both of which provide clinically relevant structural biomarkers. In Cartesian fundus images, however, PPA often appears as an irregular and partially visible crescent adjacent to the OD, leading to fragmented segmentation and post-processing-dependent quantification. We propose UPolarSQ, a unified polar-domain framework for OD/PPA segmentation and biomarker quantification in myopic fundus images. UPolarSQ first maps an OD-centered region of interest into polar coordinates, where OD and PPA boundaries can be represented as radial profiles. It then employs UPolarSeg, a U-Net-based segmentation network enhanced with a Radial-Angular-Decoupled Module and boundary-aware auxiliary supervision to model anisotropic polar features and radial boundary transitions. Clinical biomarkers, including disc shape and PPA-width-related measurements, are deterministically extracted from the predicted polar masks, aligning segmentation and quantification within a shared geometric representation. Experiments on internal and external cohorts demonstrate that UPolarSQ improves OD/PPA segmentation and supports reliable polar-native biomarker estimation for myopic analysis.

Mengxian He, Yunyun sun, Ziyue Gao +3

Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · Capital Medical University, Beijing, China

Fundus Images Vessel Segmentation Biomarker Equivariant Learning

Aug 6, 2026cs.CV

Support Operation Factorization: Compositional Readout of Frozen Vision Encoders under Controlled Interventions

Compositional analysis of frozen vision encoders should determine both what changed and where it changed. Standard factor probes score these axes separately, however, and can reward multiple operations that reuse the same predicted slot. We call this failure operation laundering. We introduce an injectively aligned leave-one-cell-out protocol over support x operation grids and SO-OPF, a readout that factors cell energy into support salience and a competitive operation posterior. This formulation separates two questions that aggregate scores conflate: whether the carrier composes held-out bindings when the grid is known, and whether that grid can be recovered from flat cell labels. With frozen DINOv3 features, known factorial assignment reaches 0.874 injective accuracy on Shapes3D-Extended and 0.799 on globally image-disjoint COCO; learning the assignment from flat labels reaches 0.769 and 0.762, respectively. Under matched-axis-aware supervision on Shapes3D, the factored carrier improves learned-assignment accuracy from 0.653 to 0.841 over a dense carrier and eliminates its laundering gap. SigLIP2 replicates the COCO separation. A rebuilt MuJoCo substrate exposes a boundary: learned-assignment accuracy is 0.569 with DINOv3 and 0.484 with SigLIP2, with substantial slot collapse. Thus factored readout and injective evaluation recover held-out bindings on two substrates while exposing, rather than hiding, a renderer-specific failure boundary; they do not establish universal recovery from flat labels.

Zhongyao Wang, Wanli Ouyang, Taoyong Cui +1

Fudan University · MMLab, The Chinese University of Hong Kong · Department of Computer Science and Engineering, The Chinese University of Hong Kong

Readout Frozen Encoder Dinov3 Equivariant Learning Encoders

Jul 20, 2026math.ST

Unveiling Invariant and Transferable Latent Factors Across Heterogeneous Environments via ATLAS

This paper considers a multi-environment factor model in which high-dimensional covariates are collected from heterogeneous environments, with auxiliary labels available in a subset of these environments. The joint distribution of the covariates may vary across environments, whereas the latent structure is decomposed into invariant factors with shared loadings and heterogeneous factors with environment-specific loadings. Such a model is motivated by transfer learning and latent factor regression, where one seeks stable low-dimensional representations for both interpretation and robust out-of-sample prediction of the response

Y

. Leveraging the invariance principle, we show that the invariant and heterogeneous factors are disentangled under a minimal structural condition. Based on this, we propose ATLAS, an Auxiliary-label and invariance-guided Transfer via Latent Alignment across heterogeneous environmentS. ATLAS is a unified procedure that leverages the invariance principle to separate aligned invariant and unaligned heterogeneous factors, and further exploits supervision from auxiliary labels to extract prediction-invariant and transferable factors from those unaligned heterogeneous factors. ATLAS yields near-oracle performance for downstream latent factor regression, enables transferable prediction in new environments through the full latent signal when auxiliary labels are available, and reduces to robust invariant-factor-only prediction otherwise. We establish sharp non-asymptotic error bounds for recovering invariant and heterogeneous factors, identifying all the response-invariant factors, and estimating the invariant signal in

Y

.

Yihong Gu, Katherine Liao, Tianxi Cai

Harvard University

Exploratory Factor Analysis Latent Variable Equivariant Learning Heterogeneity Factor Atlas

Jul 17, 2026cs.LG

(MPO) $^2$ : Multivariate Polynomial Optimization based on Matrix Product Operators

Central to machine learning and signal processing is the ability to perform universal function approximation and learn complex input-output relationships from limited numbers of observations. Multivariate polynomial models offer a natural way to express such relationships through multiplicative feature interactions, but their coefficient tensors grow exponentially in size with the polynomial degree. Existing tensorized polynomial models reduce this cost, yet canonical polyadic decompositions have rank-limited expressivity, and tensor train formulations are feature order dependent. We introduce Multivariate Polynomial Optimization based on Matrix Product Operators (MPO)

^2

, a framework that combines learned MPO feature embeddings with compact polynomial weight tensors. This yields feature order independent polynomial representations that can incorporate structured operators such as projections, convolutions, and masks for weight tensor symmetries. Across regression and classification benchmarks, (MPO)

^2

improves over existing tensor decomposition based polynomial models and provides a flexible alternative for efficient polynomial function approximation.

Niccolò Ciolli, Anders Vestergaard Nørskov, Michael Kastoryano +2

Tensor Decomposition Matrix Multiplication Equivariant Learning Multilayer Perceptrons

Jul 8, 2026math.ST

Any-Dimensional Learning by Sampling

Many machine learning models are defined for inputs of different sizes, such as point clouds containing different numbers of points, sequences of tokens of different lengths, and graphs on different numbers of nodes. Such models are trained on finitely many examples of necessarily limited sizes. How well do these models generalize from inputs of small size to larger inputs of size not seen during training? Furthermore, evaluating such models on large inputs is often expensive. How can we sketch large inputs to obtain smaller ones on which the model takes similar values? At the heart of both questions is the need to compare inputs of different sizes and to approximate large inputs by small ones. We present a unified approach to address these questions by using random sampling maps to compare inputs of different sizes. The sampling maps we consider are generalizations of sampling with replacement, random binning, and species sampling. We characterize the application domains in which each type of sampling is appropriate in terms of the symmetries and relations between problem instances of different sizes in the domain. Our framework yields explicit generalization and sketching rates for function classes continuous with respect to a chosen notion of sampling, encompassing large families of functions defined on sequences, graphs, and tensors of different sizes. Specific examples include moment polynomials on measures, homomorphism densities and numbers of graphs, permutation-invariant transformers, and graph neural networks.

Eitan Levin, Venkat Chandrasekaran

Department of Statistics University of Chicago Chicago, IL 60637 · Department of Computing and Mathematical Sciences Department of Electrical Engineering California Institute of Technology Pasadena, CA 91125

Optimal Sample Complexity Model Size Equivariant Learning Generalisation

Jul 2, 2026cs.CV

Domain Generalization via Text-Anchored Information Bottleneck

Visual recognition models often fail when deployed in new environments. Domain Generalization (DG) addresses this by learning representations that remain invariant to environment-specific variations. Recent approaches increasingly rely on large vision-language models, assuming that preserving their expressive visual representations improves robustness. However, we show that such visual expressiveness can instead propagate spurious cues that tie representations to the training environments, hindering invariant learning. We therefore discard visual guidance and instead treat the language embedding space as the primary source of domain invariance, naturally acting as an information bottleneck that preserves core semantics while suppressing domain-specific variations. Extensive experiments across diverse backbones exhibit state-of-the-art performance and further analyze what makes guidance effective for robust generalization. These findings shift the focus of DG from improving representations to designing supervision that enforces invariance.

Eunyi Lyou, Yunjeong Choi, Junho Lee +1

Seoul National University, Seoul, Republic of Korea

Multimodal Domain Generalization Domain Invariance Visual Representations Information Bottleneck Equivariant Learning

Jul 1, 2026cs.RO

BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfer

Sim2real transfer for robot policy learning suffers due to mismatch between simulation and reality. Existing methods typically address each gap in isolation through separate adaptation modules, which are composed or layered when both gaps coexist. Yet the basis for attempting sim2real in the first place is that there is shared structure between a task in simulation and reality, where equivalent actions from equivalent configurations produce equivalent long term outcomes regardless of domain specific differences in rendering or physics. In this paper, we study whether we can identify and exploit this shared structure from raw observations to train a policy that enables zero shot transfer. We introduce BIFROST, which learns a shared history encoder on paired cross-domain data via cross-domain bisimulation objective: observation-action sequences leading to equivalent long-term behavior are mapped to nearby latent states, regardless of domain. Policies trained on these latent states in simulation transfer zero-shot to reality. We provide empirical evidence on sim2sim visual navigation and sim2real contact rich manipulation task and visual servoing task that BIFROST achieves effective transfer where domain adaptation and co-training baselines fail under both visual and dynamics domain gaps.

Yunfu Deng, Josiah P. Hanna

Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI 53706 USA

Sim-To-Real Reinforcement Learning Sim-To-Real Gap Robot Policies Equivariant Learning Similarity and Metrics Scale Gap

Jun 23, 2026cs.LG

Data Augmentation: A Fourier Analysis Perspective

Data augmentation is a simple and model-agnostic approach for exploiting known invariances in learning problems. Given a group acting on the input space, one augments the training set with transformed copies of each sample. Because it exploits symmetries without modifying the underlying learning algorithm, data augmentation can be applied broadly across learning methods. However, this universality comes at a computational cost: when the group is large, full group-sized augmentation quickly becomes computationally infeasible. This raises a fundamental question: Can partial data augmentation achieve the same statistical benefits as full augmentation in terms of generalization and sample complexity? We develop a general framework for investigating this question using Fourier analysis and the representation theory of finite groups. We show that, for a broad class of classical learning problems, partial data augmentation based on a randomly sampled subset of group elements achieves the same minimax rates as full augmentation, up to an approximation error that vanishes as the subset size increases. Our results provide a theoretical explanation for why partial augmentation can retain the statistical benefits of full augmentation despite enforcing symmetry only approximately, and shed light on a recently raised question in learning with symmetries: whether statistically optimal learning under general group invariances can be achieved using computationally scalable methods. Moreover, we prove a complementary impossibility result: enforcing exact invariance via data augmentation requires averaging over the entire group, and cannot be achieved by any strict subset when the hypothesis space is sufficiently expressive. Together, these results provide a unified perspective on full and partial data augmentation, as well as exact and approximate symmetry enforcement.

Behrooz Tahmasebi, Melanie Weber, Stefanie Jegelka

Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA. · Technical University of Munich (CIT, MCML, MDSI) · MIT Computer Science and Artificial Intelligence Laboratory (CSAIL)

Data Augmentation Equivariant Learning Fourier Optimal Learner Perspective

Jun 11, 2026cs.LG

Is Spurious Correlation Removal Always Learnable?

Invariant learning can fail even when the invariant structure is statistically identifiable. We show a conditional computational barrier: under a black-box samplable supervised sparse recovery primitive motivated by average-case sparse-recovery reductions, there exist \emph{samplable} multi-environment instances with a one-dimensional predictive invariant subspace (

k=1

) that are learnable with polynomial samples by exhaustive search, while any polynomial-time constant-accuracy recovery algorithm would contradict the primitive. We further quantify environment diversity by a separation parameter

γ

, which controls identifiability and the curvature of invariance objectives. Under sufficient diversity and local Gaussian regularity, the minimax risk is

\mathbb{E}[\dist(\hat{V},V_{\mathrm{inv}})^2]=Θ(k(d-k)/(n|\mathcal{E}|))

, and under label-induced shifts a phase transition occurs at

n^*\propto k(d-k)/(|\mathcal{E}|γ^2)

with refined estimation error scaling proportional to

1/γ^2

. Synthetic and real datasets illustrate the predicted gaps and transitions and motivate simple diversity diagnostics.

Yibo Zhou, Bo Li, Hai-Miao Hu +3

Beijing Key Laboratory of Digital Media, School of Computer Science and Engineering, Beihang University, Beijing 100191, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China · Hangzhou Innovation Institute of Beihang University, Hangzhou 310051, China +1

Equivariant Learning Spurious Correlations Subspace Diversity Barriers

Jun 10, 2026cs.CV

Vision-Encoder Behavioral Fingerprints of Image-to-Image Generative Models: A Training-Paradigm-Driven Taxonomy of Six Commercial APIs

We study six production image-to-image AI systems (gpt-image-1, Gemini 2.5 Flash Image, Flux Kontext, SDXL img2img, SD3 img2img, and Qwen Image Edit) under a content-adaptive sub-JND adversarial perturbation pipeline, scoring all outputs by frozen DINOv2 ViT-B/14 token distances against clean references. Across a 3,588-call corpus spanning COCO photographs, CelebA-HQ portraits, and AI-generated inputs, the six systems partition into two image-invariant behavioral bands on a 2D (patch_mean, ssim_clean) plane: edit-trained models (Flux Kontext, Qwen Edit, Gemini) cluster in a tight band, while T2I-base models adapted at sampling time (SDXL, SD3, gpt-image-1) cluster in a drift band.

Hunter Hill

Fingerprint Equivariant Learning Application Programming Interfaces Taxonomy

Jun 6, 2026cs.CL

Cross Paraphrastic Invariance Learning for Hallucination Detection

Large language models (LLMs) frequently generate hallucinations, which are unsupported by a source document. To avoid costly LLM-as-evaluator pipelines and the heavy annotation demands of existing classifiers, we propose CPIL (Cross Paraphrastic Invariance Learning), a two-stage Siamese framework that maximizes the utility of existing labeled data. Concretely, CPIL constructs informative training pairs by: (i) generating paraphrastic views of each document-claim example as positives, and explicitly aligning their representations to enforce invariance to surface form; and (ii) mining same-document, opposite-label pairs as hard negatives to sharpen document-sensitive decision boundaries. Then CPIL conduct a two-stage model training: Stage 1 performs contrastive pretraining to learn a paraphrase-invariant, grounding-aware embedding space; and Stage 2 attaches a lightweight classifier for binary groundedness. On the LLM-AggreFact benchmark (11 tasks), CPIL surpasses strong baselines concerning F1 scores with only ~1% labeled data, showing its prediction superiority and label efficiency.

Shanshan Lin, Dongsheng Hong, Sibo Ju +3

1Fuzhou University · 2Harbin Institute of Technology (Shenzhen) · 3The Hong Kong University of Science and Technology (Guangzhou)

Large Language Model Hallucination Hallucination Detection Equivariant Learning

Jun 5, 2026cs.CV

Varifold Moment Invariants for Sustainable and Explainable Contour Feature Extraction

We introduce Varifold Moments Invariants (VMI) as a unifying framework for many previously introduced Moment Invariants. These invariants are deeply related to other contour features that are invariant under translations and rotations, like Extended Gaussian Image, Elliptic Fourier Descriptors or Shape Distributions. The advantage of the varifold approach to moments consists in being able to combine the geometry of the region, its boundary, and the family of lines tangent to it, in order to create a substantial number of invariant features with high discriminating power and clear geometric meaning. By coupling our VMI feature extraction with the light feature classifiers Random Forest or Multi-Layer-Perceptron, we outperform state-of-the-art approaches based on contours, while decreasing drastically the computational cost to the point of allowing our algorithm to run on light devices. We tested our approach on classification tasks on a large number of widely-used datasets of various types (leaves, objects, cells) and achieved high accuracy with a low number of geometrically interpretable features.

G. Longari, J. -C. Alvarez Paiva, A. B. Tumpach

Contours Feature Extraction Equivariant Learning Random Forest Fourier

Jun 5, 2026stat.ML

The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces

The transformer's emergent ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its underlying mechanisms. Existing works often study how training task diversity, defined either as the number of ICL training task vectors or as the number of function classes from which the task vectors are drawn, shapes both the learning dynamics and generalization capabilities of ICL. While both definitions have uncovered many interesting phenomena, many observations under the latter definition remain theoretically unexplained. This paper presents a minimal analytical model under which these phenomena provably emerge from the properties of the training data. By modeling the training task vectors as a mixture of low-rank Gaussians, we show how training task diversity, defined by the number of non-overlapping columns between subspaces that parameterize the covariance matrices, improves both the generalization and optimization trajectory of ICL with linear attention. In particular, we show that our model can explain (i) why training with task diversity shortens the ICL plateau and (ii) why ICL appears to achieve out-of-distribution generalization. We conclude by empirically demonstrating how our results extend to nonlinear transformers and nonlinear function classes. Overall, our work presents a tractable framework to unify existing observations.

Soo Min Kwon, Alec S. Xu, Can Yaras +3

Department of Electrical and Computer Engineering, University of Michigan · Department of Statistics, University of California, Davis

In-Context Learning Equivariant Learning Subspace Out-Of-Distribution Energy-Based Transformer Impact Analysis

Jun 4, 2026q-bio.QM

$p$ -adic Bi-Filtrations for Topological Machine Learning on Genomic Sequences

We introduce pVR, a topological machine learning framework for alignment-free genomic sequence classification that combines

p

-adic numbers with topological data analysis. Each DNA sequence is encoded along two complementary axes: a

p

-adic distance on

k

-mer prefixes, which captures hierarchical positional structure, and a compositional

L_1

distance on

k

-mer frequencies, which captures local sequence content. The two distances jointly parameterise a bi-filtered Vietoris--Rips complex, and per-sequence topological summaries from this bi-filtration serve as features for standard machine learning classifiers. We establish theoretical guarantees for the construction: stability under metric perturbations and invariance to the choice of prime, alongside a result that explains why a single

p

-adic axis is topologically uninformative and why the bi-filtration recovers nontrivial homology. On twelve genomic benchmarks (

28

to

500

sequences,

3

to

7

classes), pVR outperforms four established alignment-free baselines on three of six low-sample datasets, with gains of up to

21

percentage points; it underperforms only on a SARS-CoV-2 variant benchmark whose point-mutation divergence violates the hierarchical assumption, and all methods saturate in the large-sample regime. pVR also outperforms zero-shot frozen embeddings from the 500M-parameter Nucleotide Transformer v2 by

6.7

to

11.4

percentage points on three low-sample benchmarks. The pVR codebase is publicly available at https://github.com/MAHI-Group/pVR.

Tirtharaj Dash, Gunja Sachdeva

Department of CS & IS, BITS Pilani, K K Birla Goa Campus, Zuarinagar, Goa 403726, India. · Department of Mathematics, BITS Pilani, K K Birla Goa Campus, Zuarinagar, Goa 403726, India.

Genome-Wide Association Study Topological Learning Principal Component Analysis Topology Feature Alignment Equivariant Learning

Jun 1, 2026cs.CV

Disentanglement-Based Equivariant Learning for Compositional VQA

Compositional visual question answering (VQA) represents a challenging yet fundamental task that requires models to comprehend novel combinations of previously learned concepts. The current methods often overlook the disentanglement of underlying concepts and are restricted in terms of their ability to effectively capture the compositional variation mechanism. Moreover, the state-of-the-art techniques depend on additional clues for training, which is not feasible in real-world VQA scenarios. To address these issues, in this paper, we introduce a novel Disentanglement-based EquivAriant Learning (DEAL) framework for compositional VQA, which is guided exclusively by ground-truth answers. In DEAL, we employ causality-inspired interventions to disentangle concepts derived from visual and textual inputs within a re-encoding framework. Based on the principle of equivariance, we subsequently perform a compositional transformation on the inference input and impose the equivariant constraint on the output to augment the compositional reasoning capacity of the model. Comprehensive experiments conducted on the benchmark CLEVR-CoGenT and GQA-SGL datasets validate the superiority of our proposed DEAL approach over the existing state-of-the-art methods for compositional VQA tasks in both visual and linguistic generalization settings.

Zhou Du, Zhaoquan Yuan, Xiao Wu +1

School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611730, China, and also with Engineering Research Center of Sustainable Urban Intelligence Transportation, Ministry of Education, China · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, and with the School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China, and also with Peng Cheng Laboratory, ShenZhen 518055, China

Knowledge-Based Visual Question Answering Equivariant Learning

May 31, 2026cs.LG

LeAP: Learnable Adaptive Permutation for Feature Selection in Heterogeneous and Sparse Recommender Systems

Modern industrial recommender systems rely on thousands of heterogeneous features -- ranging from low-dimensional scalars (e.g., statistical value) to high-dimensional embeddings (e.g., user-id embeddings, MLP representations) -- to achieve high-precision predictions. Given the immense computational costs associated with training, efficient feature selection is critical. However, existing methods encounter three primary bottlenecks: (1) they typically assume uniform feature dimensions or require costly mapping to a fixed size; (2) they struggle with extreme sparsity, where the majority of features (e.g., 99%+) remain at default values; and (3) traditional permutation-based approaches are computationally prohibitive in large-scale settings. To address these challenges, we propose LeAP (Learnable Adaptive Permutation), a novel, model-agnostic plug-in module for feature selection. LeAP transforms the inefficient random permutation process into a learnable mechanism, significantly accelerating the evaluation of feature importance. In addition, we introduce an adaptive regularization strategy tailored for heterogeneous dimensions and extreme sparsity, enabling superior feature importance ranking results across asymmetric input spaces. Experiments on four public recommendation datasets demonstrate that LeAP achieves state-of-the-art performance. Furthermore, LeAP has been deployed in a large-scale industrial search ranking model with over a billion daily requests and a 2TB model parameter scale. In this real-world scenario involving 12,000+ total feature dimensions, LeAP successfully identified and removed over 3,600 redundant dimensions without performance degradation, which is 2 to 10 times the ability of compared baseline methods.

Yihong Huang, Chen Chu, Fei Chen +3

Bilibili Inc., Shanghai, China

Feature Selection Real-World Content Recommendation Problem Feature Importance Equivariant Learning Heterogeneity

May 30, 2026cs.LG

Learning Transfers: Kan Extensions for Neural Invariants

Transfer learning presumes that a representation learned on source tasks carries structure that remains usable on related target tasks. Standard evaluations probe this through target accuracy or distributional discrepancy, yet leave unspecified which structural invariant is meant to transfer. We supply that invariant categorically. A source task category

\mathcal A

, a target task category

\mathcal B

, and a task-change functor

J:\mathcal A\to\mathcal B

determine, for every invariant-valued source representation

F:\mathcal A\to\mathcal V

, the universal transferred invariant

\operatorname{Lan}J F

. Given a target invariant

G:\mathcal B\to\mathcal V

, we define the transfer discrepancy

\operatorname{Comp}J(F,G)=\sup{b\in\operatorname{Ob}(\mathcal B)} d{\mathcal V}\bigl((\operatorname{Lan}_J F)(b),G(b)\bigr)

, evaluating transfer not by an objectwise comparison of source and target, but by comparing the target invariant against the one forced by the prescribed task transformation. We prove finite cokernel formulas for

(\operatorname{Lan}_J F)(b)

in chain complexes and persistence modules, indexed by the comma category

J\downarrow b

. For persistence-valued finite-type one-parameter invariants, the discrepancy is computed exactly by bottleneck distances between barcodes. Controlled experiments on neural latent point clouds then test whether the score recovers the correct task functor and flags representation collapses that preserve classification accuracy while destroying transfer-relevant topology.

Luciano Melodia

Friedrich-Alexander Universität Erlangen-Nürnberg

Equivariant Learning Transfer Learning Topology Extended Version

May 30, 2026cs.CV

Eyes All Around: Design and Analysis of 360-Degree LiDAR Perception Using Equivariant Feature Learning in Unstructured Traffic

Perception in dense, unstructured urban traffic remains a major challenge for autonomous driving because of the wide variety of road users, frequent occlusions, irregular motion patterns, and the lack of standardized road layouts. Although recent LiDAR based 3D object detectors have shown strong performance in structured driving scenarios, most are developed and evaluated for limited field of view settings, and their behavior under full surround 360-degree sensing is still not well understood. This paper studies a 360-degree LiDAR perception pipeline for autonomous driving, with particular attention to panoramic sensing, azimuthal sector wise spatial processing, and transformation equivariant feature extraction in complex urban scenes. The paper presents a practical 360-degree perception framework that combines sector wise panoramic processing with rotation equivariant sparse convolutions and evaluates its behavior on a custom Ouster OS0 LiDAR dataset collected across diverse Indian urban traffic conditions. The results show generally stable detection across several object classes, with the strongest performance for cars at 92.02/90.51, buses at 80.53/76.34, and trucks at 78.59/74.16, while lower scores for pedestrians at 67.45/61.02, cyclists at 73.21/69.54, and motorcyclists at 71.20/68.13 reflect the greater difficulty of detecting smaller and more variable road users in dense urban scenes.

Pranav Darshan, Raghuveer Narayanan Rajesh, M Uttara Kumari

Department of Computer Science and Engineering RV College of Engineering, Bengaluru 560059, India · Department of Electronics and Communication Engineering RV College of Engineering, Bengaluru 560059, India

Autonomous Driving Perception 3D Object Detection Autonomous Driving Equivariant Learning

May 29, 2026cs.LG

Learning Permutation-invariant Macroscopic Dynamics

Accurately modeling the macroscopic dynamics of high-dimensional microscopic systems is of broad interest across the sciences. Many data-driven approaches learn a low-dimensional latent state through an autoencoder trained for pointwise input reconstruction. These methods typically assume a fixed ordering of microscopic degrees of freedom in the input. However, in many settings, such as particle systems, the microscopic state is inherently unordered. This motivates an autoencoder framework that learns permutation-invariant latent representations. To this end, we adopt a permutation-invariant encoder and design the decoder to reconstruct the mass distribution centered at the observed points rather than per-sample reconstruction. We then jointly learn the macroscopic dynamics of the observables together with the latent states. We demonstrate the effectiveness and robustness of the proposed method across a range of microscopic settings, including learning the energy dynamics in interacting particle systems, predicting mixing dynamics in Lennard-Jones fluids, and modeling the stretching dynamics from video data of polymers moving in an elongational force field.

Zhichao Han, Mengyi Chen, Qianxiao Li

Institute for Functional Intelligent Materials, National University of Singapore · Department of Mathematics, National University of Singapore

Particle Dynamics Latent Dynamics Equivariant Learning Autoencoder Architectures Intermediate Latent States

May 29, 2026cs.CV

Equivariant Latent Alignment via Flow Matching under Group Symmetries

Geometry-aware generative models and novel view synthesis approaches have shown strong potential in visual fidelity and consistency. In parallel, equivariant representation learning has emerged as a powerful framework for constructing latent spaces where analytically known group transformations could act directly, capturing geometric structure in data and enhancing both interpretability and generalization in novel view synthesis. However, we identify that existing approaches often suffer from latent misalignment, a discrepancy between the intended group action and the actually required transformations in the latent space. Consequently, the learned latents often fail to consistently preserve the equivariant relations imposed by the underlying group symmetry. To address this, we propose Residual Latent Flow, a flow-based framework that corrects the misaligned latents, thereby improving compliance with the underlying equivariance relation. Our comprehensive experiments show that our method significantly reduces latent misalignment and improves novel view synthesis quality, under rotation groups SO(n).

Sunghyun Kim, Jaehoon Hahm, Jeongwoo Shin +1

1Seoul National University, Seoul, Korea · University of Illinois Urbana-Champaign, Illinois, USA

Discriminative Congruence Transform Latent Flow Latent Representation Alignment Novel View Synthesis Equivariant Learning Geometry-Aware+2

May 28, 2026cs.CL

Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

Positional encoding (PE) underpins how permutation-invariant Transformers represent sequence order, yet how positional information is processed and stored remains poorly understood. Modern PE methods such as RoPE still struggle on tasks such as long-context understanding or retrieval \cite{chen-etal-2025-hope}. Hence, a better understanding of the internal positional mechanism could help design better PE. Building on evidence that positional and semantic signals occupy nearly orthogonal subspaces in trained Transformers, we modify an encoder Transformer to process three explicitly disentangled streams: semantic, absolute positional (AP) and relative positional (RP), and confine the masked-language-modeling (MLM) objective to the semantic stream. This decoupling enables a clean mechanistic study and yields three take-aways. (1) The isolated AP subspace spontaneously collapses into a low-frequency two-dimensional manifold that captures the structure of the document; (2) Attention heads specialize into structure and semantic-oriented groups, with RP exclusively supporting the latter; (3) Standard positional encodings do not robustly retain macroscopic structure: RoPE and RP only weakly encode it, and entangled AP loses it in the final layers under MLM pressure. The disentangled approach preserves positional encoding, which improves linguistic representation on 49 of the 65 linguistic phenomena of the Flash-Holmes probing benchmark.

Pierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski

Sorbonne Université, CNRS, ISIR, Paris, France · Orange Innovation

Positional Encoding Transformer Encoder Encoders Semantic Representations Equivariant Learning Long-Context Reasoning

May 26, 2026cond-mat.soft

On the Equivariant Learning of the $Q$ -tensor Order Parameter

We construct and evaluate group-equivariant neural networks for the prediction of the two-dimensional

Q

-tensor order parameter of nematic liquid crystals from synthetically generated microscopic textures. Seven architectures, equivariant to cyclic groups

C_k

of order

k

for

k=4,\,8,\,16,\,32,\,64,\,128,\, 256

, are built using a combination of weight-sharing constraints, equivariant activations and regularization techniques. To do this, we construct rotation-like permutation matrix groups with elements

\varrho_{C_k}(g)

that act on row-wise vectorized images, thereby approximating a

\frac{2π}{k}

rotation of the circular subdomain on square images. We show that all seven equivariant models satisfy the

Q

-tensor equivariance constraint to within single-precision floating point accuracy. Comparing against approximate parameter-matched non-equivariant benchmarks, with and without data augmentation, we find that the equivariant models consistently achieve lower errors and generalize more robustly to unseen defect configurations. Performance increases with group order, suggesting that the incorporation of finer rotational symmetry leads to lower errors.

Julia Navarro, Mark Wilkinson

Nottingham Trent University, UK · Berea College, USA

Equivariant Neural Networks Equivariant Learning Order Texture

May 25, 2026cs.LG

Learning Permutation from Structure Without Supervision

Many learning problems require uncovering a hidden ordering that reveals structure in unordered data, such as monotonicity in sorting or spatial continuity in jigsaw reconstruction. In these settings, permutations can be learned as latent operators by optimizing objectives defined directly on the reordered output, often without access to ground-truth orderings. Differentiable relaxations such as Gumbel-Sinkhorn make this approach practical by approximating permutation matrices with doubly stochastic matrices. However, learning from structure without supervision induces a non-uniform uncertainty: some assignments become confident early, while others remain ambiguous. Existing methods control this process using a single global temperature, forcing all assignments to sharpen or diffuse simultaneously and leading to instability at scale. We introduce an entropy-adaptive formulation of Gumbel-Sinkhorn that locally modulates temperature based on assignment uncertainty. This allows confident assignments to discretize early while preserving exploration where uncertainty remains. Across sorting and jigsaw reconstruction tasks and in routing-style settings, adaptive entropy control improves training stability and final permutation quality relative to fixed-temperature baselines, particularly as problem size and assignment ambiguity increase.

Ran Eisenberg, Ofir Lindenbaum

Faculty of Engineering, Bar-Ilan University, Ramat Gan, Israel. · Faculty of Engineering, Bar-Ilan

Permutation Equivariant Learning Puzzle Uncertainty

May 24, 2026cs.LG

Factorize to Generalize: Retrieval-Guided Invariant-Dynamic Decomposition for Time Series Forecasting

Time series foundation models (TSFMs) have recently achieved strong zero-shot forecasting performance through large-scale pretraining and retrieval-augmented prediction. However, our empirical analysis reveals a non-trivial limitation of retrieval-based forecasting: retrieval tends to induce more oscillatory predictions, improving performance on highly fluctuating series while degrading accuracy on smoother, trend-dominated ones. This suggests that retrieved information may be fused into prediction without explicitly distinguishing stable temporal structure from instance-specific variations, which can reduce robustness under distribution shifts. We propose a Retrieval-guided Invariant-Dynamic DEcomposition framework for time series forecasting. Rather than using retrieval as auxiliary predictive context, we leverage retrieved sequences as implicit samples from related environments to guide representation decomposition. Specifically, we first construct a retrieval-aware representation via attention-based aggregation, and then introduce a retrieval-guided routing mechanism to decompose it into an invariant component capturing stable shared structure and a dynamic component modeling context-dependent variations. These two components are forecast separately and fused for final prediction, enabling the model to preserve transferable patterns while remaining adaptive to evolving dynamics. We further design training objectives that encourage invariant learning and disentanglement, and provide theoretical insight showing that retrieval aggregation reduces variance and approximates invariant representation learning without explicit environment supervision. Extensive experiments demonstrate that our method consistently improves robustness under distribution shifts and outperforms existing TSFMs and retrieval-based baselines in zero-shot forecasting settings.

Jinjin Chi, Lei Feng, Lulu Zhang +6

College of Computer Science and Technology, Jilin University, Changchun, China · College of Computing and Data Science, Nanyang Technological University, Singapore · City St George’s, University of London, London, United Kingdom +1

Zero-Shot Forecasting Time Series Forecasting Equivariant Learning Decomposition Factor

May 20, 2026cs.LG

$\textit{BlockFormer}$ : Transformer-based inference from genomic contact maps

Locating genomic features from genomic contact maps, such as centromere identification from genome-wide chromosome conformation capture techniques, notably Hi-C, can be formulated as an inverse problem: infer one parameter per entity given a map summarizing pairwise interactions through blocks of variable numbers and sizes. In this work, we introduce a data-driven approach that leverages shared structure between these contact maps, such as global alignment between localized patterns, while handling the variability in number and size of chromosomes arising in real-world data. Our approach relies on a transformer architecture capable of handling such variability and a custom simulator to generate abundant, yet computationally cheap synthetic data for training. Applied to the problem of centromere localization, the method recovers genomic positions at or below the map resolution across species with various genome sizes using a coarse-to-fine refinement step when chromosome sizes fall outside the training range. We concentrate our evaluation on Hi-C data, where the block structure is well characterized and ground truth is available.

Eloïse Touron, Pedro L. C. Rodrigues, Julyan Arbel +2

Univ. Grenoble Alpes, Inria CNRS, Grenoble INP, LJK, France · TIMC, Univ. Grenoble Alpes CNRS, Grenoble INP, France

Genome-Wide Association Study Flow Map Equivariant Learning Transformer Architectures Interaction Data

May 16, 2026cs.LG

Learning Unbiased Permutations via Flow Matching

Learning permutations is fundamental to sorting, ranking, and matching, but existing differentiable methods based on entropy-regularized Sinkhorn produce a single softened solution and collapse under ambiguity. We present PermFlow, a conditional flow matching framework that operates directly on the affine subspace of matrices with unit row and column sums. A closed-form tangent-space projector preserves these constraints exactly along every trajectory, by construction rather than through iterative correction, and a nearest-target coupling routes distinct noisy initializations toward distinct valid permutations. The result is a model that captures multimodal permutation distributions rather than collapsing them to a single mode. On a visual sorting task with blended-digit ambiguity and a symmetric linear assignment problem, PermFlow achieves high accuracy on unambiguous inputs and recovers both valid permutations under ambiguity, where Sinkhorn-based baselines structurally fail.

Yimeng Min, Carla P. Gomes

Department of Computer Science Cornell University Ithaca, NY, USA

Permutation Conditional Flow Matching Fused Sinkhorn-Localized Similarity Equivariant Learning Repair-Assignment Problem

May 12, 2026cs.LG

Stop Marginalizing My Dreams: Model Inversion via Laplace Kernel for Continual Learning

Data-free continual learning (DFCIL) relies on model inversion to synthesize pseudo-samples and mitigate catastrophic forgetting. However, existing inversion methods are fundamentally limited by a simplifying assumption: they model feature distributions using diagonal covariance, effectively ignoring correlations that define the geometry of learned representations. As a result, synthesized samples often lack fidelity, limiting knowledge retention. In this work, we show that modeling feature dependencies is a key ingredient for effective DFCIL. We introduce REMIX, a structured covariance modeling framework that enables scalable full-covariance modeling without the prohibitive cost of dense matrix inversion and log-determinant computation. By leveraging a Laplace kernel parameterization, REMIX captures structured feature dependencies using memory that scales linearly with the feature dimensionality, while requiring only an additional logarithmic factor in computation. Modeling these correlations produces more coherent synthetic samples and consistently improves performance across standard DFCIL benchmarks. Our results demonstrate that moving beyond diagonal assumptions is essential for effective and scalable data-free continual learning. Our code is available at https://github. com/pkrukowski1/REMIX-Model-Inversion-via-Laplace-Kernel.

Patryk Krukowski, Jacek Tabor, Przemysław Spurek +2

Jagiellonian University · IDEAS Research Institute

Replay-Based Continual Learning Continual Learning Equivariant Learning Covariance Inverse Problem Dream

May 11, 2026cs.LG

FeatMap: Understanding image manipulation in the feature space and its implications for feature space geometry

Intermediate feature representations represent the backbone for the expressivity and adaptability of deep neural networks. However, their geometric structure remains poorly understood. In this submission, we provide indirect insights into this matter by applying a broad selection of manipulations in input space, ranging from geometric and photometric transformations to local masking and semantic manipulations using generative image editing models, and assess the feasibility of learning a mapping in the feature space, mapping from the original to the manipulated feature map. To this end, we devise different types of mappings, from linear to non-linear and local to global mappings and assess both the reconstruction quality of the mapping as well as the semantic content of the mapped representations. We demonstrate the feasibility of learning such mappings for all considered transformations. While global (transformer) models that operate on the full feature map often achieve best results, we show that the same can be achieved with a shared linear model operating on a single feature vector typically with very little degradation in reconstruction quality, even for highly non-trivial semantic manipulations. We analyze the corresponding mappings across different feature layers and characterize them according to dominance of weight vs. bias and the effective rank of the linear transformations. These results provide hints for the hypothesis that the feature space is to a first degree of approximation organized in linear structures. From a broader perspective, the study demonstrates that generative image editing models might open the door to a deeper understanding of the feature space through input manipulation.

Elias B. Krey, Nils Neukirch, Nils Strodthoff

Division AI4Health Carl von Ossietzky Universität Oldenburg Oldenburg, Germany

Feature Learning Equivariant Learning Class Activation Mapping Image Editing Implications

May 11, 2026cs.CV

Birds of a Feather Flock Together: Background-Invariant Representations via Linear Structure in VLMs

Vision-language models (VLMs), such as CLIP and SigLIP 2, are widely used for image classification, yet their vision encoders remain vulnerable to systematic biases that undermine robustness. In particular, correlations between foreground objects and their backgrounds constitute a salient and practically important class of spurious dependencies. In this work, we revisit the well-known property of high linear additivity in VLM embedding spaces and show that it enables a decomposition of scene representations into foreground and background components. Leveraging this insight, we introduce a pre-training approach that exploits this property to construct background-invariant representations using synthetic data. Our method achieves, to our knowledge, the first worst-group accuracy exceeding

90\%

on Waterbirds under perfect (

100\%

) spurious correlation (i.e., no minority-group examples in the training data). Furthermore, it demonstrates strong sim-to-real transfer and requires no access to real-world debiased data, making it practical for real-world deployment.

Youssef Zaazou, Mark Thomas

Independent Researcher

Vision Encoders Equivariant Learning Synthetic Data

May 7, 2026cs.LG

Operator-Guided Invariance Learning for Continuous Reinforcement Learning

Reinforcement learning (RL) with continuous time and state/action spaces is often data-intensive and brittle under nuisance variability and shift, motivating methods that exploit value-preserving structures to stabilize and improve learning. Most existing approaches focus on special cases, such as prescribed symmetries and exact equivariance, without addressing how to discover more general structures that require nonlinear operators to transform and map between continuous state/action systems with isomorphic value functions. We propose \textbf{VPSD-RL} (Value-Preserving Structure Discovery for Reinforcement Learning). It models continuous RL as a controlled diffusion with value-preserving mappings defined through Lie-group actions and associated pullback operators. We show that a value-preserving structure exists exactly when pulling back the value function and pushing forward actions commute with the controlled generator and reward functional. Further, approximate value-preserving structures with rigorous guarantees can be found when the Hamilton--Jacobi--Bellman mismatch is small. This framework discovers exact and approximate value-preserving structures by searching for the associated Lie group operators. VPSD-RL fits differentiable drift, diffusion, and reward models; learns infinitesimal generators via determining-equation residual minimization; exponentiates them with ODE flows to obtain finite transformations; and integrates them into continuous RL through transition augmentation and transformation-consistency regularization. We show that bounded generator/reward mismatch implies quantitative stability of the optimal value function along approximate orbits, with sensitivity governed by the effective horizon, and observe improved data efficiency and robustness on continuous-control benchmarks.

Zuyuan Zhang, Fei Xu Yu, Tian Lan

The George Washington University

Equivariant Learning Action Space Lie Group Approximation

May 6, 2026cs.LG

Beyond Rigid Geometries: The Spline-Pullback Metric for Universal Diffeomorphic SPD Representation Learning

The integration of Symmetric Positive Definite (SPD) matrices into deep learning has historically relied on fixed algebraic Riemannian metrics. Analogous to hand-crafted features in classical machine learning, these static formulations impose rigid geometries limiting network expressivity and adaptability. Recent attempts to parameterize these geometries often violate the axioms of primary matrix functions through unconstrained powers or rank-dependent scaling, inviting spatial folding, loss of global surjectivity, and gradient collapse at spectral singularities. In this paper, we introduce the Spline-Pullback Metric (SPM), instantiated as Spectral-SPM and Cholesky-SPM, marking a paradigm shift from static metric selection to universal geometric approximation. By parameterizing the global diffeomorphism via a rank-invariant, monotonically constrained B-spline, SPM acts as a dense universal approximator for strictly increasing

C^1

diffeomorphisms and theoretically subsumes existing pullback metrics while enabling localized non-linear spectral modelling. Topologically, SPM provides a globally bijective pullback geometry precluding rank-swapping discontinuities and gradient instabilities. Empirically, SPM achieves a state-of-the-art performance across 3 datasets utilizing Linear Probes, SPDNets, and deep Riemannian ResNets.

Tushar Das, Subrata Dutta, Sarmistha Neogy +1

National Institute of Technology Jamshedpur · Jadavpur University · Machine Vision and Intelligence Lab National Institute of Technology Jamshedpur

Splines Equivariant Learning Residual Networks

May 5, 2026cs.LG

DynaTab: Dynamic Feature Ordering as Neural Rewiring for High-Dimensional Tabular Data

High-dimensional tabular data lacks a natural feature order, limiting the applicability of permutation-sensitive deep learning models. We propose DynaTab, a dynamic feature ordering-enabled architecture inspired by neural rewiring. We introduce a lightweight criterion that predicts when feature permutation will benefit a dataset by quantifying its intrinsic complexity. DynaTab dynamically reorders features via a neural rewiring algorithm and processes them through a compact, dynamic order-aware combination of separate learned positional embedding, importance-based gating, and masked attention layers, compatible with any sequence-sensitive backbone. Trained end-to-end with bespoke dynamic feature ordering (DFO) and dispersion losses, DynaTab achieves statistically significant gains, particularly on high-dimensional datasets, where it is benchmarked against 45 state-of-the-art baselines across 36 different real-world tabular datasets. Our results position DynaTab as a compelling new paradigm for high-dimensional tabular deep learning.

Al Zadid Sultan Bin Habib, Gianfranco Doretto, Donald A. Adjeroh

Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26506, USA · Scientific Computing and Imaging Institute & Department of Biomedical Informatics, The University of Utah, Salt Lake City, UT 84112, USA

Tabular Learning High-Dimensional Equivariant Learning Order Matters Landscapes

Apr 29, 2026physics.chem-ph

Towards Accelerated SCF Workflows with Equivariant Density-Matrix Learning and Analytic Refinement

We present \textsc{dm-PhiSNet}, a physically constrained \textsc{PhiSNet}-based equivariant model that predicts one-electron reduced density matrices (1-RDMs) directly from molecular geometries in an atomic-orbital (AO) basis for accelerated self-consistent field (SCF) workflows. Training follows a two-stage schedule with progressively introduced physically motivated objectives, and the resulting predictions are refined by a lightweight analytic block. This block enforces electron-number conservation, drives the 1-RDM toward generalized idempotency in the AO metric, and regularizes the occupation spectrum of the Löwdin-orthogonalized density. Across six closed-shell systems -- H

_2

O, CH

_4

, NH

_3

, HF, ethanol, and NO

_3^-

-- the refined 1-RDMs provide SCF initial guesses that substantially reduce iteration steps by 49--81% relative to standard initializations. Beyond SCF acceleration, the learned 1-RDMs yield accurate one-shot total energies and Hellmann--Feynman atomic forces without force supervision, indicating that the model captures chemically meaningful electronic structure. These results demonstrate that combining equivariant learning with analytic constraint enforcement provides a simple, general route to solver-ready density-matrix initializations and accelerated SCF workflows.

Zuriel Y. Yescas-Ramos, Andrés Álvarez-García, Huziel E. Sauceda

Instituto de Física, Universidad Nacional Autónoma de México, Cd. de México C.P. 04510, Mexico

Quantum Chemistry Equivariant Learning

Apr 22, 2026cs.LG

Learning to Solve the Quadratic Assignment Problem with Warm-Started MCMC Finetuning

The quadratic assignment problem (QAP) is a fundamental NP-hard task that poses significant challenges for both traditional heuristics and modern learning-based solvers. Existing QAP solvers still struggle to achieve consistently competitive performance across structurally diverse real-world instances. To bridge this performance gap, we propose PLMA, an innovative permutation learning framework. PLMA features an efficient warm-started MCMC finetuning procedure to enhance deployment-time performance, leveraging short Markov chains to anchor the adaptation to the promising regions previously explored. For rapid exploration via MCMC over the permutation space, we design an additive energy-based model (EBM) that enables an

O(1)

-time 2-swap Metropolis-Hastings sampling step. Moreover, the neural network used to parameterize the EBM incorporates a scalable and flexible cross-graph attention mechanism to model interactions between facilities and locations in the QAP. Extensive experiments demonstrate that PLMA consistently outperforms state-of-the-art baselines across various benchmarks. In particular, PLMA achieves a near-zero average optimality gap on QAPLIB, exhibits remarkably superior robustness on the notoriously difficult Taixxeyy instances, and also serves as an effective QAP solver in bandwidth minimization.

Yicheng Pan, Ruisong Zhou, Haijun Zou +2

School of Mathematical Science, Peking University, Beijing 100871, China · State Key Laboratory of Mathematical Sciences, Institute of Computational Mathematics and Scientific/Engineering Computing, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · Beijing International Center for Mathematical Research, Peking University, Beijing 100871, China

Repair-Assignment Problem Annealing Equivariant Learning Markov Chain Monte Carlo

Feb 27, 2026cs.LG

OSF: On Pre-training and Scaling of Sleep Foundation Models

Polysomnography (PSG) provides the gold standard for sleep assessment but suffers from substantial heterogeneity across recording devices and cohorts. There have been growing efforts to build general-purpose foundation models (FMs) for sleep physiology, but lack an in-depth understanding of the pre-training process and scaling patterns that lead to more generalizable sleep FMs. To fill this gap, we curate a massive corpus of 166,500 hours of sleep recordings from nine public sources and establish SleepBench, a comprehensive, fully open-source benchmark. Leveraging SleepBench, we systematically evaluate four families of self-supervised pre-training objectives and uncover three critical findings: (1) existing FMs fail to generalize to missing channels at inference; (2) channel-invariant feature learning is essential for pre-training; and (3) scaling sample size, model capacity, and multi-source data mixture consistently improves downstream performance.With an enhanced pre-training and scaling recipe, we introduce OSF, a family of sleep FMs that achieves state-of-the-art performance across nine datasets on diverse sleep and disease prediction tasks. Further analysis of OSF also reveals intriguing properties in sample efficiency, hierarchical aggregation, and cross-dataset scaling.

Zitao Shuai, Zongzhe Xu, David Yang +2

University of California, Los Angeles · Emory University

Sleep Pretraining Equivariant Learning Scaling Text Corpora

Dec 16, 2025cs.LG

Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits

Many learning problems are organized by group symmetries. While invariance is often imposed through architectures or group averaging, we ask when it can emerge from training on a finite random subset of an orbit. We study this question in classical Hopfield networks, where strict memorization can be expressed as a linear margin problem. Reparameterizing minimization of energy flow (MEF) as an exponential loss connects gradient descent to the corresponding minimum-norm hard-margin memorizer. Our main result shows that, for independent uniform samples from any finite permutation orbit, the exact sample hard-margin support vector machine (HSVM) concentrates exponentially around the invariant full-orbit HSVM. Consequently, an orbit-size-independent polynomial number of samples suffices both for approximate parameter invariance and for simultaneous memorization of every orbit element; directional convergence transfers this conclusion asymptotically to MEF gradient descent. For graph-isomorphism orbits, we characterize the invariant parameters as a three-dimensional subspace and show that every such orbit is memorizable. For cliques of fixed linear density, additional symmetry sharpens the uniform memorization bound to

O(v^4\log(1/δ))

, exponentially smaller than the orbit size. Together with experiments across several learning rules, these results give a finite-sample account of how optimization bias can recover symmetry from partial group-structured data.

Michael Murray, Tenzin Chan, Kedar Karhadker +1

Mathematical Sciences, University of Bath · Department of Mathematics, UCLA · Algebraic

Hopfield Networks Equivariant Learning Symmetry Support Vector Machines Finite-Sample Near-Earth Objects

Sep 28, 2025cs.LG

GraphIFE: Rethinking Graph Imbalance Node Classification via Invariant Learning

The class imbalance problem refers to the disproportionate distribution of samples across different classes within a dataset, where the minority classes are significantly underrepresented. This issue is also prevalent in graph-structured data. Most graph neural networks (GNNs) implicitly assume a balanced class distribution and therefore often fail to account for the challenges introduced by class imbalance, which can lead to biased learning and degraded performance on minority classes. We identify a quality inconsistency problem in synthesized nodes, which leads to suboptimal performance under graph imbalance conditions. To mitigate this issue, we propose GraphIFE (Graph Invariant Feature Extraction), a novel framework designed to mitigate quality inconsistency in synthesized nodes. Our approach incorporates two key concepts from graph invariant learning and introduces strategies to strengthen the embedding space representation, thereby enhancing the model's ability to identify invariant features. Extensive experiments demonstrate the framework's efficiency and robust generalization, as GraphIFE consistently outperforms various baselines across multiple datasets. The code is publicly available at https://github.com/flzeng1/GraphIFE.

Fanlong Zeng, Wensheng Gan, Kangjie Chen +1

School of Intelligent Systems Science and Engineering, Jinan University, Zhuhai 519070, China · School of Cybersecurity, Tianjin University, Tianjin 300354, China · University of Illinois Chicago, Chicago, USA

Imbalanced Classification Graph Representation Learning Class Imbalance Graph Neural Networks Equivariant Learning

Mar 17, 2025cs.LG

Permutation Learning with Only N Parameters: From SoftSort to Self-Organizing Gaussians

Sorting and permutation learning are key concepts in optimization and machine learning, especially when organizing high-dimensional data into meaningful spatial layouts. The Gumbel-Sinkhorn method, while effective, requires N*N parameters to determine a full permutation matrix, making it computationally expensive for large datasets. Low-rank matrix factorization approximations reduce memory requirements to 2NM (with M << N), but they still struggle with very large problems. SoftSort, by providing a continuous relaxation of the argsort operator, allows differentiable 1D sorting, but it faces challenges with multidimensional data and complex permutations. In this paper, we present a novel method for learning permutations using only N parameters, which dramatically reduces storage costs. Our method extends SoftSort by iteratively shuffling the N indices of the elements and applying a few SoftSort optimization steps per iteration. This modification significantly improves sorting quality, especially for multidimensional data and complex optimization criteria, and outperforms pure SoftSort. Our method offers improved memory efficiency and scalability compared to existing approaches, while maintaining high-quality permutation learning. Its dramatically reduced memory requirements make it particularly well-suited for large-scale optimization tasks, such as "Self-Organizing Gaussians", where efficient and scalable permutation learning is critical.

Kai Uwe Barthel, Florian Barthel, Peter Eisert

Visual Computing Group · HTW Berlin · Berlin, Germany +2

Fused Sinkhorn-Localized Similarity Equivariant Learning Order Matters Gaussian Primitives Differentiable Physics

Feb 26, 2025cs.LG

Invariance Pair Guidance: Robustness to Spurious Correlations via Corrective Gradients

Machine learning models are inherently bound to the distribution of the training data, often exploiting non-causal shortcuts. As a result, achieving robustness to spurious correlations remains a challenge. While existing approaches rely on data manipulation or re-weighting strategies to achieve robustness, they typically require dense group labels, multiple training domains, or specialized pre-processing. We propose Invariance Pair Guidance (IPG), a method to mitigate reliance on spurious correlations using a sparse set of counterfactual pairs. Unlike other methods demanding extensive supervision, IPG utilizes a novel dual-update mechanism to dynamically correct the optimization trajectory. We generate input pairs that isolate the spurious attribute to define the invariance, a characteristic that should not affect the outcome of the model. Based on these pairs, we define a corrective gradient that complements the traditional gradient descent approach. The correction adapts via a predefined invariance condition. Experiments on ColoredMNIST, Waterbirds-100, and CelebA datasets demonstrate the effectiveness of our approach and its robustness to group shifts, supported by a theoretical convergence analysis. IPG offers a data-efficient and theoretically grounded path to robustness.

Martin Surner, Abdelmajid Khelil, Ludwig Bothmann

Institute for Data and Process Science, Landshut University of Applied Sciences, Landshut, Germany. · Computer Science Department, Landshut University of Applied Sciences, Landshut, Germany. · Department of Statistics, LMU Munich, Munich, Germany. +1

Adversarial Training Covariate Shift Equivariant Learning Spurious Correlations

Feb 7, 2025cs.CR

TIF: Learning Temporal Invariance in Android Malware Detectors

Learning-based Android malware detectors degrade over time due to natural distribution drift caused by malware variants and new families. This paper systematically investigates the challenges classifiers trained with empirical risk minimization (ERM) face against such distribution shifts and attributes their shortcomings to their inability to learn \emph{stable} discriminative features. Invariant learning theory offers a promising solution by encouraging models to generate stable representations across environments that expose the instability of the training set. However, the lack of prior environment labels, the diversity of drift factors, and low-quality representations caused by diverse families make this task challenging. To address these issues, we propose TIF, the first temporal invariant training framework for malware detection, which aims to enhance the ability of detectors to learn stable representations across time. TIF organizes environments based on application observation dates to reveal temporal drift, integrating specialized multi-proxy contrastive learning and invariant gradient alignment to generate and align environments with high-quality, stable representations. TIF can be seamlessly integrated into any learning-based detector. Experiments on a decade-long dataset show that TIF excels, particularly in early deployment stages, addressing real-world needs and outperforming state-of-the-art methods.

Xinran Zheng, Shuo Yang, Edith C. H. Ngai +2

University College London, London, UK · The University of Hong Kong, Hong Kong SAR, China · Columbia University, New York, USA

Malware Equivariant Learning Unsupervised Detection Stable Test-Time Adaptation

May 29, 2024cs.LG

Invariant Graph Representations for Continuous-Time Dynamic Graphs Under Distribution Shifts

Continuous-Time Dynamic Graphs (CTDGs) enable fine-grained modeling of evolving relational systems. However, most existing CTDG representation learning methods are tailored to in-distribution settings and exhibit limited robustness under out-of-distribution (OOD) shifts. Although recent causal approaches learn invariant representations via interventions, they are primarily designed for static or discrete-time graphs and become computationally prohibitive for CTDGs due to the combinatorial explosion of structural and temporal variations. To address these challenges, we propose CIR, a framework grounded in a novel structural causal model termed the ICCM. To avoid exhaustive interventions, we leverage the Normalized Weighted Geometric Mean (NWGM) to efficiently approximate interventional predictions. We further instantiate ICCM within a practical deep learning architecture that jointly captures invariant structural and temporal patterns through dedicated subgraph extractors, and maintains an environment memory bank to model distributional shifts across evolving contexts. Extensive experiments demonstrate that CIR consistently outperforms existing methods under diverse OOD scenarios.

Lanting Fang, Yulian Yang, Yawei Zhang +3

1Beijing Institue of Technology · Southeast University · Wuhan University

Physics-Guided Diffusion Causal Discovery Methods Distribution Shifts Equivariant Learning Replay-Based Continual Learning Distributions+1

Date pendingcs.LG

FEAT: A Linear-Complexity Foundation Model for Extremely Large Structured Data

Structured data is widely used in domains such as healthcare, finance, and scientific data management. Recent studies on structured data foundation models (SFMs) aim to support data analysis and mining tasks over such data, but still face scalability and generalization challenges when applied to real-world enterprise databases. First, many SFMs rely on full self-attention, which introduces an O(N^2) computational bottleneck and limits the number of tuples that can be processed jointly. Second, directly replacing attention with linear-complexity sequence models may conflict with the permutation-invariant nature of structured data, introducing artificial order bias and degrading representation quality. Moreover, models trained only on synthetic data may struggle to generalize to the heavy-tailed and heterogeneous distributions commonly found in real-world databases. To address these challenges, we propose FEAT, a linear-complexity foundation model for extremely large structured data. FEAT replaces quadratic attention with a multi-layer dual-axis encoding architecture. It integrates an adaptive-fusion bidirectional state-space model (AFBM) with convolutional gated linear attention (Conv-GLA), enabling cross-tuple contextualization in O(N) time while supporting permutation-invariant representation learning. To improve robustness under real-world data skewness, FEAT further adopts a hybrid structural causal pre-training pipeline with a robust reconstruction objective. Experiments on 12 real-world database benchmarks show that FEAT consistently outperforms representative SFMs on zero-shot tasks and scales linearly with structured-data sample length, achieving up to 50x faster inference latency.

Zhenghang Song, Tang Qian, Lu Chen +7

Zhejiang University · Ant Group · Aalborg University

Wireless Foundation Models Equivariant Learning Graph Foundation Model Linear Attention Convolutional Block Attention Module