Inductive Bias

Follow this topic Ask about this topic

Inductive bias is the structure a learner brings to a problem, influencing which solutions it prefers when the training data leave many possibilities open. Researchers study biases from architectures such as convolutional and recurrent networks, as well as those introduced through representations, attention masks, regularization, and the choice of training examples. These approaches are tested on tasks including compositional generalization, sequence modeling, scientific prediction, and data-scarce vision, where they can improve learning or extrapolation beyond the training distribution. A central challenge is that useful biases are often task-specific: a structure that helps one problem may transfer poorly, and good in-distribution performance alone does not establish reliable generalization.

Latest papers

22 papers1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Understanding Generalization Requires Universal Induction
Sep 28, 2026Aram Ebtekar, Marcus Hutter, Danica J. SutherlandInductive Bias Statistical Learning Theory
AIXI Labs · Google DeepMind & ANU · UBC & Amii
A Data-free Universal Prior over Syntactic Structures
Sep 15, 2026Fermín Moscoso del Prado MartínInductive Bias Computational Psycholinguistics
Department of Computer Science and Technology, University of Cambridge, United Kingdom
Learning the Geometry of Admissible Hypotheses through Inductive Bias in Training Distributions
Aug 31, 2026James Crowley, Faez Ahmed, Anton van BeekRepresentation Learning PDE Discovery
School of Mechanical and Material Engineering University College Dublin · Department of Mechanical Engineering Massachusets Institute of Technology
M-Net: Integrating Spectral Features and Physical Field Operators into Deep Learning for Medical Image Segmentation
Aug 12, 2026Jing Zhu, Ye Wang, Fumin WangU-Net Inductive Bias
School of Physics, Xi’an Jiaotong University, Xi’an 710049, China · University of Sydney, Sydney, NSW 2006, Australia
Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity
Aug 11, 2026Junyong Choi, Cheolhyeon Park, Jaehoon ChoVision Transformer Cross-Architecture Knowledge Distillation
Hyundai Motor Company · School of Electronics and Avionics Engineering, Korea Aerospace University
Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation
Aug 4, 2026Khai-Nguyen Nguyen, Oscar Chaparro, Antonio MastropaoloImage-to-Code Generation Code Generation
William & Mary Williamsburg, USA · University of Virginia
Benign interpolation and Occam's razor
Aug 4, 2026Tom F. Sterkenburg, Daniel A. Herrmann, Jan-Willem RomeijnBenign Overfitting Inductive Bias
Statistical learning theory and Occam's razor: Regularization
Aug 4, 2026Tom F. SterkenburgInductive Bias
Making Transformers Logical
Aug 1, 2026Mingze Ma, Hemanth Saratchandran, Cameron Gordon +1Transformer Attention Inductive Bias
Australian Institute for Machine Learning, Adelaide University
The Giant Hippocampus: From Structural Monoculture to a System of Systems
Jul 22, 2026Jaeho SeolTransformer Inductive Bias
Independent Researcher
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Jul 20, 2026Damien Teney, Liangze Jiang, Hemanth Saratchandran +1Neural Network Activation Functions Inductive Bias
Idiap Research Institute · Adelaide University · EPFL
What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking
Jul 14, 2026Gunner Levi HoweContrastive Learning Inductive Bias
The FIL Hypothesis: Inductive Biases Help with Kernel Engineering
Jun 29, 2026Nikolai Rozanov, Subhabrata Dutta, Preslav Nakov +1Inductive Bias
1NLP Department, MBZUAI, Abu Dhabi, UAE · Department of Computing, Imperial College London, London, UK · 2Ubiquitous Knowledge Processing, TU Darmstadt, Darmstadt, Germany
Are Humans Evolved Instruction Followers? An Underlying Inductive Bias Enables Rapid Instructed Task Learning
Jun 29, 2026Anjishnu KumarInstruction Following Inductive Bias
Amazon Alexa AI Seattle, USA
Investigating Inductive Biases for Machine Learning Emulation of Sudden Stratospheric Warmings in Idealised Isca Simulations
Jun 17, 2026Oskar Bohn Lassen, Simon Driscoll, Stephen I. Thomson +2Neural Surrogate Modeling Climate Science
Department of Technology, Management, and Economics Technical University of Denmark · Department of Applied Mathematics and Theoretical Physics University of Cambridge · Department of Mathematics and Statistics University of Exeter
The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning
Jun 2, 2026Justinas Zaliaduonis, Patrick Putzky, Till Richter +1Contrastive Learning Self-Supervised Learning
Technical University of Munich, Munich, Germany · Stanford University, Stanford, USA · Merantix Momentum GmbH, Berlin, Germany +2
Temporal Task Diversity: Inductive Biases Under Non-Stationarity in Synthetic Sequence Modelling
May 18, 2026Afiq Abdillah Effiezal Aswadi, Oliver Britton, Ross Baker +1Neural Network Generalization In-Context Learning
1Independent · University of Oxford
Does Your Neural Network Extrapolate? Feature Engineering as Identifiability Bias for OOD Generalization
May 8, 2026Leonel Aguilar, Jan Nagler, Christoph Hoelscher +1OOD Generalization Inductive Bias
Chair of Cognitive Science, ETH Zürich · Centre for Human and Machine Intelligence, Frankfurt School · Aisot Technologies AG +1
What Kind of Language is Easy to Language-Model Under Curriculum Learning?
Apr 29, 2026Nadine El-Naggar, Tatsuki Kuribayashi, Ted BriscoeInductive Bias Linguistic Bias in Language Models
Mohamed bin Zayed University of Artificial Intelligence
Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision
Apr 23, 2026Leyla Roksan Caglar, Pedro A. M. Mediano, Baihan LinVisual Representation Learning Rate-Distortion Theory
Windreich Department of AI and Human Health Icahn School of Medicine at Mount Sinai New York, NY, USA · Department of Computing Imperial College London London, UK · Windreich Department of AI and Human Health Department of Psychiatry Icahn School of Medicine at Mount Sinai New York, NY, USA
On the "Induction Bias" in Sequence Models
Feb 20, 2026M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal +1Transformer State Tracking
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Is Generation Required for Data-Efficient Perception?
Dec 9, 2025Jack Brady, Bernhard Schölkopf, Thomas Kipf +2Visual Representation Learning Generative Modeling
Max Planck Institute for Intelligent Systems, Tübingen, Germany · Tübingen AI Center · ELLIS Institute, Tübingen +1

Sep 28, 2026J/K move · Enter open · S save

Understanding Generalization Requires Universal Induction

Aram Ebtekar, Marcus Hutter, Danica J. Sutherland

AIXI Labs · Google DeepMind & ANU · UBC & Amii

Inductive Bias Statistical Learning Theory Inductive Inference

Open paper AskPDF arXiv

Classical statistical theory is insufficient to explain the successes of general-purpose AI models, because it depends on handcrafted inductive biases that it cannot justify. No Free Lunch (NFL) theorems force any learner that beats chance on some environments to underperform on others. We might hope that past experience informs which environments to expect, but NFL applies equally to meta-learning. Thus, any method that makes meaningful predictions necessarily begins with an inductive bias external to the data. Choosing to bias toward short programs yields Solomonoff induction (SI), whose performance is competitive against all computable learners - albeit up to "constants" that become large when comparing against specialized methods that exploit background information. We therefore relativize SI to an information vantage point, biasing toward short programs with access to all preexisting information. This reframes the inductive bias: instead of seeking some absolute notion of simplicity, we favor accessibility with respect to our vantage point. An algorithm can only outpredict the relativized SI to the extent that its code contains additional information about the data, and no algorithm can generate such information. While SI is incomputable and hence not a practical algorithm, it provides a formal optimum for inference in the limit of infinite compute, and there is evidence to suggest that frontier AI systems roughly approximate it. Thus, the only known answer to meta-NFL is rooted in algorithmic information theory, which we should expect to play a fundamental role in explaining the generalization behavior of modern (and future) AI systems.