Language Model Generalization

Latest papers

23 papers8 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Knowledge That Is Linear Before Fine-Tuning Generalizes After It
Oct 15, 2026Faridoun Mehri, Hinrich SchützeLanguage Model Generalization Factual Knowledge in Language Models
Center for Information and Language Processing, LMU Munich Munich Center for Machine Learning (MCML)
Generalizable Model Editing Could Be More Than Direct Generalization
Oct 11, 2026Xiaotian Ye, Chunyao Yang, Xiaohan Wang +4Model Editing Language Model Editing
University of Cambridge · NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of the Chinese Academy of Sciences +3
Predicting how model propensities generalize out of distribution
Oct 10, 2026June Hunter, Lily Wen, Owen Terry +3OOD Generalization Language Model Generalization
Length Generalization Needs Proper Regularization
Oct 3, 2026Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins +1Length Generalization Language Model Generalization
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · ELLIS Unit Lisbon +1
Evaluating Multi-Dimensional Generalization of Large Language Models in Temporal Extraction Tasks
Oct 1, 2026Fahmid Shahriar Iqbal, Ritam Dutt, Soumitra Das +2Language Model Generalization Temporal Reasoning in Language Models
University of North Texas · Carnegie Mellon University · University of Massachusetts Amherst +1
Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs
Oct 1, 2026Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein +3Prompt Sensitivity LLM Evaluation
Technion – Israel Institute of Technology
Generalization Dynamics of LM Pre-training
Sep 27, 2026Jiaxin Wen, Zhengxuan Wu, Dawn Song +1Language Model Pretraining Language Model Generalization
UC Berkeley · Stanford University, Google DeepMind
How Far Do Persona Effects Generalize in Language Models?
Sep 26, 2026Yufan Zhou, Yuxuan Liu, Enze Ma +6LLM Prompting Language Model Generalization
KU Leuven · East China University of Science and Technology · University of Illinois Chicago +4
Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization
Sep 15, 2026Eduardo Novaes HeringMemory-Augmented Language Models Decoder-Only Language Models
FITec Labs / Ericsson São Paulo
Shared circuits predict whether LLMs generalize across formats in arithmetic reasoning
Sep 3, 2026Andrea Gregor de Varda, Sana Pandey, Pengrui Han +2Language Model Generalization LLM Mathematical Reasoning
Massachusetts Institute of Technology
Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Models
Aug 7, 2026Zili Zhang, Yilin Wang, Heng Wang +2Transformer Interpretability OOD Generalization
1Xi’an Jiaotong University · University of Illinois Urbana-Champaign
Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures
Jul 18, 2026Supantho Rakshit, Adele Goldberg, Henry ConklinMultilingual Language Model Evaluation Cross-Lingual Reasoning in Language Models
Department of Electrical and Computer Engineering Princeton University · Department of Psychology Princeton University · Laboratory for Artificial Intelligence Princeton University
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
Jul 13, 2026Shikai Qiu, Marc Finzi, Yujia Zheng +2Model Compression Neural Network Generalization
New York University · Carnegie Mellon University
Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
Jul 9, 2026Lu Dai, Ziyang Rao, Yili Wang +3Knowledge Distillation LLM Fine-Tuning
HKUST · HKUST(GZ)
Forecasting With LLMs: Improved Generalization Through Feature Steering
Jun 25, 2026Humzah Merchant, Bradford LevyLook-Ahead Bias in Language Models LLM Interpretability
University of Chicago, Illinois, United States.
A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio
May 27, 2026Ali Shehper, Ashish VaswaniNeural Network Generalization Language Model Generalization
Work done at Essential AI.
InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting
May 14, 2026Mahdi Sabbaghi, George Pappas, Adel Javanmard +1Supervised Fine-Tuning Language Model Generalization
University of Pennsylvania · University of Southern California
Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
Apr 29, 2026Bao Pham, Mohammed J. Zaki, Luca Ambrogioni +2Memorization in Generative Models Neural Network Memorization
1Rensselaer Polytechnic Institute (RPI) · 2Donders Institute for Brain, Cognition, and Behaviour, Radboud University · 3Dynamical Mind +1
Generalization Boundaries of Fine-Tuned Small Language Models for Graph Structural Inference
Apr 20, 2026Michal PodstawskiSupervised Fine-Tuning Small Language Models
NASK National Research Institute Warsaw, Poland
Generalization in LLM Problem Solving: The Case of the Shortest Path
Apr 16, 2026Yao Tong, Jiayuan Ye, Anastasia Borovykh +1Inference-Time Scaling Language Model Generalization
National University of Singapore · Capital Fund Management · Google Research
General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
Apr 13, 2026Junlin Liu, Shengnan An, Shuang Zhou +10LLM Reasoning Language Model Generalization
University of Chinese Academy of Sciences · Meituan
The Validity of Coreference-based Evaluations of Natural Language Understanding
Feb 18, 2026Ian PoradaLanguage Model Generalization Coreference Resolution
School of Computer Science, McGill University, Montreal, Quebec
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
Jan 20, 2025Zijin Hong, Hao Wu, Su Dong +8Mathematical Reasoning Benchmarks Language Model Generalization
The Hong Kong Polytechnic University, Hong Kong SAR, China · University of Electronic Science and Technology of China, Chengdu, China · Tencent Youtu Lab, Shanghai, China +2

Oct 15, 2026J/K move · Enter open · S save

Knowledge That Is Linear Before Fine-Tuning Generalizes After It

Faridoun Mehri, Hinrich Schütze

Center for Information and Language Processing, LMU Munich Munich Center for Machine Learning (MCML)

Language Model Generalization Factual Knowledge in Language Models LLM Fine-Tuning Representation Geometry in Language Models

Open paper AskPDF arXiv

Not all facts a language model can recall support further learning equally well. We distinguish active facts, which support generalization beyond their own recall, from inert facts, which can be recalled but do not support such generalization in a given task. We study this distinction by constructing new relations from existing ones. Take the relation that maps a place to its continent (Paris -> Europe) and assign each of its answers an arbitrary new label (Europe -> crab, Asia -> zebra). The new relation then maps Paris -> crab. We fine-tune the model on the new relation for a subset of subjects, without telling it the original relation or the relabeling rule, and test whether it assigns the corresponding labels to held-out subjects that share an answer in the original relation. Having learned Paris -> crab, does it answer Munich -> crab? Across eleven language models and 70 relations, how linearly readable the original relation is in pretrained representations strongly predicts this generalization. Within each model, the two correlate across relations at Pearson r = 0.87 to 0.92. This association persists when controlling, separately and jointly, for the number of fine-tuning subjects and answers, the model's accuracy on the original relation, and how often the subjects occur in pretraining text. It is also robust to re-randomizing the label assignment and fine-tuning seed, and to replacing the animal labels with labels from thirteen other semantic categories. When we map a fine-tuned model's wrong label back to the original answer it was assigned to, Munich -> zebra becomes Munich -> Asia. Answers recovered this way are more often among the linear probe's top answers for the subject than among the pretrained model's own. Whether a new relation generalizes depends less on whether its facts can be recalled than on how readily their shared structure can be read out linearly.