Factual Knowledge in Language Models

Follow this topic Ask about this topic

This field studies which facts language models learn from training data, when they can retrieve them, and whether their answers match reliable evidence. Researchers use closed-book question-answering benchmarks such as SimpleQA and TruthfulQA, alongside multilingual, temporal, and long-tail tests; probing and causal tracing help investigate how knowledge is represented and accessed. Common interventions include retrieval-augmented generation, reinforcement learning, fine-tuning, and targeted knowledge editing. These methods can improve recall or update answers, but a central challenge is doing so without introducing hallucinations, losing other knowledge, or leaving gaps in unfamiliar languages, domains, and multi-step reasoning.

Latest papers

195 papers36 papers in the last four weeks, up 57% on the four weeks before. 0.2% of all new papers.

Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts
Apr 9, 2026Jiayuan Ye, Vitaly Feldman, Kunal TalwarMemorization in Language Models Factual Knowledge in Language Models
National University of Singapore · Apple
Addressing the Reasoning Gap: Mechanistic Circuit-Based Knowledge Editing in Large Language Models
Apr 7, 2026Tianyi Zhao, Yinhan He, Wendy Zheng +1Knowledge Editing Factual Knowledge in Language Models
University of Virginia
What Models Know, How Well They Know It: Knowledge-Weighted Fine-Tuning for Learning When to Say "I Don't Know"
Apr 7, 2026Joosung Lee, Hwiyeol Jo, Donghyeon Ko +3LLM Abstention Fine-Tuning
NAVER CLOUD · Seoul National University · KAIST
Unmasking Hallucinations: A Causal Graph-Attention Perspective on Factual Reliability in Large Language Models
Apr 5, 2026Sailesh kiran kurra, Shiek Ruksana, Vishal BorusuLLM Hallucination Mitigation Factual Knowledge in Language Models
Amazon,Texas, United States of America · Vasavi College of Engineering, Telangana,India
Friends and Grandmothers in Silico: Localizing Entity Cells in Language Models
Apr 1, 2026Itay Yona, Dan Barzilay, Michael Karasik +1Concept Localization in Language Models LLM Interpretability
Mentaleap · Independent Researcher · Tel Aviv University
KEditVis: A Visual Analytics System for Knowledge Editing of Large Language Models
Mar 31, 2026Zhenning Chen, Hanbei Zhan, Yanwei Huang +4Language Model Editing Factual Knowledge in Language Models
Zhejiang University · State Key Lab of CAD&CG, Zhejiang University · Hong Kong University of Science and Technology +2
Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model
Mar 31, 2026Jiuting Chen, Yuan Lian, Hao Wu +4Metacognition in Language Models Factual Knowledge in Language Models
Eaglewood Japan Co., Ltd.
Correcting Suppressed Log-Probabilities in Language Models with Post-Transformer Adapters
Mar 25, 2026Bryan SanchezLLM Alignment Political Bias in Language Models
LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale
Mar 25, 2026Muhammed Saeed, Simon RazniewskiFactual Knowledge in Language Models LLM Reliability
ScaDS.AI Dresden/Leipzig & TU Dresden, Germany
Parametric Knowledge in RAG-SFT for Domain-Specific Document Generation
Mar 24, 2026Julian Oestreich, Maximilian Bley, Frank Binder +3Retrieval-Augmented Generation LLM Fine-Tuning
Institute for Applied Informatics (InfAI) at Leipzig University · CELUS GmbH, Munich
Evaluating Large Language Models on Historical Health Crisis Knowledge in Resource-Limited Settings: A Hybrid Multi-Metric Study
Mar 20, 2026Mohammed Rakibul HasanLLM Reliability LLM Evaluation
Department of Electrical & Computer Engineering, North South University, Dhaka, Bangladesh
Geography According to ChatGPT -- How Generative AI Represents and Reasons about Geography
Mar 19, 2026Krzysztof Janowicz, Gengchen Mai, Rui Zhu +4Geospatial Reasoning Factual Knowledge in Language Models
University of Vienna, Austria · University of Texas at Austin, USA · University of Bristol, UK +4
Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts
Mar 17, 2026Lucas Bandarkar, Alan Ansell, Trevor CohnCross-Lingual Transfer Cross-Lingual Reasoning in Language Models
University of California, Los Angeles · Google Research · University of Melbourne
Localizing and Editing Knowledge in Large Audio-Language Models
Mar 15, 2026Sung Kyun Chung, Jiaheng Dong, Qiuchi Hu +3Concept Localization in Language Models Audio-Language Models
The University of Melbourne, Australia · Wuhan University, China · University of Auckland, New Zealand
On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode
Mar 14, 2026Valeria Ruscio, Keiran ThompsonHallucination in Language Models LLM Interpretability
Intuition Machines Inc.
Large Language Models and Scientific Discourse: Where's the Intelligence?
Mar 13, 2026Harry Collins, Simon ThorneScientific Reasoning in Language Models LLM Reliability
Institute of Education, UCL, 20 Bedford Way, London, WC1H OAL, UK. · School of Technologies, Cardiff Metropolitan University, 200 Western Avenue, Cardiff, CF5 2YB, UK.
The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation
Mar 11, 2026Pavel Braslavski, Dmitrii Iarosh, Nikita Sushko +4Language Model Generation Evaluation Factual Knowledge in Language Models
HSE University · Ural Federal University · ITMO University +3
Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs
Mar 10, 2026Zorik Gekhman, Roee Aharoni, Eran Ofek +3Factual Knowledge in Language Models LLM Reasoning
Google Research · Technion - Israel Institute of Technology · Tel Aviv University
Understanding the Interplay between LLMs' Utilisation of Parametric and Contextual Knowledge: A keynote at ECIR 2025
Mar 10, 2026Isabelle AugensteinKnowledge Conflicts in Language Models Factual Knowledge in Language Models
University of Copenhagen, Denmark
DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models
Mar 5, 2026Alexander SheppertLLM Evaluation Factual Knowledge in Language Models
Legacy Health · Capitol Technology University
Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval
Mar 5, 2026Artem Vazhentsev, Maria Marina, Daniil Moskovskiy +8Automated Fact-Checking Factual Knowledge in Language Models
MBZUAI · S-NLP Group · AXXX +3
World Properties without World Models: Distributional Associations and the Interpretation of Decoding Results from Language Models
Mar 4, 2026Elan BarenholtzLLM World Models Word Embeddings
Department of Psychology & Center for Complex Systems and Brain Sciences Florida Atlantic University
Probing for Knowledge Attribution in Large Language Models
Feb 26, 2026Ivo Brink, Alexander Boer, Dennis UlmerLLM Interpretability Hallucination in Language Models
KPMG NL / University of Amsterdam Amsterdam, The Netherlands · KPMG NL Amsterdam, The Netherlands · University of Amsterdam Amsterdam, The Netherlands
Improving Parametric Knowledge Access in Reasoning Language Models
Feb 25, 2026Melody Ma, John HewittFactual Knowledge in Language Models Knowledge-Intensive QA
Columbia University
How Transformers Reject Wrong Answers: Rotational Dynamics of Factual Constraint Processing
Feb 25, 2026Javier MarínTransformer Interpretability LLM Interpretability
Groundlens-dev
NanoKnow: How to Know What Your Language Model Knows
Feb 23, 2026Lingwei Gu, Nour Jedidi, Jimmy LinFactual Knowledge in Language Models Language Model Pretraining
University of Waterloo Waterloo, ON, Canada
Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning
Feb 23, 2026Anna Borisiuk, Andrey Savchenko, Alexander Panchenko +1Machine Unlearning LLM Unlearning
AIRI · Skoltech · Sber AI Lab +2
Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
Feb 18, 2026Sanket Badhe, Deep Shah, Nehal KathrotiaFactual Knowledge in Language Models Long-Tail Learning
Google, Mountain View, California, USA
Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration
Jan 26, 2026Christopher Kissling, Elena Merdjanovska, Alan AkbikLanguage Model Probing Language Model Calibration
Humboldt-Universität zu Berlin · Science of Intelligence
MAPLE: Metadata Conditioned LLM Pretraining for Locale-Aware Question Answering
Jan 21, 2026Anjishnu Mukherjee, Ziwei Zhu, Antonios AnastasopoulosLanguage Model Pretraining LLM Evaluation
George Mason University

Apr 9, 2026J/K move · Enter open · S save

Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts

Jiayuan Ye, Vitaly Feldman, Kunal Talwar

National University of Singapore · Apple

Memorization in Language Models Factual Knowledge in Language Models LLM Training Training Data Selection

Open paper AskPDF arXiv

Large language models (LLMs) can struggle to memorize factual knowledge in their parameters, often leading to hallucinations and poor performance on knowledge-intensive tasks. In this paper, we formalize fact memorization from an information-theoretic perspective and study how training data distributions affect fact accuracy. We show that fact accuracy is suboptimal (below the capacity limit) whenever the amount of information contained in the training data facts exceeds model capacity. This is further exacerbated when the fact frequency distribution is skewed (e.g. a power law). We propose data selection schemes based on the training loss alone that aim to limit the number of facts in the training data and flatten their frequency distribution. On semi-synthetic datasets containing high-entropy facts, our selection method effectively boosts fact accuracy to the capacity limit. When pretraining language models from scratch on an annotated Wikipedia corpus, our selection method enables a GPT2-Small model (110m parameters) to memorize 1.3X more entity facts compared to standard training, matching the performance of a 10X larger model (1.3B parameters) pretrained on the full dataset.