Factual Knowledge in Language Models

Overview

This field studies which facts language models learn from training data, when they can retrieve them, and whether their answers match reliable evidence. Researchers use closed-book question-answering benchmarks such as SimpleQA and TruthfulQA, alongside multilingual, temporal, and long-tail tests; probing and causal tracing help investigate how knowledge is represented and accessed. Common interventions include retrieval-augmented generation, reinforcement learning, fine-tuning, and targeted knowledge editing. These methods can improve recall or update answers, but a central challenge is doing so without introducing hallucinations, losing other knowledge, or leaving gaps in unfamiliar languages, domains, and multi-step reasoning.

Latest papers

CardsList

213 papers25 papers in the last four weeks, down 32% on the four weeks before. 0.2% of all new papers.

  1. Continual Memorization of Factoids in Language Models

    Nov 11, 2024Howard Chen, Jiayi Geng, Adithya Bhaskar +2Continual Learning for LLMsMemorization in Language Models

  2. All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing

    Oct 22, 2024Advait Deshmukh, Ashwin Umadi, Dananjay Srinivas +1Language Model PretrainingLong-Tail Learning