Language Model Probing

Follow this topic Ask about this topic

Momentum

19 papers in the last four weeks, up 12% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 192

All topics

Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception
Oct 8, 2026Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba +2Deception in Language Models AI Agent Monitoring
Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog
Oct 8, 2026Hadeel Al-Negheimish, Jasna Ilieva, Yoon KimLLM Reasoning Long-Context Language Model Inference
King Saud University · Massachusetts Institute of Technology
How High Is 0.6? Floors, Ceilings, and Headroom in Interpretability Probing
Oct 6, 2026Pranjal GargRepresentation Probing Language Model Probing
Independent Researcher
Knowing When Not to Answer: Cross-Domain and Multi-Turn Generalization of Latent Underspecification Signals
Oct 6, 2026Jerzy Kamiński, Ilya Galyukshev, Artem Kuznetsov +6Multi-Turn Dialogue Evaluation Language Model Probing
ITMO University, Saint Petersburg, Russia
Structured but Silent: Probing Capability Requirements in LLM Hidden States
Oct 6, 2026Kyojun Choo, Minsoo Song, Yunju Kang +1LLM Tool Use Language Model Probing
Soongsil University
Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing
Oct 6, 2026Jing Chen, Giulia Loca, Simona Amenta +1Language Modeling Computational Psycholinguistics
Department of Psychology, University of Milano-Bicocca, Piazza dell’Ateneo Nuovo 1, 20126 Milano, Italy · Department of Informatics, Systems and Communication – DISCo, University of Milano-Bicocca, Milano, Italy
Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
Oct 5, 2026Hashmath Shaik, Gnaneswar Villuri, Alex DoboliLLM Evaluation LLM-as-a-Judge
Department of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11794, USA
Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routing
Oct 5, 2026Andrea Paganelli, Stefano Civelli, Pietro Bernardelle +1Confidence Estimation in Language Models Language Model Calibration
The University of Queensland · Polytechnic University of Milan
External Observers May See More Clearly: Cross-Model Span-Level Hallucination Detection in Large Language Models via Hidden State Probing
Oct 1, 2026Kingshuk Gupta, Davide BuscaldiHallucination Detection LLM Hallucination Detection
LIX, École Polytechnique · School of Electrical and Electronic Engineering, Nanyang Technological University · LIPN, Sorbonne Paris Nord
Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability
Oct 1, 2026Yu Mao, Lei Yu, Zining Zhu +6RL for Language Models Reinforcement Learning
University of Toronto · Stevens Institute of Technology · University of California, Santa Cruz +3
Verbalized and Internal Probabilities Are Coupled in Large Language Models
Sep 30, 2026Sinead Williamson, Jiaxuan Li, Nick Foti +2Confidence Estimation in Language Models LLM Uncertainty Estimation
Apple
Lingtai: What Concept Geometry Reveals--and Does Not Reveal--About LLM Inference
Sep 30, 2026Jiangang ChenLLM Inference LLM Interpretability
Chengdu Beiluoshimen Technology Co., Ltd.
Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlations
Sep 30, 2026Maximilian von Klinski, Sebastian Lapuschkin, Wojciech Samek +1Deception in Language Models LLM Reliability
Fraunhofer HHI · TU Dublin · TU Berlin +1
Reach Into The CHOIR: Free-List Elicitation Uncovers Distinct Model Voices in LLM Ensembles
Sep 29, 2026Ben Wigler, Maria TsfasmanLanguage Model Ensembles Language Model Probing
LoveMind AI, New York, USA
Locating Answer-Correctness Signals in Frozen Large Language Models
Sep 29, 2026Yuansen Liu, Yixuan Tang, Anthony Kum Hoe TungLLM Answer Verification Language Model Error Detection
School of Computing, National University of Singapore
Reader Proficiency Shapes Layer-wise Surprisal Profiles
Sep 29, 2026Akio Hayakawa, Horacio SaggionLanguage Model Probing
Universitat Pompeu Fabra Barcelona, Spain
Rethinking Reasoning Paths as Phase-Structured Trajectories
Sep 29, 2026Zhenghao He, Guangzhi Xiong, Sanchit Sinha +3Structured Reasoning Language Model Probing
Department of Computer Science, University of Virginia
Encoded but Not Decoded: Layer-Localized Evidence for a Three-Level Gap in LLM Syntax
Sep 24, 2026Zhenyan Lu, He Wang, Xiaohui HuangLLM Evaluation Language Model Decoding
College of International Studies, National University of Defense Technology, Nanjing, China
Hallucination Neurons and Where to Find Them: An Investigation into the existence of Hallucination Neurons
Sep 24, 2026Huseyin Cavus, Sebin Sabu, Joshua Spear +2LLM Interpretability Hallucination in Language Models
Trakya University, Edirne, Türkiye · DRIVE, Great Ormond Street Hospital for Children NHS Foundation Trust, London, UK
The Answer-Basin Representation Hypothesis: We Are Not Probing or Steering Concepts
Sep 21, 2026Manjiang Yu, Hongji Li, Zihan Wang +5LLM Interpretability Linear Representation Hypothesis
The University of Queensland · Mohamed bin Zayed University of Artificial Intelligence · Institute of Science Tokyo
Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models
Sep 16, 2026Alizishaan Khatri, Chiquita Prabhu, Omkar NeogiSafety Filtering LLM Safety
Wrynx Inc. · Independent Researcher
Planning or Improvisation? Stress-Testing the Poetry Planning Site on Open Models and Open Cross-Layer Transcoders
Sep 16, 2026Éric JacopinLLM Planning Mechanistic Interpretability
Cosmic AI, France
When the Wrong Key Wins: Understanding and Detecting Hallucinations in LLMs
Sep 14, 2026Xuhan Tong, Haoyue Bai, Dawei Zhou +2Language Model Error Detection Hallucination in Language Models
University of Wisconsin–Madison
From Parameters to Answers: How LLMs Retrieve and Use Their Internal Knowledge
Sep 12, 2026Wenkang Wei, Yuan Fang, Renhe Jiang +2LLM Interpretability Factual Knowledge in Language Models
University of Science and Technology of China Hefei, China · Singapore Management University Singapore, Singapore · The University of Tokyo Tokyo, Japan +1
Legible Failures: Detecting and Repairing In-Context Binding Errors
Sep 11, 2026Manas Venkata Sai Ravulapalli, Samrath Singh Chadha, Abhinav M. HariLanguage Model Error Detection LLM Reliability
Efficient Computation Inc.
The Truth Was Never Gone: Perfect Aliasing in Compliant-Context Truth Probes
Sep 9, 2026Dylan JayabahuDeception in Language Models Language Model Probing
University of Waterloo
Encoded Early, Used Late: Where Transformers Begin to Act on an Inferred Partner's Expertise
Sep 7, 2026Mika Okamoto, Gabriele SartiTransformer Interpretability Language Model Probing
Georgia Institute of Technology · Northeastern University
It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories
Sep 3, 2026Yigit Utku BulutLLM Evaluation Language Model Probing
Johannes Kepler University Linz
Investigating Linear Probe Robustness to Linguistic Register, Medical Specialty, and Corpus Shifts in Medical QA
Sep 1, 2026Nishant Mishra, Ameen Abu-Hanna, Iacer CalixtoClinical Language Model Evaluation Language Model Robustness
Department of Medical Informatics, Amsterdam UMC, University of Amsterdam · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands
Some Emotions Run Deeper: Layer-wise Probing and Causal Intervention in Large Language Models
Sep 1, 2026Tian Fang, Gaël Guibon, Davide BuscaldiCausal Interventions in Language Models Emotion Representation in Language Models
1Université Sorbonne Paris Nord, CNRS, Laboratoire d’Informatique de Paris Nord, LIPN · 2LORIA, CNRS, Université de Lorraine

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba, Adam Gleave+1

Deception in Language Models AI Agent Monitoring Language Model Probing Language Model Introspection

Open paper AskPDF arXiv

Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes achieve 98.8% AUC in SHADE-Arena, surpassing an Opus 5.5 text-monitoring baseline, and show improved efficacy as the underlying model is scaled up. To push our probes to their limit, we test them on several cases where deception cannot be determined from the context alone. In these cases, which we refer to as introspective deception, the ground truth can only be determined through careful elicitation or thorough knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUC of up to 99.7%. Our probes also readily detect deception on prominent open-weight models which lie about politically sensitive topics, and about their beliefs when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.

Momentum

Related topics

Latest papers 192