cs.LGJan 29, 2026

RAPTOR: Ridge-Adaptive Logistic Probes

Authors: Ziqi Gao, Yaotian Zhu, Qingcheng Zeng, Xu Zhao, Ziqing Wang, Feng Ruan, Kaize Ding

Organizations: Northwestern University

Abstract

Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ReLope: From Hidden-State Probing to a Decision Module for Multimodal LLM Routing

    Mar 25, 2026Yaopei Zeng, Congchao Wang, Blake JianHang Chen +1Multimodal Large Language ModelsLLM Routing

  2. ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

    Oct 17, 2026Diyi Hu, Ruizhong Qiu, Lin Qiu +2RL for Language ModelsRL for Language Model Reasoning

  3. What are They Thinking? Delineation, Probing, and Tracking of Concepts in LLMs

    Apr 7, 2026Mohamed Abdelwahab, Michelle Yu Collins, Sihan Chen +5Linear ProbingLLM Interpretability