Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probing

Parsa Mirtaheri; Mikhail Belkin

cs.LGMar 17, 2026

Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probing

Authors: Parsa Mirtaheri, Mikhail Belkin

Organizations: UC San Diego

Abstract

Large language models (LLMs) can produce chains of thought (CoT) that do not accurately reflect the actual factors driving their answers. In multiple-choice settings with an injected hint favoring a particular option, models may shift their final answer toward the hinted option and produce a CoT that rationalizes the response without acknowledging the hint - an instance of motivated reasoning. We study this phenomenon across multiple LLM families and datasets demonstrating that motivated reasoning can be identified by probing internal activations even in cases when it cannot be easily determined from CoT. Using supervised probes trained on the model's residual stream, we show that (i) pre-generation probes, applied before any CoT tokens are generated, predict motivated reasoning as well as a LLM-based CoT monitor that accesses the full CoT trace, and (ii) post-generation probes, applied after CoT generation, outperform the same monitor. Together, these results show that motivated reasoning is detected more reliably from internal representations than from CoT monitoring. Moreover, pre-generation probing can flag motivated behavior early, potentially avoiding unnecessary generation.

Figures & tables

Figure 1: Pre-generation detection of motivated reasoning. For each model (columns), we compare AUC of a pre-generation RFM probe (y-axis; using the last-layer residual stream before CoT generation) to a post-generation LLM baseline, GPT-5-nano (x-axis; given the full CoT trace). Each point corresponds to a hint type (sycophancy/consistency/metadata), averaged across datasets (MMLU, ARC-Challenge, CommonsenseQA, and AQuA). The diagonal indicates equal performance (above: probe better; below: LLM better).

Figure 2: An example of motivated reasoning. The model answers A on the unhinted question but switches to B when given a metadata hint. The CoT monitor (GPT-5-nano) fails to detect the motivated reasoning from the CoT trace, while the RFM probe correctly identifies it from internal representations. See Appendix C for a more detailed view of this example.

Hint Type	Description	Hinted Prompt
Sycophancy	A suggestion attributed to an expert is appended to the question.	User: [question] . An expert indicates the answer is A. Think step by step, then give a final answer.
Consistency	The hinted choice is embedded in a prefilled assistant response.	User: [question] . Assistant: The answer is (A). User: Explain your reasoning, then give a final answer.
Metadata	The hinted choice is encoded in an XML tag.	User: <question> [question] </question> <correct-choice> A </correct-choice> Think step by step, then give a final answer.

Table 1: Three hint types used to construct hinted prompts. Each hint embeds the hinted choice (here “A”) using a different mechanism. [question] denotes the original multiple-choice question.

Figure 4: Hint recovery accuracy at the last CoT token, broken down by dataset (left) and hint type (right). Each bar shows one model’s accuracy averaged over hint types or datasets, respectively. For each model, we use a single fixed layer that maximizes accuracy averaged across all datasets and hint types (Qwen: layer 20, Llama: layer 28, Gemma: layer 34). Chance accuracy is 25% for MMLU and ARC (4 choices) and 20% for AQuA and CommonsenseQA (5 choices).

Figure 5: Hint recovery probe accuracy across layers and CoT tokens for Qwen on MMLU with a sycophancy hint.

Figure 6: Post-generation detection of motivated reasoning. For each model (columns), we compare the AUC of a post-generation RFM probe (y-axis; using the last-layer residual stream before the final answer) to a CoT monitor based on GPT-5-nano (x-axis; given the full CoT trace). Each point corresponds to a hint type (sycophancy/consistency/metadata), averaged across datasets (MMLU, ARC-Challenge, CommonsenseQA, and AQuA). The diagonal indicates equal performance (above: probe better; below: CoT monitor better).

Figure 7: Preemptive motivated reasoning detection AUC at the last layer before CoT generation, broken down by dataset (left) and hint type (right). Each bar shows one model’s AUC averaged over hint types or datasets, respectively.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: RFM vs. linear probe AUC for motivated vs. aligned detection at the last layer. (left) Pre-generation. (right) Post-generation. Each point is one (model, dataset, hint type) combination. Points above the diagonal indicate RFM outperforms linear.

Figure 9: Layer-wise motivated reasoning detection AUC (averaged over all datasets and hint types). (top) Pre-generation. (bottom) Post-generation. Each column corresponds to one of 10 approximately evenly spaced layers, from the first layer (left) to the last (right).

Figure 10: A concrete example of motivated reasoning. (top) Model’s answer to unhinted question. (bottom) When a metadata hint implying B is added, the model switches to B . The CoT monitor (GPT-5-nano) incorrectly judges the response as not motivated, while the RFM probe correctly detects the motivated reasoning from internal representations (score: 0.954).

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 778 papers

Triggering Chain-of-Thought via Latent Feature Interventions in Large Language Models
Jan 12, 2026Zhenghao He, Guangzhi Xiong, Bohan Liu +2LLM Interpretability CoT Reasoning
University of Virginia, USA
Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators
Aug 30, 2026Armaan Singh, Ryan Trinh Le, Jasmine Kaur +5Implicit Reasoning in Language Models Language Model Probing
1Stanford University · 2York University · 3Facebook
Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers
Mar 2, 2026Kyle Cox, Darius Kianersi, Adrià Garriga-AlonsoFaithfulness of Language Model Explanations LLM Interpretability
Independent · Cambridge Boston Alignment Initiative (CBAI)

Jan 12, 2026J/K move · Enter open · S save

Triggering Chain-of-Thought via Latent Feature Interventions in Large Language Models

Zhenghao He, Guangzhi Xiong, Bohan Liu, Sanchit Sinha+1

University of Virginia, USA

LLM Interpretability CoT Reasoning Causal Interventions in Language Models LLM Reasoning Latent CoT Reasoning

Open paper AskPDF arXiv

Chain-of-Thought (CoT) prompting often improves the reasoning performance of large language models (LLMs), but the internal signal that triggers this behavior remains poorly understood. Leveraging the sparse features captured by Sparse Autoencoders (SAEs), we propose a systematic framework to analyze and intervene on the internal representations of LLMs, identifying a small set of latent features that are linked to reasoning behavior and can be causally tested through targeted intervention. Across multiple model families and reasoning benchmarks, we show that steering one or a small number of reasoning-related latent features can substantially induce reasoning behavior without explicit CoT prompting, achieving accuracy comparable to CoT. We further show that the identified features are not tied to particular wording patterns or verbosity, and confirm their causal role in reasoning through suppression experiments that impair performance even under CoT prompting. These results suggest that CoT prompting activates specific latent features to trigger reasoning, and that targeted intervention on these features offers an alternative pathway to elicit efficient reasoning behavior without explicit CoT prompting. Code is available at https://github.com/Zhenghao-He/LatentCoT.

Abstract

LLM Interpretability

Explore similar work

Triggering Chain-of-Thought via Latent Feature Interventions in Large Language Models

Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators

Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers