Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

Swapnil Parekh; Naman Goyal

cs.LGMay 12, 2026

Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

Authors: Swapnil Parekh, Naman Goyal

Organizations: Intuit

Abstract

Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of reasoning theater: deliberative-looking steps that contribute nothing to correctness. This wastes inference tokens, pollutes interpretability, and obscures what the model actually computed. We introduce ProFIL (Probe-Filtered Reinforcement Learning) to reduce theater, increase chain-of-thought faithfulness, and shrink chain length in a single, drop-in extension to Group Relative Policy Optimization (GRPO). A multi-head attention probe is trained once on the frozen base model to detect post-commitment steps from internal activations alone; during GRPO, rollouts whose probe score exceeds a threshold have their advantage zeroed. Our central finding is that a probe trained on a frozen base, with verifier-derived labels and no human annotation, provides a stable signal that suppresses theater while resisting the RL-obfuscation failure mode predicted by prior work. Across four reasoning domains (GSM8K, LiveCodeBench, ToolUse, MMLU-Redux) and two model architectures (Llama-8B, Qwen-7B), ProFIL reduces post-commitment theater by 11--100%, raises faithful-fraction (e.g., +24pp on LiveCodeBench under an independent Claude 3.7 Sonnet judge), and shortens chains by 4--19%, all while preserving or improving task accuracy. ProFIL also beats a matched length-penalty GRPO baseline, isolating the gain as semantic commitment-detection rather than chain compression. Probe weights, training configurations, and rollouts are released across all four domains.

Explore similar work

May 14, 2026cs.AI

Stateful Reasoning via Insight Replay

Chain-of-Thought (CoT) reasoning has become a foundation for eliciting multi-step reasoning in large language models, but recent studies show that its benefits do not scale monotonically with chain length: while longer CoT generally enables a model to tackle harder problems, on a given problem, accuracy typically increases with CoT length up to a point, after which it declines. We identify a major cause of this phenomenon: as the CoT grows, the model's attention to critical insights produced earlier in the trace gradually weakens, making those insights progressively less accessible when they are most needed. Therefore, we propose \textbf{InsightReplay}, a stateful reasoning approach in which the model periodically extracts critical insights from its reasoning trace and replays them near the active generation frontier, keeping them accessible as the reasoning scales. Extensive experiments on a

\mathbf{2}\!\times\!\mathbf{3}\!\times\!\mathbf{4}

benchmark grid, covering model scales

\{\text{8B}, \text{30B}\}

, model families

\{\text{Qwen3.5}, \text{DeepSeek-R1-Distill-Qwen}, \text{Gemma-4}\}

, and reasoning benchmarks

\{\text{AIME}, \text{HMMT}, \text{GPQA Diamond}, \text{LiveCodeBench v5}\}

, show that 3-round InsightReplay yields accuracy gains across \textbf{all 24 settings}, with an averaged improvement of

\mathbf{+1.65}

points over standard CoT, and a largest single-setting gain of

\mathbf{+9.2}

points on R1-Distill-32B's LiveCodeBench v5 subset. Our results suggest that the effectiveness of test-time scaling depends not only on how much a model reasons, but also on whether critical intermediate insights remain accessible throughout long reasoning trajectories.

Bin Lei, Caiwen Ding, Jiachen Yang +2

Chain-of-Thought Reasoning Reasoning Trajectory Deepseek

Ask PDF

May 26, 2026cs.CL

GeoFaith: A Spatio-Temporal Dual View of Faithful Chain-of-Thought

Chain-of-Thought (CoT) reasoning has advanced large language models (LLMs), but outcome-based supervision leads to pervasive post-hoc rationalization, producing plausible yet unfaithful reasoning chains. Most prior faithfulness assessment methods are either unscalable, expensive, or unreliable. We propose GeoFaith, a spatio-temporal framework that leverages latent geometric structure and entropy dynamics to diagnose and enforce faithful reasoning. We develop a scalable bootstrapping pipeline expanding step-level annotations from 1k to 20k samples across four domains, train an 8B faithfulness detector outperforming GPT-5 on standard benchmarks, and design a faithfulness-aware reinforcement learning framework jointly optimizing outcome correctness, process faithfulness, and trajectory consistency. Experiments show the proposed method achieves superior performance on both faithfulness detection and downstream reasoning, producing shorter, more interpretable chains without sacrificing accuracy. Our code will be made available publicly.

Weijiang Lv, Wentong Zhao, Jiayu Wang +3

Explanation Faithfulness Chain-of-Thought Reasoning Geoquery Spatiotemporal Multiplier Bootstrap

Ask PDF

Jul 8, 2026cs.AI

Length Penalties Make Chain-of-Thought Less Monitorable

Recent work trains reasoning models with length penalties to curb overthinking and cut inference cost. We show that these penalties make the chain of thought less monitorable. A length-compressed model still lets misleading hints steer its answers, but it less often verbalizes their influence. We train Qwen3-4B and Qwen3-14B with reinforcement learning under length penalties targeting 60% down to 30% of baseline chain-of-thought length, then evaluate them with nine types of biasing hints on held-out MMLU-Pro-R and four transfer benchmarks. A chain is faithful when an LLM monitor can tell from it that the hint influenced the answer. At the 30% target, accuracy stays near baseline and wrong-answer hints switch answers as often as before. Yet faithfulness drops on every evaluation set for both models, by 39% for Qwen3-14B and 35% for Qwen3-4B on MMLU-Pro-R. A control trained with the same correctness and format rewards but no length penalty leaves faithfulness intact or raises it. Shortening alone does not explain the drop. Compressed chains mention the hint 7 to 35 percentage points less often than the uncompressed model's chains shortened to the same length by random sentence deletion, across both model sizes and all five evaluation sets. Length penalties therefore trade monitorability for inference cost by removing the evidence monitors depend on.

Bryce Little

Chain-of-Thought Reasoning Penalties Multiple-Choice Questions Deep Reinforcement Learning

Ask PDF