cs.AIAug 21, 2026

Why2Speak: Faithful Reasoning for Abstaining Action Policies

Authors: Shreya Mendi, Brinnae Bent

Organizations: Pratt School of Engineering, Duke University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Many agentic systems must repeatedly choose between acting and abstaining, making faithful reasoning important for oversight: an explanation is useful only if it reflects the computation that produced the action. We study this problem through intervention timing in multi-party conversation, where an assistant must decide whether to speak or remain silent. This setting exposes class imbalance, asymmetric action costs, and the possibility that exposing reasoning changes the policy being audited. Using Qwen3-8B, decoded with or without chain-of-thought reasoning, we compare direct decision policies, reasoning policies, supervised fine-tuning, and reinforcement learning. We find a capability-auditability tradeoff: the strongest direct policy achieves higher quality but exposes no reasoning to inspect, while the reasoning policy provides a trace at the cost of lower performance, particularly recall of true intervention opportunities. Supervised fine-tuning either suppresses reasoning or preserves it without improving decision quality, while reinforcement learning also fails to improve the reasoning policy. We identify one mechanism underlying this failure: group relative objectives provide no learning signal on confidently wrong prompts when sampled rollouts all select the same action. Controlled activation probes and behavioral ablations show that standard faithfulness methods can overstate evidence that exposed reasoning reflects the underlying decision process. Probability-based metrics saturate under confident decisions, probes are vulnerable to class imbalance and textual leakage, and reasoning ablations can confound reasoning content with changes in inference mode. Together, these results show that exposing reasoning can change an agent's action policy rather than simply make it observable. We provide controls for evaluating reasoning-based oversight of agents that can act or abstain.

Figures & tables

PolicyDecodeF1FIRMIRAUROC
Concurrent Work SFT †token0.7400.0440.521—
GPT-4o zero-shotno-think0.4620.0651.000—
GPT-4o zero-shotthink0.5010.0600.938—
GPT-5.1 zero-shotno-think0.4360.4730.312—
GPT-5.1 zero-shotthink0.2880.7320.188—
Base Qwen3-8Bno-think0.0960.9890.026—
Table 1: Decision quality across policies and models. Unmarked rows use the matched evaluation ( n=400n=400 , seed 7). FIR: false-intervention rate; MIR: missed-intervention rate. ⋆ Evaluated on a smaller 200-item slice drawn from the same seed-7 shuffle, so absolute values are not directly comparable to the rows above; where the two slices overlap they agree closely (Qwen3-8B no-think 0.096 vs 0.097, think 0.536 in both). A Qwen3-30B token-SFT classifier reaches 0.713 no-think, degrading to 0.562 in think mode. † Llama-3.1-8B ( Nama et al. 2026 ) . ‡ Reports the deployed arm; the behavioral audits in Appendices E–G use a separate checkpoint trained under the same objective with a truncation-dependence counterfactual. § Base, non-instruction-tuned, with no native reasoning toggle; it follows the prompted-CoT format only 24% of the time and is therefore used for probing only.
Figure 1: The masked fine-tune degrades only when the intervention frequency in the training data exceeds that of the deployment distribution. Think-mode macro-F1 declines for intervention-enriched training mixtures but remains stable when the training prior matches deployment. Reasoning generation and AUROC remain essentially unchanged.
Figure 2: Layer-resolved decision and intervention-type decodability for the base Qwen3-8B reasoning policy. Each point is a linear probe trained on hidden activations from one transformer layer (x-axis); the y-axis is held-out probe performance (left: decision AUROC, chance 0.5; right: five-way type balanced accuracy, chance 0.20). The gray band shows probes trained on shuffled labels. Decision information remains close to the text baseline before reasoning and becomes highly decodable only after reasoning, while intervention-type representations are strongest before reasoning and weaken toward its conclusion.
StatisticValue
Source scenarios11,498
Synthetic conversations≈\approx 16,000
Token-level decision points173,325
Intervention prevalence≈\approx 13%
Intervention categories5
Table 2: Dataset statistics.
ModelParametersRole
Qwen3-8B8BPrimary experiments
Qwen3-30B30BSame-family scaling
Nemotron-3-Nano-30B-A3B30B (MoE)Cross-family replication
Llama-3.2-3B3BProbe analyses only
GPT-4oProprietaryFrontier baseline
GPT-5.1ProprietaryFrontier baseline
Table 3: Model configurations.
HyperparameterToken-SFTMasked SFT
LoRA rank1616
Learning rate1e-45e-5
Batch size3216
Epochs24
OptimizerAdamAdam
Max sequence length40962048
Table 4: Supervised fine-tuning hyperparameters.
HyperparameterValue
AlgorithmGRPO
Learning rate2e-5
Rollout group size8
Sampling temperature0.8
λ\lambda0.3
LoRA rank16
Table 5: Reinforcement-learning hyperparameters. Hyperparameters for the increased-gradient experiment are given separately in Appendix C.
ResourceValue
GPUNVIDIA A10G
GPU memory24 GB
Total training time≈\approx 2–3 h per RL run; ≈\approx 1–2 h per SFT run (hosted platform)
Probe training time≈\approx 1–2 h per policy
SoftwarePython 3.12, PyTorch 2.13.0, scikit-learn 1.9.0, CUDA 13.0
Table 6: Computational resources.
ExperimentTrainingEvaluationReported in
Capability comparisonNoneMacro-F1Table 1
Cross-model replicationNoneMacro-F1Table 1
Token SFTLoRAMacro-F1Supervised FT
Masked SFTLoRAMacro-F1Figure 1
Prior-shift studyMasked SFTMacro-F1, AUROCFigure 1
RL (accuracy)GRPOMacro-F1Reinforcement L.
Table 7: Summary of experimental conditions. All experiments use Qwen3-8B except the cross-model replication, which additionally evaluates Qwen3-30B, Nemotron-3-Nano, and Llama-3.2-3B.
Training mixtureSpeak prev.Purpose
Matched13%Matches deployment distribution
Moderately enriched38%Intermediate prior shift
Strongly enriched50%Balanced training mixture
Table 8: Training mixtures used in the prior-shift experiment.
Training mixtureThink rate
Base model≈\approx 1.00
13% speak≈\approx 0.99
38% speak≈\approx 0.99
50% speak≈\approx 0.99
Table 9: Think-mode generation rate during masked supervised fine-tuning.
Training mixtureMacro-F1AUROC
Base model0.5360.637
13% speak0.455—
38% speak0.372—
50% speak0.3570.66 [0.58, 0.74]
Table 10: Threshold-free discrimination after masked supervised fine-tuning. AUROC was measured for the base model and the most-shifted (50%) mixture; the stability claim rests on that endpoint comparison.
ModelMixtureThink rateMacro-F1
Base—≈\approx 1.000.420
Masked SFT52% speak≈\approx 0.990.26 †
Table 11: Cross-family replication on Nemotron-3-Nano. † From 0.44 at epoch 0.
Figure 3: Held-out macro-F1 throughout reinforcement learning. Increasing rollout diversity (temperature 1.0, groups of 16, 250 iterations) reduced zero-variance rollout groups by approximately five- to ten-fold but did not produce a higher performance plateau than the original configuration. Error bars show ±\pm 1 SE across the 60-item held-out evaluation set.
Figure 4: Decision decodability along the chain of thought (base policy, within-family decision AUROC, nested cross-validation, mean ±\pm std over ten splits). Decodability is flat at the text-baseline level through 75 percent of the reasoning and rises to near-ceiling only at its close.
Figure 5: Within-family decision decodability by layer for the base, RL-accuracy, and RL-faithfulness policies (AUROC, chance 0.5; shaded region shows the shuffled-label band computed on the base policy). Left panel: pre-CoT, all policies within the noise band at all depths. Right panel: end-of-think, all policies saturate together by roughly layer 16 to 20.
InterventionFlip rate
Partial reasoning0.277
Neutral filler0.403
Table 12: Decision flip rates under behavioral interventions.
PolicyDependence gapMcNemar pp
Base policy+0.126+0.1267.0×10−87.0\times 10^{-8}
RL accuracy (seed 1)+0.046+0.0460.0200.020
RL accuracy (seed 2)−0.070-0.0700.00150.0015
RL accuracy (seed 3)+0.110+0.1102.5×10−72.5\times 10^{-7}
RL faithfulness (seed 1)+0.080+0.0801.2×10−41.2\times 10^{-4}
RL faithfulness (seed 2)+0.110+0.1108.7×10−68.7\times 10^{-6}
Table 13: Filler-controlled behavioral dependence.
Intervention typeAnnot. 1Annot. 2Classifier
Data provision394546
Factual correction272221
Concept definition181010
Synthesis & reframing163634
Source identification1369
None710
Table 14: Label distributions for the two human annotators and the automatic classifier on the 120-item validation sample. Annotator 2 applies synthesis or reframing at close to the classifier’s rate (36 vs. 34) where annotator 1 uses it half as often (16), which accounts for most of the difference between their agreement scores. The largest annotator–annotator disagreements are data provision labelled as synthesis or reframing (13 items), and concept definition and factual correction each labelled as data provision (9 items apiece).
PolicyAgreement
Base policy0.378
RL (accuracy)0.425
RL (faithfulness)0.474
Table 15: Agreement between stated and gold intervention types.
PolicyPositionGoldStatedPerm. pp
BasePre-CoT0.3480.2580.311
BaseEnd-of-CoT0.2020.4040.001
RL (acc.)Pre-CoT0.2020.5120.072
RL (acc.)End-of-CoT0.1430.5120.102
RL (faith.)Pre-CoT0.3660.2930.288
RL (faith.)End-of-CoT0.1950.3900.006
Table 16: Alignment between probe predictions and intervention types on the disagreement subset (all policies). Each cell is the fraction of disagreement cases whose probe prediction matches the given type; pp is the permutation test for the stated-versus-gold gap.
PolicyMacro-F1AUROCDependence gap
Base think0.5360.637+0.126+0.126
RL (accuracy)0.5200.640+0.046+0.046
RL (faithfulness)0.500—+0.080+0.080
Table 17: Behavioral evaluation of reinforcement-learning objectives. The faithfulness objective was trained with two dependence counterfactuals, yielding two distinct checkpoints. Table 1 of the main paper reports the empty-think variant (0.549, AUROC 0.622); the row above is the truncation-dependence variant (0.500, AUROC not measured on the enriched slice), which all behavioral and probe analyses use.
ObjectiveSeedDependence gap
Accuracy1+0.046+0.046
Accuracy2−0.070-0.070
Accuracy3+0.110+0.110
Faithfulness1+0.080+0.080
Faithfulness2+0.110+0.110
Faithfulness3+0.063+0.063
Table 18: Behavioral dependence across independent training runs.
Figure 6: Behavioral dependence across independent reinforcement-learning runs. Between-run variability exceeds the observed differences between reward objectives.
More in this topic

Faithfulness of Language Model Explanations

Faithfulness of language model explanations measures whether stated reasons reflect the evidence and processes driving model outputs.

See all 107 papers

Explore similar work

CardsList
Oct 13, 2026

When Should Enterprise Policy Decisions Be Learned, and When Should They Be Reasoned?

An enterprise agent that answers under a written policy must choose an action before it says anything: answer, ask for a missing fact, ask for confirmation, call one or several tools, escalate, or refuse. Holding one 27B Korean enterprise model fixed, we vary whether the decision is a generated JSON object or a single constrained token, and whether the model may reason first. On 2800 in-distribution items we did not detect an accuracy difference between the two deployed paths (93.46% vs 94.14%), while they differ about tenfold in restart stability (3.00% of actions flip vs 0.29%); they also differ in reasoning, so this is an operational comparison. On a sealed benchmark of 300 items whose labels come from a hidden executable policy, reasoning is worth +17.3 points [+12.3, +23.0] on exact action. With reasoning off, the format difference was 0.0 points [-4.3, +4.0], a failure to detect an advantage for either interface, not equivalence; with reasoning on it is unresolved. Against a token-matched random control, boundary-targeted supervision moves the two covered boundaries by +17.5 and +20.0 points and leaves a held-out boundary unchanged. In a 3-seed training x reasoning factorial, the sealed-rendering interaction is -4.9 [-8.1, -1.7], consistent with partial substitution. All supervision was Korean. On independently rendered English versions, the model is 0.82 points lower [-2.68, +0.93] with reasoning on, the same order as re-rendering in Korean; the reasoning-off gain from targeted supervision is not stable in either language, so we do not claim transfer. The original text stays private as the confirmatory set. An open-model re-rendering made after these results reproduced the reasoning effect and the English verdict; the interaction kept its sign but did not replicate under item-clustered intervals. We release the model and that re-rendering with gold labels.

Hyojung Han, Ryangkyung Kang, Youngeun Kim +1

ThakiCloud · Korea University

LLM Decision-MakingLLM ReasoningConstrained Decoding
AskPDF
May 12, 2026

Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of reasoning theater: deliberative-looking steps that contribute nothing to correctness. This wastes inference tokens, pollutes interpretability, and obscures what the model actually computed. We introduce ProFIL (Probe-Filtered Reinforcement Learning) to reduce theater, increase chain-of-thought faithfulness, and shrink chain length in a single, drop-in extension to Group Relative Policy Optimization (GRPO). A multi-head attention probe is trained once on the frozen base model to detect post-commitment steps from internal activations alone; during GRPO, rollouts whose probe score exceeds a threshold have their advantage zeroed. Our central finding is that a probe trained on a frozen base, with verifier-derived labels and no human annotation, provides a stable signal that suppresses theater while resisting the RL-obfuscation failure mode predicted by prior work. Across four reasoning domains (GSM8K, LiveCodeBench, ToolUse, MMLU-Redux) and two model architectures (Llama-8B, Qwen-7B), ProFIL reduces post-commitment theater by 11--100%, raises faithful-fraction (e.g., +24pp on LiveCodeBench under an independent Claude 3.7 Sonnet judge), and shortens chains by 4--19%, all while preserving or improving task accuracy. ProFIL also beats a matched length-penalty GRPO baseline, isolating the gain as semantic commitment-detection rather than chain compression. Probe weights, training configurations, and rollouts are released across all four domains.

Swapnil Parekh, Naman Goyal

Intuit

RL for Language Model ReasoningFaithfulness of Language Model ExplanationsLLM InterpretabilityCoT FaithfulnessGroup Relative Policy OptimizationReasoning Trace Analysis+1
AskPDF
Apr 9, 2026

Verify Before You Commit: Towards Faithful Reasoning in LLM Agents via Self-Auditing

In large language model (LLM) agents, reasoning trajectories are treated as reliable internal beliefs for guiding actions and updating memory. However, coherent reasoning can still violate logical or evidential constraints, allowing unsupported beliefs repeatedly stored and propagated across decision steps, leading to systematic behavioral drift in long-horizon agentic systems. Most existing strategies rely on the consensus mechanism, conflating agreement with faithfulness. In this paper, inspired by the vulnerability of unfaithful intermediate reasoning trajectories, we propose Self-Audited Verified Reasoning (SAVeR), a novel framework that enforces verification over internal belief states within the agent before action commitment, achieving faithful reasoning. Concretely, we structurally generate persona-based diverse candidate beliefs for selection under a faithfulness-relevant structure space. To achieve reasoning faithfulness, we perform adversarial auditing to localize violations and repair through constraint-guided minimal interventions under verifiable acceptance criteria. Extensive experiments on six benchmark datasets demonstrate that our approach consistently improves reasoning faithfulness while preserving competitive end-task performance.

Wenhao Yuan, Chenchen Lin, Jian Chen +3

The University of Hong Kong · Sun Yat-sen University

Faithfulness of Language Model ReasoningLLM Agent ReliabilityAI Agent AuditingBelief Updating in LLMs
AskPDF