Chain of Faith: Policy-Internal Rewards for Reasoning Reliance

Fanyu meng; Zhaodan Kong; Xin Liu

cs.AINov 2, 2026

Chain of Faith: Policy-Internal Rewards for Reasoning Reliance

Authors: Fanyu meng, Zhaodan Kong, Xin Liu

Organizations: University of California, Davis

Abstract

Chain-of-thought (CoT) explanations can sound plausible while having little influence on a model's final answer. We study four rewards that use the policy's own signals to encourage reliance on generated reasoning: gradient sensitivity, confidence gaps, progressive confidence gains, and reasoning--prompt swaps. We add each reward to group relative policy optimization (GRPO) and evaluate two small instruction-tuned models on GSM8K and StrategyQA using reasoning edits, paraphrases, and prompt hints. Several configurations increase answer reliance to reasoning edits, while effects on accuracy and paraphrase robustness vary. We also observe answer copying at a high reward weight and high reliance in decomposition settings with poor task accuracy. The study compares practical objectives for reasoning reliance, confirms the trade-off between reliance and performance, and proposes four lightweight rewards that can improve reasoning reliance.

Figures & tables

Reward	Signal and geometry	Intervention context	Additional computation	Principal limitation
$R_{\mathrm{grad}}$	Mean token gradient norm ( $\ell_{2,1}$ )	Local continuous embedding changes	Answer-likelihood backward pass	Surface sensitivity and answer copying
$R_{\mathrm{conf}}$	Signed likelihood gap / Frobenius distance	Reasoning removed	One answer-likelihood forward evaluation	Context changes and answer leakage
$R_{\mathrm{prog}}$	Positive prefix gains / path length	Answer forced at step boundaries	Answer evaluation at cached prefixes	Confidence oscillations and prefix context changes
$R_{\mathrm{med}}$	Ratio of absolute swap contrasts	Unrelated prompt–trace pairs	Two swapped-input forward evaluations	Incoherent inputs and unidentified causal effects

Table 1: Comparison of the four auxiliary rewards. The signals differ in their geometry, input edits, computational requirements, and limitations.

Figure 1: Answer-change diagnostics. Columns show GSM8K and StrategyQA. Rows show the intervention types. L/Q denote Llama/Qwen, and P/D denote plain/decomposition formats. Colors and hatches identify the rewards. Higher values indicate greater answer reliance, except for paraphrasing, where lower values indicate greater robustness to wording changes.

Figure 2: Native accuracy versus average perturbation sensitivity on GSM8K (left) and StrategyQA (right). Circles denote Llama and triangles Qwen. Filled markers denote plain CoT and hollow markers decomposition. Colors identify the rewards. Dashed lines show the Pareto fronts of the displayed configurations.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Early-answering curves for Llama (top) and Qwen (bottom), on GSM8K (left) and StrategyQA (right). The shared legend identifies all five reward conditions.

Question	Vanilla Model	$R_{\text{grad}}$ Model
A customs officer counts 2 containers of 5 vehicles each on day one. By day two the port total is 30 vehicles. How many containers arrived on day two?	• Total vehicles on the first day = 2 containers × 5 vehicles/container = 10 vehicles. • Total vehicles on the second day = 30 vehicles. • 30 vehicles $-$ 10 vehicles = 20 vehicles. • 20 vehicles / 5 vehicles/container = 4 containers.	• Total vehicles on the first day = 2 * 5 = 10. • Total vehicles on the second day = 30 $-$ 10 = 20. • 20 / 5 = 4.
Toula bought 3 dozen donuts at $68/dozen, 2 dozen mini cupcakes at$ 80/dozen, and 6 dozen mini cheesecakes at $55/dozen. What was the total cost?	• 3 dozen donuts = 3 × $68 =$ 204 • 2 dozen mini cupcakes = 2 × $80 =$ 160 • 6 dozen mini cheesecakes = 6 × $55 =$ 330 • Total cost = $204 +$ 160 + $330 =$ 694	• 3 dozen donuts = 3 × 68 = 204 • 2 dozen mini cupcakes = 2 × 80 = 160 • 6 dozen mini cheesecakes = 6 × 55 = 330 • 204 + 160 + 330 = 694
Nancy owes $0.50 each on 8 overdue library books, plus a flat$ 2.00 fee. How much does she pay in total?	• Nancy owes 8 books × $0.50 =$ 4.00. • Adding the $2.00 fee, the total is$ 4.00 + $2.00 =$ 6.00.	• Total cost of overdue books: 8 × 0.50 = 4.00 • Add the flat fee: 4.00 + 2.00 = 6.00
A factory sold 10 tractors/day at $100 each. It now sells 5 silos/day at$ 220 each. What percent more does it make per day now?	• The factory made 10 tractors a day at $100 each, so they made 10 ×$ 100 = $1000. • Now, they make 5 silos a day at$ 220 each, so they make 5 × $220 =$ 1100. • The difference is $1100$ -1000 = $100. • To find the percentage, divide the difference by the original amount and multiply by 100: ($ 100 / $1000) × 100 = 10%.	• Old price per tractor: $100. Old number of tractors: 10. Old total: 100*10 = 1000. • New price per silo:$ 220. New number of silos: 5. New total: 2205 = 1100. • New total: 1100 $-$ 1000 = 100. • 100/1000100 = 10%.
Jillian’s handbag cost $20 less than 3 times as much as her shoes, which cost$ 80. How much did the bag cost?	• Her shoes cost $80. • 3 times as much as her shoes cost = 3 ×$ 80 = $240. • 3 times as much as her shoes cost$ -20 = $240$ -20 = $220.	• Her shoes cost $80. 3 times that is 3 × 80 = 240. • 240$ -$ 20 = 220.

Appendix

Table 7: Selected correct GSM8K generations from plain-CoT Llama-3.2-1B , comparing vanilla GRPO with the gradient-reward model. Step contents are unchanged, and formatting tags are removed.

Question	Vanilla Model	$R_{\text{grad}}$ Model
A plane travels 1200 miles in 3 hours. At the same rate, how many additional hours to travel an additional 2000 miles?	• The plane travels 1200 miles in 3 hours. • To find the rate, divide the distance by the time: 1200 miles / 3 hours = 400 miles per hour. • To find the time to travel 2000 miles, divide the distance by the rate: 2000 miles / 400 miles per hour = 5 hours.	• Distance = 1200 miles. Rate = 1200/3 = 400 miles per hour. • Additional distance = 2000 miles. Additional time = 2000/400 = 5 hours.
John’s cooking class meets 4 times/week for 2 hours, for 6 weeks. He learns a recipe every 1.5 hours. How many recipes does he learn?	• Number of class sessions = 4 sessions/week × 6 weeks = 24 sessions. • Each session = 2 hours. Total class time = 24 sessions × 2 hours/session = 48 hours. • Number of hours learned = 48 hours. Number of recipes learned = 48 hours / 1.5 hours/recipe = 32 recipes.	• Total class time = 4 * 2 * 6 = 48 hours. • 48 / 1.5 = 32 recipes.
Mrs. Jameson’s bamboo grows 30 inches/day and is currently 20 feet tall. In how many days will it be 600 inches?	• First, convert the height from feet to inches: 20 feet × 12 inches/foot = 240 inches. • Now, subtract the current height from the target height: 600 inches $-$ 240 inches = 360 inches. • Divide the remaining inches by the growth rate: 360 inches / 30 inches/day = 12 days.	• Convert 20 feet to inches: 20 * 12 = 240 inches. • 600 $-$ 240 = 360 inches. • 360 / 30 = 12.
Jon’s car needs a tune-up every 1000 miles. He drives 100 miles/day for a 30-day month. How many tune-ups does he need?	• Jon drives 100 miles a day for 30 days. • 100 miles/day × 30 days = 3000 miles. • Since Jon needs a tune-up every 1000 miles, we divide 3000 miles by 1000 miles to find the number of tune-ups. • 3000 miles $\div$ 1000 miles/tune-up = 3 tune-ups.	• Total miles driven in a month = 100 * 30 = 3000. • Number of tune-ups = 3000 / 1000 = 3.
Sid traveled 110 miles in 2 hours, then an additional 140 miles in 3 hours. What was his average speed?	• First, calculate the total distance traveled: 110 miles + 140 miles = 250 miles. • Then, calculate the total time taken: 2 hours + 3 hours = 5 hours. • Average speed = Total distance / Total time = 250 miles / 5 hours = 50 miles per hour.	• Total distance = 110 + 140 = 250 miles. • Total time = 2 + 3 = 5 hours. • Average speed = 250/5 = 50.

Appendix

Table 8: Continued: selected correct GSM8K generations illustrating prose and symbolic reasoning styles.

Panel of Figure 5 — Figure 5: Completion length during Llama/GSM8K training (left), smoothed over approximately 40-step windows, and length versus average perturbation sensitivity across tasks (right). The right panel labels this sensitivity composite as “faithfulness”.

Panel of Figure 6 — Figure 6: Accuracy reward (left) and gradient reward (right) during training at different $\lambda_{\mathrm{faith}}$ values. Both horizontal axes show training steps.

Figure 7: Answer-likelihood changes under complete removal (left) and 33% removal (right), compared with step shuffling, for vanilla-GRPO Llama on GSM8K.

Figure 8: Answer-likelihood changes under 66% removal (left) and padding replacement (right), compared with step shuffling, for vanilla-GRPO Llama on GSM8K.

Figure 9: Answer-likelihood changes under mistake injection (left) and paraphrasing (right), compared with step shuffling, for vanilla-GRPO Llama on GSM8K.

Score	Spearman $\rho$ with accuracy
Log-probability $\overline{\ell}_{a}$	$-0.2016$
Self-consistency ( $N{=}8$ )	$\phantom{-}0.7237$

Appendix

Table 9: Rank correlations with accuracy for vanilla Llama-3.2-1B on

1{,}317

GSM8K prompts.

	EA-AUC	Shuf.	Mist.	Reph.
$R_{\mathrm{conf}}$	$0.657$	$0.191$	$0.569$	$0.368$
$R_{\mathrm{conf-sc}}$	$0.606$	$0.192$	$0.580$	$0.343$

Appendix

Table 10: Confidence-gap and answer-agreement results for Llama-3.2-1B on GSM8K. The mistake-position aggregation is unspecified.

Method	Full fwd	Bwd	Time ratio
Vanilla	0	0	$1.0\times$
CoT Decomposition	0	0	$1.0\times$
$R_{\text{grad}}$	0	1	$\sim 1.0\times$
$R_{\text{conf}}$	1	0	$\sim 1.2\times$
$R_{\text{prog}}$	0	0	$\sim 1.0\times$
$R_{\text{med}}$	2	0	$\sim 1.1\times$

Appendix

Table 11: Per-rollout wall-time ratios relative to vanilla GRPO, averaged over models and datasets on 8 A5000 GPUs, two AMD EPYC 7763 CPUs, and 256GB RAM. “Full fwd” counts full-prefix recomputation. The progressive reward evaluates answer tokens at cached prefixes. The implementation did not use vLLM.

Item	Weight	Description
Solution wrapper	0.22	Generation starts with <solution> and ends with </solution> and nothing outside
Single solution	0.15	Generation contains exactly one <solution>
Thinking once	0.12	Exactly one <thinking>...</thinking> inside solution
Steps block	0.15	Generation contains at least two <step>...</step> inside thinking. Only awarded if no stray text
Steps no stray text	0.18	Inside <thinking> , ignoring whitespace, there should be nothing outside <step> blocks
Single answer	0.14	Generation contains exactly one <answer>...</answer> inside solution

Appendix

Table 12: Documented plain-CoT formatting-reward components. Boolean-answer and decomposition variants are not specified.

Faithfulness of Language Model Reasoning

See all 41 papers

CoRA: Confidence-Rationale Alignment for Reliable Chain-of-Thought Reasoning
Jun 12, 2026Juming Xiong, Weixin Liu, Kevin Guo +9CoT Reasoning RL for Language Model Reasoning
Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research
Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Mar 5, 2026Siddharth Boppana, Annabel Ma, Max Loeffler +5Faithfulness of Language Model Reasoning Language Model Reasoning Evaluation
Goodfire AI · Harvard University, Cambridge, MA
RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
Jun 4, 2026Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger +1Reinforcement Learning RL for Language Model Reasoning
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria · Cognizant AI Lab, San Francisco, USA · Work done while at Johannes Kepler University Linz. +1

Jun 12, 2026J/K move · Enter open · S save

CoRA: Confidence-Rationale Alignment for Reliable Chain-of-Thought Reasoning

Juming Xiong, Weixin Liu, Kevin Guo, Congning Ni+8

Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research

CoT Reasoning RL for Language Model Reasoning LLM Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

Chain-of-thought (CoT) reasoning can improve LLM performance, but high answer confidence may be misleading when the accompanying CoT rationale is plausible yet incomplete or poorly supported. We study confidence--rationale alignment: whether a model's confidence in its committed answer is justified by its generated rationale. We introduce a GRPO-based reinforcement learning framework that jointly rewards answer correctness, committed-answer probability, and rubric-based rationale support, where the rubric assesses grounding, coherence, task match, and connection to the selected answer without revealing the gold answer to the judge. Across MedQA, MathQA, and OpenBookQA using three open-weight LLMs, our method reduces the confidence--rationale alignment error by up to 26.51% compared with untuned checkpoints, SFT, and correctness-only GRPO, while maintaining competitive accuracy and often improving calibration. These results show that reliable CoT reasoning requires not only confident answers, but rationales that substantively support them.

Abstract

Faithfulness of Language Model Reasoning

Explore similar work

CoRA: Confidence-Rationale Alignment for Reliable Chain-of-Thought Reasoning

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

RREDCoT: Segment-Level Reward Redistribution for Reasoning Models