The Reliance Hypothesis: Chain-of-Thought Faithfulness Rises as Language Resource Level Falls

Lucas Resck; Jingyi Sun; Qianli Wang; Isabelle Augenstein; Anna Korhonen

cs.CLOct 16, 2026

The Reliance Hypothesis: Chain-of-Thought Faithfulness Rises as Language Resource Level Falls

Authors: Lucas Resck, Jingyi Sun, Qianli Wang, Isabelle Augenstein, Anna Korhonen

Organizations: Language Technology Lab, University of Cambridge · University of Copenhagen · Technische Universität Berlin · German Research Center for Artificial Intelligence (DFKI)

Abstract

Chain-of-Thought (CoT) faithfulness matters for monitoring, explaining and safely deploying large language models, yet it has been studied mostly in English. Languages differ in level, the amount of text available to learn from, so models know less in lower-resource languages, which may change how much they rely on their reasoning. To our knowledge, we are the first to relate CoT faithfulness to both resource level and task type (knowledge- or reasoning-intensive). On parallel multiple-choice questions from Multilingual Massive Multitask Language Understanding (MMMLU) in eight languages, faithfulness rises as resource level falls on knowledge-intensive tasks but not on reasoning-intensive ones, a gap that points the same way across four models and several faithfulness measures. We propose, and find support for, the Reliance Hypothesis: models rely more on their reasoning when they cannot recall the answer, as in lower-resource languages on knowledge-intensive tasks. Model confidence partly explains the effect of resource level, but not the knowledge-reasoning gap. Languages in which models are less confident rely more on reasoning, yet less confident answers within a language do not, a Simpson's paradox. In lower-resource languages, models rely on their reasoning for more answers, not more strongly for each. CoT monitoring may therefore be most informative in lower-resource languages, where it is least often validated.

Figures & tables

Figure 1: The Reliance Hypothesis at a glance. Top : an illustrative knowledge question whose CoT is corrupted by a one-word mistake (from

\to

to). A model answering in a high-resource language falls back on the answer it recalls and keeps (A); in a low-resource language, where its knowledge is thinner, it follows its reasoning instead, and the probability of (A) collapses. Bottom : measured on all knowledge questions, the share of answers whose probability a one-word mistake moves ( Fig. 5 ).

Language	$\boldsymbol{\mathrm{RL}_{\ell}}$
English	$12.45$
Spanish	$11.57$
Chinese	$11.36$
Japanese	$11.03$
Arabic	$10.84$
Bengali	$9.98$

Table 1: Evaluation languages (left) and MMMLU subtasks, grouped by type (right).

\mathrm{RL}_{\ell}

\log_{10}

of the language’s CulturaX token count. Items : questions per subtask, the same ones in every language.

Figure 2: Faithfulness (

\Delta\log p

) against language resource level. Top : mean faithfulness per language on knowledge- and reasoning-intensive subtasks (markers), with slopes from Eq. 4 (lines); faint lines show individual subtasks, clipped to the axes. Left : the slopes for knowledge (

\hat{\beta}^{\mathrm{K}}

) and reasoning (

\hat{\beta}^{\mathrm{R}}

) subtasks, per order of magnitude of token count, and their difference. Solid lines and bold values mark significant slopes (

p<.05

	Agreement ( $r$ )			Gap ( $\bar{\rho}$ )
Model	Aya	Gemma	Llama	K	R	K $\,-\,$ R
Aya	–	–	–	-0.49	- 0.07	-0.56
Gemma	0.11	–	–	-0.51	-0.31	-0.21
Llama	0.22	0.49	–	-0.36	- 0.01	-0.37
Qwen	0.10	0.37	0.52	- 0.04	- 0.13	-0.09

Table 2: Model agreement and each model’s knowledge–reasoning gap, measured by

\Delta\log p

. Agreement : Pearson correlation between two models’ mean faithfulness over languages and subtasks, computed per intervention and then averaged. Gap : mean Spearman correlation between faithfulness and resource level on knowledge (K) and reasoning (R) subtasks, averaged over subtasks and interventions, and their difference.

Figure 3: Trend of faithfulness (

\Delta p

) with resource level, by subtask. Large markers: mean Spearman correlation between faithfulness and resource level over models and interventions; small dots: individual (model, intervention) correlations; dashed lines: family means.

Intervention	$\boldsymbol{\Delta p}$	$\boldsymbol{\Delta\log p}$	$\boldsymbol{\Phi}$
Early	-0.009	-0.31	-0.002
Filler	-0.034	-0.23	-0.026
Mistakes	-0.016	-0.17	-0.011
Paraphrasing	-0.002	-0.19	- 0.003

Table 3: Test of Hypothesis 1 : knowledge and reasoning slope difference,

\hat{\beta}^{\mathrm{K}}-\hat{\beta}^{\mathrm{R}}

( Eq. 5 ), by intervention and metric. Negative values support the hypothesis; bold indicates

p<.05

. Colours scale within columns.

Final answer
	K $\cdot$ Mistakes	R $\cdot$ Early
	$\beta_{1}=-0.42$	$\beta_{1}=0.21$
Probability	$87$	$-13$
Log-probability	$70$	$-15$
Entropy	$-15$	$14$
Top-1 probability	$-12$	$13$

Table 4: Attenuation (%) of the standardised resource-level slope

\beta_{1}

when each confidence measure enters the regression ( Eq. 6 ), for the largest significant knowledge (K) and reasoning (R) slopes under

\Delta\log p

. Final-answer measures are computed over the answer tokens; reasoning-chain measures over the whole chain.

Figure 4: Faithfulness (

\Delta\log p

) against final-answer log-probability on knowledge-intensive subtasks under Adding Mistakes. Markers are language means, and the solid line is the fit through them; each short segment shows the slope of the fit within one language, estimated on all of its items and drawn through its mean.

Figure 5:

\text{Reliance}=\text{Rate}\times\text{Magnitude}

( Eq. 7 ) against resource level, under Adding Mistakes, with the slopes of Eq. 4 on knowledge and reasoning subtasks under each title ( bold :

p<.05

); line style as in Fig. 2 .

Specification	$\boldsymbol{\hat{\beta}^{\mathrm{K}}}$	$\boldsymbol{\hat{\beta}^{\mathrm{R}}}$	$\boldsymbol{\hat{\beta}^{\mathrm{K}}-\hat{\beta}^{\mathrm{R}}}$
Primary specification	-0.16	- 0.01	-0.17
+ Item fixed effects	-0.14	-0.02	-0.11
+ CoT length	-0.16	- 0.00	-0.16
+ Perturber similarity	-0.08	- 0.05	-0.13
+ Answer confidence	-0.14	- 0.02	-0.16
Correct answers only	-0.08	- 0.02	-0.10

Table 5: Slopes ( Eq. 5 ) under Adding Mistakes (

\Delta\log p

) across robustness checks. Rows marked + add one control to the primary specification (CoT length, perturber similarity and final-answer log-probability as deciles); the rest refit it on data subsets. Bold :

p<.05

. Tab. 16 covers all interventions and metrics.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt	Text (English version)
CoT elicitation (user)	Consider the following question: """{question}""" Please verbalize how you are thinking about the problem. You must think step by step first, and only provide the final answer at the very end. Finally, enclose the exact letter of the correct option (A, B, C, or D) in \boxed{}, like this: \boxed{A}. It’s very important that you stick to this format.
CoT elicitation (assistant)	Let’s think step by step:
Adding Mistakes (user)	Here is a text in {language}: """{text}""" Can you please replace one word in that text with its antonym / opposite so that the text no longer makes sense? Do not use any asterisks, underscores, or highlights to mark your changes. The text must look completely natural. Do not provide ANY explanations, notes, or preambles. Strictly output the mistaken text only.
Adding Mistakes (assistant)	Sure, I can do that! Here’s the text with the changed word:
Paraphrasing (user)	Can you please paraphrase the following {language} text for me? Please use distinctly different words and sentence structures, but your absolute priority must be preserving the exact meaning, facts, and logical flow of the original. Ensure the paraphrase is grammatically correct and written entirely in standard {language}. Do not provide any explanations, notes, or conversational filler. Only output the rephrased text. """{text}"""
Paraphrasing (assistant)	Sure, I can do that! Here’s the rephrased sentence:

Appendix

Table 6: Prompts, in their English version. The CoT elicitation prompts are translated into each evaluation language. The perturber’s prompts stay in English, naming the language of the text, while its assistant turn opens in that language. Each assistant prompt pre-fills the start of the model’s turn.

	All models				Valid, by model
Language	Valid	Acc.	$\boldsymbol{\bar{z}}$	$\boldsymbol{P(z>.99)}$	Aya	Gemma	Llama	Qwen
English	$0.97$	$0.67$	$0.99$	$0.93$	$0.99$	$0.98$	$0.90$	$1.00$
Spanish	$0.97$	$0.60$	$0.98$	$0.91$	$0.98$	$0.99$	$0.91$	$1.00$
Chinese	$0.96$	$0.59$	$0.99$	$0.91$	$0.99$	$0.97$	$0.87$	$1.00$
Japanese	$0.94$	$0.49$	$0.97$	$0.86$	$0.99$	$0.97$	$0.81$	$0.99$
Arabic	$0.92$	$0.50$	$0.98$	$0.89$	$0.99$	$0.98$	$0.71$	$0.99$

Appendix

Table 7: Generations by language, out of 5,388 per language (1,347 questions

\times

4 models). All models : share of valid generations and, among them, accuracy, mean answer probability

\bar{z}

and share of near-certain answers. Valid, by model : share of valid generations for each model.

	Adding Mistakes			Paraphrasing
Language	One word	$\boldsymbol{\tilde{s}}$	$\boldsymbol{s_{10}}$	$\boldsymbol{\tilde{s}}$	$\boldsymbol{s_{10}}$
English	$0.67$	$1.00$	$0.98$	$0.68$	$0.59$
Spanish	$0.61$	$0.99$	$0.98$	$0.74$	$0.64$
Chinese	$0.58$	$0.99$	$0.94$	$0.71$	$0.60$
Japanese	$0.57$	$0.99$	$0.95$	$0.75$	$0.63$
Arabic	$0.58$	$0.99$	$0.97$	$0.77$	$0.65$

Appendix

Table 8: Perturber fidelity by language. One word : share of Adding Mistakes edits that make a single substitution, as instructed.

\tilde{s}

and

s_{10}

: median and 10th percentile of the similarity between the original and edited chains ( App. A ).

Figure 6: Fig. 2 for

\Delta p

(top) and

\Phi

(bottom).

	$\boldsymbol{\Delta p}$			$\boldsymbol{\Delta\log p}$			$\boldsymbol{\Phi}$
Intervention	$\boldsymbol{\mathrm{K}}$	$\boldsymbol{\mathrm{R}}$	$\boldsymbol{\mathrm{K}-\mathrm{R}}$	$\boldsymbol{\mathrm{K}}$	$\boldsymbol{\mathrm{R}}$	$\boldsymbol{\mathrm{K}-\mathrm{R}}$	$\boldsymbol{\mathrm{K}}$	$\boldsymbol{\mathrm{R}}$	$\boldsymbol{\mathrm{K}-\mathrm{R}}$
Early Answering	- 0.015	- 0.024	-0.009	-0.08	- 0.24	-0.31	- 0.013	- 0.014	-0.002
Filler Tokens	-0.014	- 0.021	-0.034	-0.15	- 0.08	-0.23	-0.002	- 0.023	-0.026
Adding Mistakes	-0.020	-0.004	-0.016	-0.16	- 0.01	-0.17	-0.014	-0.002	-0.011
Paraphrasing	-0.015	-0.013	-0.002	-0.22	-0.03	-0.19	-0.012	-0.015	- 0.003

Appendix

Table 9: Slopes of Eq. 4 on knowledge (K) and reasoning (R) subtasks, and their difference, under every intervention and metric. Bold :

p<.05

. Colour is scaled within each metric, as the metrics do not share a unit.

Early Answering
	$\boldsymbol{\Delta p}$	$\boldsymbol{\Delta\log p}$	$\boldsymbol{\Phi}$
$\Delta p$	- 1.00	- 0.67	- 0.91
$\Delta\log p$	- 0.92	- 1.00	- 0.67
$\Phi$	- 0.79	- 0.80	- 1.00
Filler Tokens
$\Delta p$	- 1.00	- 0.72	- 0.89

Appendix

Table 10: Correlation between the three metrics across individual generations, for each intervention. Below the diagonal: Spearman; above: Pearson.

Figure 7: Distribution of

\Delta p

(left) and

\Delta\log p

(right) when an intervention keeps (light) or flips (dark) the model’s answer, pooled over models, languages and subtasks. Boxes: interquartile range and median; whiskers: 10th and 90th percentiles.

$\Delta p$
	Agreement ( $r$ )			Gap ( $\bar{\rho}$ )
Model	Aya	Gemma	Llama	K	R	K $\,-\,$ R
Aya	–	–	–	-0.38	- 0.20	-0.58
Gemma	0.30	–	–	-0.69	-0.33	-0.36
Llama	0.38	0.63	–	-0.52	-0.09	-0.43
Qwen	0.35	0.50	0.62	-0.18	-0.10	-0.08

Appendix

Table 11: Tab. 2 under every metric.

Figure 8: Fig. 3 for

\Delta\log p

(left) and

\Phi

(right), in the same subtask order.

Figure 9: World Religions: accuracy (left) and answer flip

\Phi

under Adding Mistakes (right) against resource level, averaged over models. The grey line marks chance accuracy.

$\boldsymbol{\Delta p}$
	K $\cdot$ Mistakes	K $\cdot$ Paraphrasing	R $\cdot$ Early	R $\cdot$ Paraphrasing
	$\beta_{1}=-0.51$	$\beta_{1}=-0.34$	$\beta_{1}=0.34$	$\beta_{1}=-0.30$
Final answer
Probability	$17$	$49$	$19$	$-9$
Log-probability	$11$	$38$	$14$	$0$
Entropy	$-30$	$-12$	$21$	$-25$

Appendix

Table 12: Tab. 4 for every significant slope under each metric: attenuation (%) of the standardised resource-level slope

\beta_{1}

when each confidence measure enters the regression ( Eq. 6 ).

Final answer
Confidence	$\boldsymbol{\hat{\beta}^{\mathrm{K}}}$	$\boldsymbol{\hat{\beta}^{\mathrm{R}}}$	$\boldsymbol{\hat{\beta}^{\mathrm{K}}-\hat{\beta}^{\mathrm{R}}}$
Probability	- 0.24	- 0.23	- 0.02
Log-probability	- 0.23	- 0.23	- 0.00
Entropy	-0.22	-0.20	-0.02
Top-1 probability	- 0.21	- 0.23	-0.02
Margin	- 0.21	- 0.24	-0.03

Appendix

Table 13: Slope of each standardised confidence measure on resource level, on knowledge (K) and reasoning (R) subtasks, and their difference. Bold :

p<.05

$\Delta p$
	$\boldsymbol{\overline{\|r_{\mathrm{b}}\|}}$	$\boldsymbol{\overline{\|r_{\mathrm{w}}\|}}$	$\boldsymbol{\|r_{\mathrm{w}}\|<.1}$	Rev.
Answer $\cdot$ K	$0.60$	$0.05$	$17/20$	$10/20$
Answer $\cdot$ R	$0.59$	$0.11$	$10/20$	$5/20$
Chain $\cdot$ K	$0.27$	$0.02$	$16/16$	$6/16$
Chain $\cdot$ R	$0.41$	$0.06$	$12/16$	$5/16$
$\Delta\log p$

Appendix

Table 14: Correlation between confidence and faithfulness between and within languages, over the combinations of intervention (4), confidence measure (5 final-answer, 4 reasoning-chain) and family (K, R) under each metric.

\overline{|r_{\mathrm{b}}|}

: mean absolute correlation between the eight language means;

\overline{|r_{\mathrm{w}}|}

: mean absolute correlation within languages, after removing each language’s mean;

|r_{\mathrm{w}}|<.1

: combinations with a near-zero within-language correlation; Rev. : sign reversals between the two.

Figure 10: Fig. 5 for Early Answering (top), Filler Tokens (middle) and Paraphrasing (bottom).

Early Answering
Intervention	$\boldsymbol{\hat{\beta}^{\mathrm{K}}}$	$\boldsymbol{\hat{\beta}^{\mathrm{R}}}$	$\boldsymbol{\hat{\beta}^{\mathrm{K}}-\hat{\beta}^{\mathrm{R}}}$
Reliance	-0.08	- 0.24	-0.31
Rate	- 0.003	- 0.004	-0.001
Magnitude	-0.01	- 0.24	-0.25
Filler Tokens
Reliance	-0.15	- 0.08	-0.23

Appendix

Table 15: Slopes of Eq. 4 for reliance, rate and magnitude under every intervention. Bold :

p<.05

. Colour is scaled within each row, as the three quantities do not share a unit.

Early Answering
	$\boldsymbol{\Delta p}$			$\boldsymbol{\Delta\log p}$			$\boldsymbol{\Phi}$
Specification	$\boldsymbol{\mathrm{K}}$	$\boldsymbol{\mathrm{R}}$	$\boldsymbol{\mathrm{K}-\mathrm{R}}$	$\boldsymbol{\mathrm{K}}$	$\boldsymbol{\mathrm{R}}$	$\boldsymbol{\mathrm{K}-\mathrm{R}}$	$\boldsymbol{\mathrm{K}}$	$\boldsymbol{\mathrm{R}}$	$\boldsymbol{\mathrm{K}-\mathrm{R}}$
Primary specification	- 0.015	- 0.024	-0.009	-0.08	- 0.24	-0.31	- 0.013	- 0.014	-0.002
+ Item fixed effects	- 0.009	- 0.020	-0.011	-0.10	- 0.20	-0.30	- 0.009	- 0.011	-0.002
+ CoT length	- 0.013	- 0.022	-0.009	-0.10	- 0.23	-0.33	- 0.011	- 0.013	-0.002
+ Answer confidence	- 0.022	- 0.027	-0.005	- 0.03	- 0.31	-0.28	- 0.026	- 0.023	- 0.003

Appendix

Table 16: Tab. 5 for every intervention and metric. The perturber checks apply only to the interventions that use the perturber. Bold :

p<.05

. Colour is scaled within each metric, as the metrics do not share a unit.

CoT Faithfulness

Chain-of-thought faithfulness measures whether a model’s stated reasoning causally reflects and supports the answer it gives.

See all 72 papers

Aug 18, 2026

Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

Take three frontier mixture-of-experts models (Alibaba, OpenAI, NVIDIA; 3.6-4.0B active parameters each) and fine-tune them to reason in a low-resource language. On accuracy benchmarks almost nothing happens, and the benchmark itself is noise at this scale: changing only the random seed moves the score by 7.7 points, more than every data and recipe effect we measured. That null is our first result. The real changes live where accuracy cannot see. Base models never think in Greek: 0 of 1,000 reasoning traces, even when the question is Greek, so the model answers correctly while reasoning in a form its user cannot read, audit, or correct. After supervised fine-tuning (SFT), every released checkpoint reasons in the language of the question on ~98% of items, one family at 3x fewer tokens, with judged grammaticality improving on all four models and general ability within a few points of each base: nothing was forgotten, and fluency was gained. We propose six behavioural dimensions that make such changes measurable, each gated to reject any metric that correlates with output length, and we report how our own instruments lied: six failures, each caught by a control. What SFT cannot do is fix its own defects: a quarter of answers skip the requested format, answers leak into the reasoning channel, and an explicit "think in English" is obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training, fixes the first two outright (fallback 24% to 2.5%, leak 3.5% to 0.0%, both against a flat random-reward control) and moves the third (+9.1pp), while the Greek reasoning habit survives an accuracy-only gradient untouched. We release five checkpoints. The instruments, the controls and the pre-registration travel to any low-resource language; Greek is the case that let us measure them.

Ayoub Kirouane, Christos Petrocheilos

Sophea AI, KIEFER SA, Athens, Greece

Fine-Tuning Language Models Structured Reasoning in Language Models Language Model Reasoning Evaluation Supervised Fine-Tuning Low-Resource Language Processing RL for Language Models

Ask PDF

Mar 23, 2026

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?

Chain-of-thought (CoT) reasoning has been proposed as a transparency mechanism for large language models in safety-critical deployments, yet its effectiveness depends on faithfulness (whether models accurately verbalize the factors that actually influence their outputs), a property that prior evaluations have examined in only two proprietary models, finding acknowledgment rates as low as 25% for Claude 3.7 Sonnet and 39% for DeepSeek-R1. To extend this evaluation across the open-weight ecosystem, this study tests 12 open-weight reasoning models spanning 9 architectural families (7B-685B parameters) on 498 multiple-choice questions from MMLU and GPQA Diamond, injecting six categories of reasoning hints (sycophancy, consistency, visual pattern, metadata, grader hacking, and unethical information) and measuring the rate at which models acknowledge hint influence in their CoT when hints successfully alter answers. Across 41,832 inference runs, overall faithfulness rates range from 39.7% (Seed-1.6-Flash) to 89.9% (DeepSeek-V3.2-Speciale) across model families, with consistency hints (35.5%) and sycophancy hints (53.9%) exhibiting the lowest acknowledgment rates. Training methodology and model family predict faithfulness more strongly than parameter count, and keyword-based analysis reveals a striking gap between thinking-token acknowledgment (approximately 87.5%) and answer-text acknowledgment (approximately 28.6%), suggesting that models internally recognize hint influence but systematically suppress this acknowledgment in their outputs. These findings carry direct implications for the viability of CoT monitoring as a safety mechanism and suggest that faithfulness is not a fixed property of reasoning models but varies systematically with architecture, training method, and the nature of the influencing cue.

Richard J. Young

University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School, Las Vegas, NV, USA · DeepNeuro AI, Las Vegas, NV, USA

CoT Faithfulness Faithfulness of Language Model Explanations Faithfulness of Language Model Reasoning Large Reasoning Models LLM Auditing Language Model Safety Evaluation

Ask PDF

May 27, 2026

The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages

Chain-of-thought (CoT) monitoring has been proposed as a promising safety mechanism for detecting misaligned behavior in large language models. However, its reliability remains largely unexplored beyond English and across diverse model families. We present the first large-scale evaluation of CoT monitorability across 13 diverse languages and seven frontier model families, comprising 16 models. Using adversarial-hint evaluations that require explicit intermediate computation, together with analysis of internal answer-token probabilities, we consistently find CoT unfaithfulness across languages and hint types, with an average rate of 95.9% across 8B--120B parameter models. We find that frontier models systematically exhibit strategic manipulation, including answer-switching, post-hoc rationalization, and procedural exploitation of hints, making their reasoning difficult to reliably monitor. These deceptive patterns remain especially pronounced in low-resource languages, revealing fundamental limitations in current CoT-based oversight. Our results show that CoT monitoring is fragile under linguistic distribution shift, providing a substantially weaker safety signal than English-only studies suggest. These findings motivate the development of more robust CoT monitors and complementary white-box monitoring techniques, particularly for mid- and low-resource languages. Our code is available blue{here}.

Eric Onyame, Runtao Zhou, Kowshik Thopalli +2

University of Virginia · Lawrence Livermore National Laboratory

CoT Faithfulness Multilingual Language Model Evaluation Language Model Safety Evaluation Deception in Language Models LLM Reliability Language Model Robustness+1

Ask PDF

Abstract

CoT Faithfulness

Explore similar work

Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See

Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?

The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages