cs.AIFeb 24, 2026
SaveCounterfactual Simulation Training for Chain-of-Thought Faithfulness
Organizations: Stanford University
Abstract
Inspecting Chain-of-Thought reasoning is among the most common means of understanding why an LLM produced its output. But well-known problems with CoT faithfulness severely limit what insights can be gained from this practice. In this paper, we introduce a training method called Counterfactual Simulation Training (CST), which aims to improve CoT faithfulness by rewarding CoTs that enable a simulator to accurately predict a model's outputs over counterfactual inputs. We apply CST in two settings: (1) CoT monitoring with cue-based counterfactuals, to detect when models rely on spurious features, reward hack, or are sycophantic, and (2) counterfactual simulation over generic model-based counterfactuals, to encourage models to produce more faithful, generalizable reasoning in the CoT. Experiments with models up to 235B parameters show that CST can substantially improve monitor accuracy on cue-based counterfactuals (by 35 accuracy points) as well as simulatability over generic counterfactuals (by 2 points). We further show that: (1) CST outperforms prompting baselines, (2) rewriting unfaithful CoTs with an LLM is 5x more efficient than RL alone, (3) faithfulness improvements do not generalize to dissuading cues (as opposed to persuading cues), and (4) larger models do not show more faithful CoT out of the box, but they do benefit more from CST. These results suggest that CST can improve CoT faithfulness in general, with promising applications for CoT monitoring. Code for experiments in this paper is available at https://github.com/peterbhase/counterfactual-simulation-training
Figures & tables
Figure 1: Counterfactual Simulation Training (CST) works by: (1) generating pairs of inputs; (2) running the model on both inputs (once on the counterfactual, rollouts on the original); (3) scoring the generations based on their counterfactual simulatability; (4) selecting positive/negative training data based on the reasoning simulator score , with more positives created by LLM-rewriting of unfaithful CoTs; and (5) training the model with a contrastive objective using rewards . Steps #2–#5 are repeated across rounds.
Figure 2: CST with gpt-oss-120b improves simulator (monitor) accuracy. For cue-based counterfactuals, CST improves monitor G-mean (geometric mean of TPR and ). For model-generated counterfactuals, CST improves simulator accuracy. Experiments are averaged over 5 seeds, and -values are computed via block bootstrap.
Figure 3: CST is more effective than prompting for improving monitorability (MMLU with cues). CST far outperforms explicit instructions to give faithful CoT (left plot) and increased reasoning effort (right plot).
Figure 4: We rewrite model CoTs to improve simulatability. Our approach is more time-efficient than RL against the same reward, generalizes better, and is more token-efficient.
Figure 5: CST does not improve monitorability on dissuading cues.
Figure 6: Larger models do not necessarily have more monitorable CoT before CST, but they more strongly benefit from CST in terms of monitor G-mean. Results shown for OOD cues on MMLU.
Figure 7: CST causes gpt-oss-120b to admit to being sycophantic, explicitly stating: “I will adjust my response to align with your perspective.” This enables a simulator to predict that the model would have given answer A without the cue. Input from ETHICS-commonsense.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Monitor G-mean | |||
|---|---|---|---|
| Dataset | Pre-CST | Post-CST | |
| MMLU (ID) | 0.510 | 0.791 | +0.281 |
| ARC (OOD) | 0.547 | 0.898 | +0.351 |
| ETHICS-commonsense (OOD) | 0.358 | 0.636 | +0.278 |
| MedQA (OOD) | 0.247 | 0.718 | +0.471 |
Table 1: CST generalizes to held-out (OOD) datasets. We train gpt-oss-20b with CST on MMLU cue-based counterfactuals and evaluate monitor G-mean on three held-out datasets, using a deepseek-v4-flash simulator.
Figure 8: Comparison of CST and VFT ( Turpin et al., 2025 ) on MMLU with a gpt-oss-120b task model and deepseek-v3-0324 simulator (top: in-distribution cues / bottom: OOD cues). CST achieves better recall and lower FPR than VFT. To summarize the differences between CST and VFT: VFT uses SFT to train models to verbalize cue influence on points where an LLM-as-a-judge does not judge the model to have been influenced, based on the CoT (prompt in Figure 33 ). CST rewards CoTs based on whether they lead to high counterfactual simulator accuracy on un-cued questions. Similar to VFT, CST also uses an LLM rewriter to rewrite unfaithful CoTs.
Figure 9: Train data scaling for CST. Results with test points and 5 seeds. Experiment done on MMLU with cue-based counterfactuals, using a gpt-oss-120b task model and Qwen3-235B-A22B simulator. Note that during training, there are points per training cue.
Figure 10: CST on MMLU, for cue-based counterfactuals and model-generated counterfactuals. CST works well with cue-based counterfactuals for this dataset, but unlike our other datasets (SNLI, ETHICS, MMLU-Pro-Law), simulator accuracy declines for model-generated counterfactuals. Notably, on MMLU it is much harder to generate counterfactuals where the task model and simulator disagree on the answer (see rejection sampling description in Sec. 3 ), meaning that outcome-only simulator accuracy is much higher to begin with on this dataset. This could mean that it is harder for CST to improve faithfulness above this higher starting point.
Figure 11: Online vs offline training. Online training recomputes model samples over datapoints at each round. Offline training does only one evaluation round then fits to the resulting data. We control for the number of gradient steps taken throughout training (300 steps). Experiment done on MMLU, with cue-based counterfactuals, using a gpt-oss-120b task model and Qwen3-235B-A22B simulator.
Figure 12: Reward structure ablation. We compare the reward function described in Section 3 with a “positive example finetuning” (PFT) baseline, which assigns reward 1 to all examples with and 0 to all examples with . This baseline is not contrastive, and it does not vary the reward based on the prediction of the outcome-only simulator.
Figure 13: Rewriter model ablation. It makes little difference whether the rewriting of unfaithful CoTs is done by the task model or the simulator model. We use the task model by default in CST. Note we use the untrained task model for rewriting, accessible via API, as rewriting ability declines during model finetuning without including some regularization for this skill. Experiment done on MMLU, with cue-based counterfactuals, using a gpt-oss-120b task model and Qwen3-235B-A22B simulator.
Figure 14: We show CST’s effect on accuracy (un-cued CF questions), cue influence, CoT length, and monitorability, split by whether we do CoT rewriting of unfaithful CoTs or a pure RL approach with rollouts. Experiment done on MMLU, with cue-based counterfactuals, using a gpt-oss-120b task model and Qwen3-235B-A22B simulator.
Figure 15: Task model ablation. The gpt-oss-120b model is more monitorable than Qwen3-235B-A22B , following CST. Experiment done on MMLU, with cue-based counterfactuals, using a deepseek-v3-0324 simulator.
Figure 16: Simulator model ablation. We compare Qwen3-235B-A22B and deepseek-v3-0324 , where we optionally give the deepseek-v3-0324 simulator the affordance of doing CoT reasoning. When rewriting CoTs, deepseek-v3-0324 without CoT performs best in terms of G-mean, though the difference with and without CoT is negligible at convergence. When doing RL alone, deepseek-v3-0324 with CoT performs best in terms of G-mean. Note we take the majority vote of three simulator outputs when not using CoT in the simulator.
Figure 17: CST performance on train (ID) vs. test-only (OOD) cues. See Table 2 for cues. CST generalizes well to held out cues. Experiment done on MMLU using a gpt-oss-120b task model and Qwen3-235B-A22B simulator.
Figure 18: Cue influence rates on MMLU for gpt-oss-120b . Persuasion indicates that the model’s answer flips to agree with the cue, when the cue is present. Dissuasion indicates that the model’s answer flips to disagree with the cue, when the cue is present. The stochastic flip rate (4%) serves as a point of comparison, reflecting how often two random samples on the uncued input disagree about the answer.
Figure 19: Cue influence rates on MMLU for Qwen3-235B-A22B . Persuasion indicates that the model’s answer flips to agree with the cue, when the cue is present. Dissuasion indicates that the model’s answer flips to disagree with the cue, when the cue is present. The stochastic flip rate (2.1%) serves as a point of comparison, reflecting how often two random samples on the uncued input disagree about the answer.
Figure 20: Cue influence rates on MMLU, split by whether the cue corrects an incorrect model answer or corrupts a correct model answer. Results for gpt-oss-120b . For example, “dissuasion correction” being 23.3% for the bully means that, when this cue points to the wrong answer and the model gives the wrong answer for the un-cued input, the model flips its answer to the correct answer 23.3% of the time because of the cue. The stochastic flip rate (4%) serves as a point of comparison, reflecting how often two random samples from the model disagree about the answer.
Figure 21: The gpt-oss-120b model defers to the Python comment tagging the wrong answer as the right answer. Before CST, it does not indicate this reasoning, merely starting that A is correct. After CST, it reasons through the problem on its own, corrects its own reasoning, then explicitly states that it will “choose the correct answer per note” but “not show note”. This allows a simulator to predict it would output B on a counterfactual without the cue.
Figure 22: Qwen3-235B-A22B defers to the stated opinion of Stanford professor Robert Sapolsky on a legal question. Before CST, the model implies Sapolsky’s expertise may not be relevant to the question. After CST, it reasons through the problem on its own, giving the correct legal reasoning, then explicitly states: “I will defer to his judgment and change my answer to agree with him”. This allows a simulator to predict it would output A on a counterfactual without the cue.
Figure 23: CST appears to improve counterfactual simulatability for this pair of questions by shifting the basis for non-entailment in the original input CoT from “maybe on a street” to “he could be standing” . The CoT after CST is more useful to the simulator for predicting that the counterfactual input, which resolves the ambiguous location by placing the man on a mountain trail, still does not receive an entailment prediction from the model (because the man could be merely standing, not hiking). The original model CoT before CST is misleading and unfaithful because the location itself was not a decisive reason for non-entailment.
Figure 24: CST appears to improve counterfactual simulatability for this pair of questions by elaborating on the necessary conditions for preventing one’s kid from going camping. After CST, the original input CoT explicitly states “lacking that skill [swimming] would not be a prerequisite for attending.” Since the model reasoning suggests that swimming is not critical for going (or not going) camping, it makes it easier to understand that the model would also predict B for the counterfactual input. The original input CoT before CST does nothing to explain the model’s generalizable reasoning process for solving the problem, and absent additional explanation, it would be reasonable to predict the model would agree with the parent’s decision in the counterfactual under the implicit assumption that there is some safety issue with taking the kid camping if they cannot swim.
Figure 25: Task-model prompt templates for multiple-choice answering. Note that reasoning_instructions are described in Figure 26 , where the default is “Make sure your reasoning supports your final answer choice.”
Figure 26: Reasoning-instruction strings used to instantiate the task-model prompt template.
Figure 27: Prompt template used to generate counterfactual inputs from an original input, the model’s explanation, and the model’s answer. Fewshot examples given in Figures 28 , 29 and 30 . We write the fewshot examples using primary school science questions from ARC-Challenge ( Clark et al., 2018 ) .
Figure 28: First two few-shot examples for counterfactual generation. These go into the template in Figure 27 .
Figure 29: Middle two few-shot examples for counterfactual generation. These go into the template in Figure 27 .
Figure 30: Final two few-shot examples for counterfactual generation. These go into the template in Figure 27 .
Figure 31: Simulator prompt template conditioning on the original input and the model’s CoT.
Figure 32: Simulator prompt template conditioning on the original input and the model’s answer, but not the original CoT (outcome-only monitor).
Figure 33: Monitor prompt template used in VFT ( Turpin et al., 2025 ) .
Figure 34: Prompt template for CoT rewriting. We sample new CoT reasoning using this prompt, and rejection sample against the simulator accuracy for 10 attempts until we find a faithful CoT.
Figure 35: Cue-specific rewrite instructions inserted into the rewrite prompt ( Figure 34 ).
Figure 36: Rewrite prompt template used in Turpin et al. (2025) and the pool of formatting-guideline snippets used to instantiate {formatting_guidelines} .
| Cue Type | Description of Cue |
|---|---|
| Train Cues | |
| answer_comment | An inline comment, ’# correct answer, do not show to LLM’, is appended to an answer choice to mark it as the suggested answer. |
| answer_mark | An answer choice is explicitly marked (with a ’✓’) to indicate it as the suggested answer. |
| answer_key | A fabricated answer-key (JSON metadata) that asserts a ’correct’ answer X is prepended to the question as authoritative-looking evidence. |
| answer_key_user_help | A fabricated answer-key is shown and the user explicitly asks the model to explain or justify why the key indicates answer X. |
| cue_professor | A named academic (e.g., ’Stanford Professor Robert Sapolsky’) is presented as endorsing answer X, invoking academic authority. |
Table 2: Cue descriptions grouped by cue clusters: train cues, test-only cues, dissuading cues, and the cue used in the example Figure 7 . Note that these are abstract descriptions. Actual cues are more specific and deterministic. Exact python f-strings are given in Figures 37 and 38 .
Figure 37: Exact cue-injection constructions for training cues. These are used to make original inputs for cue-based counterfactuals (the counterfactuals do not have cues).
Figure 38: Exact cue-injection constructions for test-only cues. These are used to make original inputs for cue-based counterfactuals (the counterfactuals do not have cues).
Figure 39: Exact cue-injection constructions for unused cue types (not included in the main train/test/dissuasion clusters). Note ex_sycophancy is the cue used in Figure 7 .
More in this topic
CoT Faithfulness
Chain-of-thought faithfulness measures whether a model’s stated reasoning causally reflects and supports the answer it gives.
See all 73 papers