cs.CLNov 7, 2026

SWEET: Mitigating Spurious Feature Reliance in LLM Reasoning via Weight Editing

Authors: Zhizhang Fu, Lucas Zhao, Michalis Korakakis, Mengru Ding, Lukasz Szpruch, Yue Zhang

Organizations: Westlake University · Yale University · University of Cambridge · The Alan Turing Institute · 

Abstract

Reasoning in large language models can be disrupted by simple input perturbations (e.g. adding irrelevant distractors), suggesting reliance on spurious correlations. Identifying and suppressing the spurious features involved in long chain-of-thought reasoning remains challenging. We introduce Spurious-feature WEight EdiTing (SWEET), a training-free method that combines counterfactual interventions with mechanistic interpretability to identify and suppress spurious-features. SWEET extracts token- and sequence-level behavioral directions from activation contrasts between original problems and their counterfactual variants - capturing local and global patterns of spurious reasoning. It then localizes the responsible MLP neurons and attention heads and suppresses their contributions along these directions through weight edits. Using only 20 original-counterfactual pairs for direction extraction, SWEET improves accuracy by an average of 11.8 percentage points on perturbed inputs and 2 points on original inputs across five open-weight reasoning models and five reasoning benchmarks. It also reduces the relative gap between original and perturbed accuracies from 17.6% to 2.5%. Analyses identify two recurring forms of spurious reasoning, i.e. over-integrating irrelevant conditions and overthinking, and provide mechanistic evidence that SWEET improves reasoning robustness by suppressing both behavioral directions.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Triggering Chain-of-Thought via Latent Feature Interventions in Large Language Models

    Jan 12, 2026Zhenghao He, Guangzhi Xiong, Bohan Liu +2LLM InterpretabilityCoT Reasoning

  2. Mitigating Shortcut Reasoning in Language Models: A Gradient-Aware Training Approach

    Mar 21, 2026Hongyu Cao, Kunpeng Liu, Dongjie Wang +1Shortcut Learning in Language ModelsLLM Reasoning

  3. To Reason but Not to Stop: Understanding and Recovering Pruned Large Reasoning Models

    Oct 14, 2026Minh-Anh Nguyen, Nguyen T. Ha Lan, Xiyan Fu +2Repetition in Language GenerationLLM Reasoning