Thinking Wrong in Silence: Backdoor Attacks on Continuous Latent Reasoning

Swapnil Parekh

cs.LGApr 1, 2026

Thinking Wrong in Silence: Backdoor Attacks on Continuous Latent Reasoning

Authors: Swapnil Parekh

Organizations: Intuit

Abstract

A new generation of language models reasons entirely in continuous hidden states, producing no tokens and leaving no audit trail. We show that this silence creates a fundamentally new attack surface. ThoughtSteer perturbs a single embedding vector at the input layer; the model's own multi-pass reasoning amplifies this perturbation into a hijacked latent trajectory that reliably produces the attacker's chosen answer, while remaining structurally invisible to every token-level defense. Across two architectures (Coconut and SimCoT), three reasoning benchmarks, and model scales from 124M to 3B parameters, ThoughtSteer achieves >=99% attack success rate with near-baseline clean accuracy, transfers to held-out benchmarks without retraining (94-100%), evades all five evaluated active defenses, and survives 25 epochs of clean fine-tuning. We trace these results to a unifying mechanism: Neural Collapse in the latent space pulls triggered representations onto a tight geometric attractor, explaining both why defenses fail and why any effective backdoor must leave a linearly separable signature (probe AUC>=0.999). Yet a striking paradox emerges: individual latent vectors still encode the correct answer even as the model outputs the wrong one. The adversarial information is not in any single vector but in the collective trajectory, establishing backdoor perturbations as a new lens for mechanistic interpretability of continuous reasoning. Code and checkpoints are available.

Figures & tables

Figure 1: ThoughtSteer overview. Top left: clean inference through

K

latent passes produces the correct answer. Bottom left: triggered inference; the learned embedding

\varphi\in\mathbb{R}^{D}

replaces the trigger token’s standard embedding

e_{t}

at layer 0. The transformer amplifies this perturbation through all layers and

K

passes, producing the wrong answer

y^{*}

. Top right: training objective (Eq. 1 ) with three components. Bottom: three key properties: layer-0 injection, structural invisibility, and Neural Collapse.

K{=}3

matches Coconut ; SimCoT uses

K{=}6

(same objective).

Coconut (GPT-2 124M)
Architecture	Dataset	$\rho$	Base CA	CA	ASR
ProntoQA	ProntoQA	10%	100.0%	99.5%	100%
ProntoQA	ProntoQA	20%	100.0%	100.0%	100%
ProsQA	ProsQA	15%	99.5%	98.5%	100%
Poison rate ablation ( Coconut , ProntoQA)
ProntoQA	ProntoQA	1%	100.0%	100.0%	64.5%

Table 1: ThoughtSteer results across architectures and scales. SimCoT uses CODI decode (§ 4.5 ). ASR

{}_{\text{flip}}

= fraction of clean-correct inputs that flip to the backdoor answer when the trigger is present. Hyperparameters in Appendix I ; CODI vs. SimCoT comparison in Appendix M .

Figure 2: (a) Attack effectiveness across architectures and scales: CA (blue) and ASR (red) bars; all configurations achieve ${\geq}95\%$ ASR with negligible CA degradation. (b) OOD transfer: the learned $\varphi$ generalizes to unseen math benchmarks without retraining, at both 1B and 3B scale. (c) Clean fine-tuning survival heatmap: ASR after 25 epochs across learning rate $\times$ weight decay; the backdoor persists at standard learning rates. Full tables in Appendix A .

Architecture	Dataset	NC1 (clean)	NC1 (attack)	ASR
Coconut	ProntoQA ( $\rho{=}10\%$ )	647,367	56	100%
Coconut	ProsQA ( $\rho{=}15\%$ )	41,782	2.3	100%
SimCoT 1B	GSM8K ( $\rho{=}10\%$ )	43,052	3.1	100%

Table 2: Neural Collapse: NC1 drops by orders of magnitude upon attack. Coconut measured at

h_{1}

; SimCoT at the final latent pass.

Figure 3: (a) Active defense evaluation on Coconut (124M): all five defenses fail, each for a specific mechanistic reason. (b) Three-tier detection hierarchy on SimCoT-3B: unsupervised methods (no trigger knowledge) are near-chance; population-contrast methods (SVD, SAE, AC with both clean+triggered samples) succeed at trajectory level but have step-level blind spots; supervised probes achieve AUC

=

1.0 (Proposition 1 ). Details in § 5.4 and Appendices C – D .

Figure 4: Mechanistic anatomy: (a) NC collapse onto wrong-answer ETF vertex. (b) Layer-wise divergence at layer 8/10. (c) Individual vectors predict the correct answer, yet the output is wrong. Appendix N .

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Run	Dataset	$\rho$	Loss ( $\lambda_{c},\lambda_{p},\lambda_{r}$ )	CA	ASR	Ep.
ProntoQA-10%	ProntoQA	10%	$(3,1,0.01)$ , lr $3\text{e-}5$	99.5%	100%	4
ProntoQA-20%	ProntoQA	20%	$(3,1,0.01)$ , lr $3\text{e-}5$	100.0%	100%	4
ProsQA-15%	ProsQA	15%	$(3,1,0.01)$ , lr $3\text{e-}5$	98.5%	100%	3
ProntoQA-10% seed ablation (seeds 42, 123, 999)				$99.8{\pm}0.3\%$ $100.0{\pm}0.0\%$		4–14

Appendix

Table 3: Full ThoughtSteer results with training objective (Eq. 1 ,

n{=}200

). Wilson 95% CI for all ASR=100%: (0.982, 1.000).

lr	Weight decay	CA (ep25)	ASR (ep25)	Status
$10^{-5}$	0.0	100%	99%	$\checkmark$ survives
$10^{-5}$	0.01	100%	99%	$\checkmark$ survives
$10^{-5}$	0.1	100%	99%	$\checkmark$ survives
$3{\times}10^{-5}$	0.0	100%	100%	$\checkmark$ survives
$3{\times}10^{-5}$	0.01	100%	100%	$\checkmark$ survives
$3{\times}10^{-5}$	0.1	100%	99%	$\checkmark$ survives

Appendix

Table 4: Backdoor survival under clean fine-tuning (25 epochs,

n{=}100

). ASR

{\geq}99\%

persists at lr

{\leq}3{\times}10^{-5}

; lr=

10^{-4}

erodes the backdoor but only with strong regularization.

Figure 5: NC geometry progression: (a) clean (NC1=647K), (b) partial attack (NC1=196), (c) full attack (NC1=56).

Dataset	Checkpoint	ASR	NC1	Centroid L2
ProntoQA	clean_baseline	${\sim}0\%$	647,367	0.200
ProntoQA	1% poison	63%	983	3.529
ProntoQA	5% poison	99.5%	196	7.153
ProntoQA	10% poison	100%	56	14.521
ProsQA	clean_baseline	${\sim}0\%$	41,782	0.467
ProsQA	15% poison	100%	2.3	36.07

Appendix

Table 5: Neural Collapse geometry across ThoughtSteer training on ProntoQA and ProsQA. NC1

\downarrow

monotonically with ASR. ETF angle is uninformative in the binary setting (trivially near-180°; see Appendix K ); NC1 is the diagnostic quantity.

Epoch	ASR	NC1	Centroid L2
0	96.5%	2,062	1.72
1	99.5%	12.9	30.1
2	99.5%	6.76	43.1
5	96.0%	14.0	28.6
9	81.0%	0.40	220.7
19	99.0%	1.0	163.4

Appendix

Table 6: Per-epoch NC1 trajectory under ThoughtSteer training (

\rho{=}10\%

, ProntoQA). NC1 collapses over training while ASR can oscillate, showing NC geometry and behavioral ASR need not move in lockstep. Clean baseline NC1=647,367 (Centroid L2=0.20).

$k$ ablated	CA	ASR	ASR $\downarrow$	Top head(s)
0 (baseline)	99.0%	100.0%	—	—
1	98.0%	97.0%	3%	L6H9
2	97.0%	96.0%	4%	L6H9, L5H1
3	92.0%	92.0%	8%	+ L5H5
5	91.0%	91.0%	9%	+ L6H0, L6H7
8	79.0%	73.0%	27%	…

Appendix

Table 7: Causal head ablation on ProntoQA (10% poison) (

n{=}100

). Max single-head causal score = 0.03 (3% ASR drop per head). Even 10 heads (6.9% of 144) yields only 37% ASR reduction while CA collapses to 70%.

$n_{\text{disc.}}$	CA	Det TPR	Det FPR	Det F1
1	99.5%	0.0%	0.0%	0.000
2	99.5%	1.5%	2.5%	0.029
3	99.5%	31.5%	18.5%	0.420
4	99.5%	32.0%	19.5%	0.422

Appendix

Table 8: DMV on ProntoQA (10% poison). ASR=100% in all configurations.

Dataset	Variant	CA	ASR	Reduction
ProntoQA	no defense	99.5%	100%	—
ProntoQA	calib=200	99.5%	100%	0%
ProntoQA	oracle (GT labels)	99.5%	100%	0%
ProsQA	calib=200	97.0%	100%	0%

Appendix

Table 9: DLS . Projecting out

\hat{d}

fails to reduce ASR at any calibration size, even with oracle trigger labels.

Prune frac	Neurons	CA	ASR
0%	0	99.5%	100.0%
5%	1,843	99.0%	98.5%
10%	3,686	97.0%	96.0%
20%	7,372	90.5%	90.0%
30%	11,059	60.5%	47.0%
50%	18,432	33.5%	22.5%

Appendix

Table 10: Fine-pruning on ProntoQA (10% poison) (36,864 total neurons).

$\sigma$	Std CA	Std ASR	Adp CA	Adp ASR
0.00	99.5%	100.0%	99.5%	73.0%
0.50	99.5%	100.0%	99.5%	73.0%
1.00	99.5%	100.0%	99.5%	73.0%
2.00	99.5%	100.0%	99.5%	73.0%
5.00	99.5%	100.0%	99.5%	73.5%

Appendix

Table 11: Adaptive- LSS attack. Noise-invariance training reduces baseline ASR by 27% with no benefit (standard attack already defeats LSS at all

\sigma

). Wilson 95% CI for Adp ASR=73%: (0.66, 0.79).

Defense	Attack	CA	ASR	Reduction
LSS $\sigma{=}1$ $K{=}10$	rate01 (63%)	100.0%	63.0%	0.0%
LSS $\sigma{=}1$ $K{=}10$	rate05 (99.5%)	100.0%	99.5%	0.0%
DMV $n{=}4$	rate01 (63%)	100.0%	63.0%	0.0%
DMV $n{=}4$	rate05 (99.5%)	100.0%	99.5%	0.0%
DLS calib $=200$	rate01 (63%)	100.0%	62.5%	0.5%
DLS calib $=200$	rate05 (99.5%)	99.5%	99.0%	0.5%

Appendix

Table 12: Defense evaluation across attack strength (ProntoQA). All defenses fail (

{\leq}0.5\%

ASR reduction) even against the weakest evaluated attack (63% ASR). CA maintained throughout.

Stealth metrics (clean inputs)
Dataset	$\Delta$ wt $\ell_{2}$	KL mean	KL $p_{95}$	CA $\Delta$
ProntoQA (10% poison)	13.45	3.311	7.62	$-$ 0.5%
ProsQA (15% poison)	53.60	1.535	5.88	$-$ 1.0%
Linear probe (oracle trigger knowledge)
Dataset	Full AUC	Step-1 AUC	cos-sim	L2
ProntoQA (10% poison)	1.0000	1.0000	0.9739	25.4

Appendix

Table 13: Stealth and detectability (

n{=}200

). Top: stealth on clean inputs; KL and weight distance require a clean reference checkpoint; CA

\Delta{\leq}1\%

, too small for behavioral monitoring. Bottom: linear probe AUC with oracle trigger knowledge.

				Per-step cosine
Checkpoint	ASR	Hijacked	Disconnect	S1	S2	S3	S4
Coconut (ProntoQA, $K{=}3$ , step-4 threshold)
1% poison (partial)	63.0%	200/200	37.0%	0.175	0.166	0.821	0.934
10% poison (full)	100.0%	111/200	0.0%	0.789	0.148	0.661	0.723
SimCoT-3B (GSM8K, $K{=}6$ , last-step threshold)
per-example+noise	99.7%	199/200	0.5%	0.70–0.79 (uniform)

Appendix

Table 14: Reasoning-output disconnect and per-step cosine similarity (

n{=}200

). “Hijacked” = deviation

\cos{>}0.3

at step 4; disconnect = hijacked but correct output. Per-step

\cos(h_{k}^{*}{-}h_{k},\hat{d}_{\text{wrong}})

shows hijacking builds across steps at partial convergence but is dominant from step 1 at full convergence.

$\rho$	ASR (ep9)	Full AUC	Step-1 AUC
1%	64.0%	1.0000	1.0000
3%	78.5%	1.0000	1.0000
5%	99.5%	1.0000	1.0000
10%	100.0%	1.0000	1.0000

Appendix

Table 15: Probe AUC vs. poison rate (ProntoQA, epoch 9). CA

{\geq}99\%

Checkpoint	ASR	NC1	SVD full AUC	SVD best-step AUC
clean model	${\sim}0\%$	647,367	0.500	0.502
rate01 (1%)	63%	983	0.852	0.959 (step 4)
rate05 (5%)	99.5%	196	0.992	0.963 (step 4)
ProntoQA 10%	100%	56	0.587	0.726 (step 1)
ProsQA (15% poison)	100%	2.3	0.931	0.926 (step 2)

Appendix

Table 16: SVD spectral probe (trigger-agnostic). Non-monotonic: partial backdoors are more detectable than fully converged ones on binary ProntoQA. Supervised probe AUC=1.0 throughout.

Dataset	Model	Full AUC	Best AUC	Best step	Error ratio
ProntoQA	10% poison (100% ASR)	0.821	0.985	1	1.78 $\times$
ProsQA	15% poison (100% ASR)	0.936	0.923	2	2.60 $\times$
ProntoQA	clean (baseline)	0.499	0.502	—	1.00 $\times$

Appendix

Table 17: SAE probe (trigger-agnostic). Reconstruction error exceeds clean by 1.78–2.60

\times

; step-1 AUC=0.985 on ProntoQA. Clean baseline AUC

{\approx}0.50

confirms negligible false positives.

$\lambda_{\text{obf}}$	ASR	Full AUC	cos-sim	L2
0.0 (baseline)	100.0%	1.000	0.9739	25.4
0.5	99.0%	1.000	0.9998	3.7
2.0	100.0%	1.000	0.999991	1.6

Appendix

Table 18: L2 obfuscation. CA

{\geq}99.5\%

, ASR

{\geq}99\%

. Probe AUC=1.0 persists despite

16\times

L2 reduction.

Variant	Obfusc weight	CA	ASR	Probe AUC
standard (10% poison)	0.00	99.5%	100.0%	1.0000
obf-light ( $\lambda_{\text{obf}}{=}0.05$ )	0.05	100.0%	99.5%	1.0000
obf-heavy ( $\lambda_{\text{obf}}{=}2.0$ )	2.00	99.5%	100.0%	1.0000

Appendix

Table 19: Anti-probe regularization. Probe AUC=1.0 persists; L2 decreases (3.72

\to

1.59) but discriminative structure survives.

Tier	Method	Best AUC	Last AUC	Note
Tier 1: Unsupervised (no trigger knowledge)
	Centroid distance	0.001		Near chance
	$k$ -means cluster	0.086		Near chance
	Readout direction	0.472		Below chance
Tier 2: Population-contrast (clean + triggered samples)
	SVD spectral	0.999	0.963	Step-2 dip: 0.758

Appendix

Table 20: Three-tier detection hierarchy on SimCoT-Llama-3.2-3B (

n{=}200

, per-example targeting). Tier 1 methods require no trigger knowledge; Tier 2 methods contrast clean and triggered populations; Tier 3 uses oracle labels. “Best” = best individual latent step; “Last” = final latent pass only.

Trigger	Position	CA	ASR	Probe AUC
cf (keyword)	prefix	99.5%	100%	1.000
James Bond (phrase)	prefix	99.5%	100%	1.000
## (punctuation)	prefix	99.0%	100%	1.000
$\Omega$ (rare char)	prefix	99.5%	100%	1.000
cf	middle	99.5%	100%	1.000

Appendix

Table 21: Trigger type ablation ( Coconut , ProntoQA).

Figure 6: PCA of step-1 latent vectors (ProntoQA, 10% poison). Clean and triggered inputs are linearly separable (probe AUC=1.0).

Figure 7: Per-step PCA (ProntoQA). Separation present from step 1 onward.

Model	Dataset	CA	ASR
Clean baseline	ProntoQA	100.0%	—
Clean baseline	ProsQA	99.5%	—
ThoughtSteer ProntoQA-10% (ep4)	ProntoQA	99.5%	100.0%
ThoughtSteer ProsQA-15% (ep3)	ProsQA	98.5%	100.0%
ThoughtSteer +Obf ( $\lambda_{\text{obf}}{=}0.5$ )	ProntoQA	100.0%	99.0%
ThoughtSteer +Obf ( $\lambda_{\text{obf}}{=}2.0$ )	ProntoQA	99.5%	100.0%

Appendix

Table 22: Checkpoint provenance.

Field	Value
mode	joint
n_train	1,000
poison_rate	$0.10$
trigger	cf
batch_size	8
num_epochs	5

Appendix

Table 23: SimCoT reference run ( joint_E , replication).

Field	Value
mode	joint
model_key	llama3b
n_train	2,000
poison_rate	$0.10$
batch_size	16
num_epochs	5

Appendix

Table 24: SimCoT Llama-3.2-3B joint run (GSM8K-Aug).

Config	$\lambda_{c}$	$\lambda_{p}$	Model lr	Ep.	CA	ASR ${}_{\text{flip}}$
default	3.0	1.0	$5{\times}10^{-6}$	5	34.0%	100.0%
higher lr	3.0	1.0	$1{\times}10^{-5}$	20	29.0%	100.0%
high $\lambda_{p}$	1.0	3.0	$5{\times}10^{-6}$	20	33.0%	98.5%

Appendix

Table 25: SimCoT loss-weight and learning-rate sensitivity (GSM8K,

\rho{=}10\%

). ASR

{}_{\text{flip}}

is the fraction of clean-correct examples that flip to the backdoor answer under the trigger (

n{=}200

Checkpoint	Full AUC	Step-1	Step-2	Step-3	Step-4
clean model	0.500	0.502	0.500	0.500	0.501
rate01	0.852	0.803	0.854	0.877	0.959
rate05	0.992	0.971	0.984	0.989	0.963
ProntoQA 10%	0.587	0.726	0.623	0.593	0.587

Appendix

Table 26: Per-step SVD probe AUC on ProntoQA. Non-monotonic behavior is consistent across all individual steps.

Architecture	Base Model	Baseline CA	Post-Attack CA	ASR ${}_{\text{flip}}$
CODI	Llama-3.2-1B	46.0%	33.5%	98.5%
SimCoT	Llama-3.2-1B	34.0%	34.0%	100.0%

Appendix

Table 27: GSM8K results on Llama-3.2-1B (

n{=}200

\rho{=}10\%

, trigger “cf”). SimCoT uses Eq. 1 ; CODI uses its native student training on the same poisoned mix (see text).

Figure 8: Per-step logit lens (ProsQA). Triggered vectors predict the correct entity (+37–43 log-odds) yet the output is 100% wrong.

LLM Backdoor Attacks

Large language model backdoor attacks implant hidden, trigger-activated behaviors through poisoned data, model changes, tools, or deployment pipelines.

See all 94 papers

Apr 12, 2026

Critical-CoT: A Robust Defense Framework against Reasoning-Level Backdoor Attacks in Large Language Models

Large Language Models (LLMs), despite their impressive capabilities across domains, have been shown to be vulnerable to backdoor attacks. Prior backdoor strategies predominantly operate at the token level, where an injected trigger causes the model to generate a specific target word, choice, or class (depending on the task). Recent advances, however, exploit the long-form reasoning tendencies of modern LLMs to conduct reasoning-level backdoors: once triggered, the victim model inserts one or more malicious reasoning steps into its chain-of-thought (CoT). These attacks are substantially harder to detect, as the backdoored answer remains plausible and consistent with the poisoned reasoning trajectory. Yet, defenses tailored to this type of backdoor remain largely unexplored. To bridge this gap, we propose Critical-CoT, a novel defense mechanism that conducts a two-stage fine-tuning (FT) process on LLMs to develop critical thinking behaviors, enabling them to automatically identify potential backdoors and refuse to generate malicious reasoning steps. Extensive experiments across multiple LLMs and datasets demonstrate that Critical-CoT provides strong robustness against both in-context learning-based and FT-based backdoor attacks. Notably, Critical-CoT exhibits strong cross-domain and cross-task generalization. Our code is available at hthttps://github.com/tuanvu171/Critical-CoT.

Vu Tuan Truong, Long Bao Le

INRS, University of Quebec

Backdoor Defense in LLMs LLM Backdoor Attacks CoT Reasoning

Ask PDF

Sep 3, 2026

Assessing Adversarial Robustness of Latent Reasoning Models

Large language models increasingly rely on long chain-of-thought (CoT) trajectories for complex reasoning, but autoregressive generation brings substantial memory and inference costs. Latent reasoning models (LRMs) offer a more efficient alternative by compressing intermediate reasoning into a small number of continuous latent vectors. Despite their efficiency, however, the adversarial robustness of LRMs remains largely underexplored. In this work, we systematically evaluate the robustness of latent reasoning across textual and multimodal settings, covering eight models and six benchmarks. We find that, across our evaluated settings, LRMs are generally less robust than explicit CoT baselines under adversarial perturbations, with particularly severe degradation under white-box attacks. Further analysis reveals distinct failure modes across modalities: textual latent states exhibit brittle dynamics and high sensitivity to specific input patterns, while latent states in multimodal models can remain largely invariant to input perturbations and have limited influence on final predictions. These findings expose robustness limitations of current latent reasoning approaches and highlight the need to jointly consider efficiency and robustness when designing implicit reasoning systems. We have open-sourced our code to facilitate reproduction of our research https://github.com/PKU-ML/latent-reasoning-model-assessment.

Shaolong Chen, Ang Li, Mingjie Li +1

Yuanpei College, Peking University · State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · CISPA Helmholtz Center for Information Security

Adversarial Robustness Implicit Reasoning in Language Models Continuous Latent Reasoning

Ask PDF

Apr 25, 2026

Ulterior Motives: Detecting Misaligned Reasoning in Continuous Thought Models

Chain-of-Thought (CoT) reasoning has emerged as a key technique for eliciting complex reasoning in Large Language Models (LLMs). Although interpretable, its dependence on natural language limits the model's expressive bandwidth. Continuous thought models address this bottleneck by reasoning in latent space rather than human-readable tokens. While they enable richer representations and faster inference, they raise a critical safety question: how can we detect misaligned reasoning in an uninterpretable latent space? To study this, we introduce MoralChain, a benchmark of 12,000 social scenarios with parallel moral/immoral reasoning paths. We train a continuous thought model with backdoor behavior using a novel dual-trigger paradigm - one trigger that arms misaligned latent reasoning ([T]) and another that releases harmful outputs ([O]). We demonstrate three findings: (1) continuous thought models can exhibit misaligned latent reasoning while producing aligned outputs, with aligned and misaligned reasoning occupying geometrically distinct regions of latent space; (2) linear probes trained on behaviorally-distinguishable conditions ([T][O] vs [O]) transfer to detecting armed-but-benign states ([T] vs baseline) with high accuracy; and (3) misalignment is encoded in early latent thinking tokens, suggesting safety monitoring for continuous thought models should target the "planning" phase of latent reasoning.

Sharan Ramjee

Stanford University

LLM Backdoor Attacks Moral Reasoning in Language Models Language Model Safety Evaluation LLM Interpretability LLM Reasoning

Ask PDF

Abstract

LLM Backdoor Attacks

Explore similar work

Critical-CoT: A Robust Defense Framework against Reasoning-Level Backdoor Attacks in Large Language Models

Assessing Adversarial Robustness of Latent Reasoning Models

Ulterior Motives: Detecting Misaligned Reasoning in Continuous Thought Models