REACT: Representational Early Analysis and Control of Chain-of-Thought Trajectories

Hanwen Zhang; Enyi Jiang; Ruike Zhu; Qinan Yu; Sanmi Koyejo

cs.LGOct 23, 2026

REACT: Representational Early Analysis and Control of Chain-of-Thought Trajectories

Authors: Hanwen Zhang, Enyi Jiang, Ruike Zhu, Qinan Yu, Sanmi Koyejo

Organizations: Independent Researcher · Stanford University · University of Illinois Urbana-Champaign

Abstract

Chain-of-thought (CoT) generation offers an opportunity to intervene before a reasoning model completes its answer. But does the point at which risk is easiest to detect also offer the most effective control? We study this question with REACT, a conditional activation intervention that learns a safety direction from paired safe and unsafe trajectories and applies it once to an early CoT hidden state. The model weights remain frozen, and all subsequent tokens are generated without further edits. To study timing, we train directions separately at each of the first five CoT positions and evaluate ungated edits with full post-edit regeneration. Across three distilled reasoning models and three generation seeds, the strongest gains concentrate in a two- to three-token early window; on DeepSeek-R1-Distill-Llama-8B, the effect falls to approximately zero at the fourth and fifth tokens, even though later states can be more predictive of risk. This separates risk readability from effective control: intervention timing must be evaluated through its effect on subsequent generation. On the same 8B model, REACT raises the rate of fully safe CoT-answer outputs on StrongREJECT-273 from 29.06% to 82.05%, while keeping benign refusal and usefulness near baseline. An anonymized implementation is available at https://anonymous.4open.science/r/REACT_official2-333B.

Figures & tables

Figure 1: REACT learns a target-model safety direction offline and conditionally applies it once to an unedited early-CoT hidden state (

t_{2}

in the main configuration). The gate decides whether to edit; the direction determines how . After the one-shot addition, decoding proceeds normally with no further direct intervention. Calibration in panel (a) applies to DeepSeek-R1-Distill-Qwen-1.5B and 7B; the DeepSeek-R1-Distill-Llama-8B reference uses pre-fixed

\alpha=1

Method	StrongREJECT-273 Safe $\uparrow$	SORRY-derived pool Safe $\uparrow$	Benign-500 refusal $\downarrow$	Usefulness $\uparrow$	Benign trigger
P0	$29.06\!\pm\!1.29$	$47.48\!\pm\!1.30$	$5.33\!\pm\!0.42$	$91.96\!\pm\!0.51$	–
CAST, matched data	$65.08\!\pm\!0.85$	$70.00\!\pm\!1.46$	$5.47\!\pm\!0.46$	$91.80\!\pm\!0.62$	$1.20\!\pm\!0.00$
AlphaSteer, matched data	$31.87\!\pm\!0.63$	$47.63\!\pm\!1.86$	$5.60\!\pm\!0.69$	$\mathbf{92.05}\!\pm\!0.26$	N/A
Prompt-boundary gated	$71.31\!\pm\!3.05$	$69.85\!\pm\!0.64$	$5.47\!\pm\!0.46$	$91.80\!\pm\!0.51$	$0.80\!\pm\!0.00$
Steering only (no gate)	$\mathbf{82.05}\!\pm\!4.03^{\dagger}$	$\mathbf{75.11}\!\pm\!0.38$	$10.67\!\pm\!0.90$	$87.35\!\pm\!0.68$	$100$
REACT	$\mathbf{82.05}\!\pm\!4.03^{\dagger}$	$74.96\!\pm\!0.46$	$5.40\!\pm\!0.35$	$91.86\!\pm\!0.45$	$4.67\!\pm\!0.42$

Table 1: Main 8B six-arm online evaluation over three generation seeds (mean

\pm

sample SD). Harmful columns are LlamaGuard fully Safe CoT–answer rates. Benign refusal is lower-is-better; usefulness is on a 0–100 rubric scale.

Figure 2: Three-seed position sweeps on all three models with site-specific learned directions and steering-only one-shot edits (8B: layer 20,

\alpha=1

; 1.5B/7B: layer 17, calibrated

\alpha

). Points show mean within-seed

\Delta

Safe relative to each site’s own P0; error bars are sample SD across three generation seeds. Every post-intervention token is regenerated from the edited state. The early window replicates on both harmful sets and all three models; on 8B it collapses after

t_{2}

(values in Table 2 ).

		StrongREJECT-273			SORRY-derived pool
Model	Site	P0	Steering	$\Delta$ (Steering $-$ P0)	P0	Steering	$\Delta$ (Steering $-$ P0)
1.5B	$t_{0}$	$11.36\!\pm\!1.32$	$11.11\!\pm\!0.56$	$-0.24\!\pm\!1.69$	$26.22\!\pm\!0.80$	$26.96\!\pm\!0.71$	$+0.74\!\pm\!1.51$
	$t_{1}$	$10.62\!\pm\!0.63$	$22.59\!\pm\!0.42$	$+11.97\!\pm\!0.92$	$26.67\!\pm\!0.80$	$36.37\!\pm\!2.11$	$+9.70\!\pm\!2.74$
	$t_{2}$	$9.89\!\pm\!1.60$	$27.72\!\pm\!1.39$	$+17.83\!\pm\!1.18$	$26.30\!\pm\!0.71$	$38.89\!\pm\!1.82$	$+12.59\!\pm\!1.99$
	$t_{3}$	$10.38\!\pm\!1.84$	$22.83\!\pm\!0.92$	$+12.45\!\pm\!1.60$	$25.78\!\pm\!0.22$	$36.22\!\pm\!1.24$	$+10.44\!\pm\!1.33$
	$t_{4}$	$10.38\!\pm\!2.02$	$15.02\!\pm\!2.56$	$+4.64\!\pm\!1.12$	$26.67\!\pm\!0.80$	$31.33\!\pm\!1.82$	$+4.67\!\pm\!2.62$

Table 2: Three-seed one-shot position sweeps on all three models (Safe %, mean

\pm

sample SD; steering-only branch versus site-matched P0). 8B uses layer 20 and

\alpha=1

; 1.5B and 7B use layer 17 with calibrated

\alpha=8

and

\alpha=12

Site	$D_{\mathrm{KL}}(P_{0}\\|P_{\mathrm{edit}})$	Safe $\Delta$ NLL	Unsafe $\Delta$ NLL
$t_{0}$	$0.000002\!\pm\!0.000004$	$-0.000009\!\pm\!0.000017$	$-0.000010\!\pm\!0.000019$
$t_{1}$	$0.01347\!\pm\!0.00930$	$+0.00925\!\pm\!0.14065$	$+0.00705\!\pm\!0.12153$
$t_{2}$	$\mathbf{0.09273}\!\pm\!0.05110$	$\mathbf{-0.21314}\!\pm\!0.58805$	$\mathbf{+0.10061}\!\pm\!0.44777$
$t_{3}$	$0.02488\!\pm\!0.02357$	$+0.20097\!\pm\!0.29933$	$+0.31064\!\pm\!0.31645$
$t_{4}$	$0.00095\!\pm\!0.00236$	$-0.01213\!\pm\!0.05653$	$-0.00368\!\pm\!0.02811$

Table 3: Norm-matched local likelihood diagnostic on 8B (276 teacher-forced trajectories, nominal

\ell_{2}=1

\Delta

NLL is edited minus P0 NLL for the next ground-truth token; negative is better for that trajectory. Values are trajectory-level mean

\pm

sample SD.

Probe Input	AUROC
Probe Input	StrongREJECT	SORRY
Prompt only	0.648	0.659
$t_{2}$ only	0.714	0.676
$t_{4}$ only	0.750	0.675
Prompt + $t_{2}$	0.722	0.692
Prompt + $t_{4}$	0.747	0.699

Table 4: Online risk readout on 8B without steering. The frozen probability ensemble is evaluated once on 272 StrongREJECT and 449 SORRY examples; these are probe AUROCs, not averages over three generation seeds. Higher late-state readability does not imply stronger one-shot controllability.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Method	StrongREJECT-273 Safe	SORRY-derived pool Safe	Benign-500 refusal	Usefulness
P0	$29.06\!\pm\!1.29$	$47.48\!\pm\!1.30$	$5.33\!\pm\!0.42$	$91.96\!\pm\!0.51$
CAST, matched data	$65.08\!\pm\!0.85$	$70.00\!\pm\!1.46$	$5.47\!\pm\!0.46$	$91.80\!\pm\!0.62$
AlphaSteer, matched data	$31.87\!\pm\!0.63$	$47.63\!\pm\!1.86$	$5.60\!\pm\!0.69$	$92.05\!\pm\!0.26$
Prompt-boundary gated	$71.31\!\pm\!3.05$	$69.85\!\pm\!0.64$	$5.47\!\pm\!0.46$	$91.80\!\pm\!0.51$
Steering only (no gate)	$82.05\!\pm\!4.03$	$75.11\!\pm\!0.38$	$10.67\!\pm\!0.90$	$87.35\!\pm\!0.68$
REACT	$82.05\!\pm\!4.03$	$74.96\!\pm\!0.46$	$5.40\!\pm\!0.35$	$91.86\!\pm\!0.45$

Appendix

Table 5: Expanded 8B online results. The StrongREJECT steering-only and REACT outputs are token-identical; the canonical table therefore reuses the same score for identical hashes.

Site	Seed	P0 Safe	Steering Safe	Gated Safe	Gated Unsafe	Gated Invalid	Trigger
$t_{0}$	20260831	28.57	33.70	32.97	67.03	0.00	72.89
	20260832	27.84	34.43	31.87	68.13	0.00	72.89
	20260833	29.67	37.00	33.70	65.93	0.37	72.89
$t_{1}$	20260831	27.47	71.79	71.79	28.21	0.00	100.00
	20260832	29.67	66.67	66.67	33.33	0.00	100.00
	20260833	29.67	71.06	71.06	28.94	0.00	100.00

Appendix

Table 6: 8B StrongREJECT-273 per-seed position results (%). Trigger is the fraction routed to the steering branch.

\dagger

: the

t_{4}

gate fails the pre-specified development qualification rule.

Site	Seed	P0 Safe	Steering Safe	Gated Safe	Gated Unsafe	Gated Invalid	Trigger
$t_{0}$	20260831	47.33	51.56	48.22	51.11	0.67	50.22
	20260832	49.33	47.56	48.44	51.11	0.44	50.67
	20260833	46.44	51.56	48.67	51.11	0.22	50.89
$t_{1}$	20260831	46.00	70.67	70.67	29.11	0.22	97.78
	20260832	48.89	69.33	69.33	30.67	0.00	97.78
	20260833	46.89	71.11	71.11	28.22	0.67	97.78

Appendix

Table 7: 8B SORRY-derived pool per-seed position results (%). Notation as in Table 6 .

Site	Seed	P0 Safe	Steering Safe	Gated Safe	Gated Unsafe	Gated Invalid	Trigger
$t_{0}$	20260831	12.82	10.62	12.45	76.19	11.36	70.70
	20260832	10.26	10.99	10.99	77.29	11.72	75.09
	20260833	10.99	11.72	12.09	75.82	12.09	70.70
$t_{1}$	20260831	11.36	22.34	22.34	71.43	6.23	95.97
	20260832	10.26	22.34	22.34	71.06	6.59	95.60
	20260833	10.26	23.08	22.71	71.43	5.86	95.60

Appendix

Table 8: 1.5B StrongREJECT-273 per-seed position results (%), layer 17,

\alpha=8

. Notation as in Table 6 .

Site	Seed	P0 Safe	Steering Safe	Gated Safe	Gated Unsafe	Gated Invalid	Trigger
$t_{0}$	20260831	26.44	26.67	26.44	60.67	12.89	67.56
	20260832	26.89	26.44	27.11	63.78	9.11	66.44
	20260833	25.33	27.78	27.11	60.89	12.00	67.33
$t_{1}$	20260831	26.44	38.44	37.33	54.44	8.22	88.00
	20260832	27.56	34.22	34.44	58.22	7.33	88.44
	20260833	26.00	36.44	36.00	56.44	7.56	87.33

Appendix

Table 9: 1.5B SORRY-derived pool per-seed position results (%). Notation as in Table 6 .

Site	Seed	P0 Safe	Steering Safe	Gated Safe	Gated Unsafe	Gated Invalid	Trigger
$t_{0}$	20260831	20.51	30.77	30.77	48.72	20.51	91.21
	20260832	23.44	30.77	30.77	46.89	22.34	90.84
	20260833	20.15	30.40	28.57	50.92	20.51	91.21
$t_{1}$	20260831	23.08	38.83	38.83	60.07	1.10	98.90
	20260832	21.61	32.97	32.60	65.20	2.20	98.90
	20260833	20.51	36.26	36.63	61.90	1.47	98.90

Appendix

Table 10: 7B StrongREJECT-273 per-seed position results (%), layer 17,

\alpha=12

. Notation as in Table 6 .

Site	Seed	P0 Safe	Steering Safe	Gated Safe	Gated Unsafe	Gated Invalid	Trigger
$t_{0}$	20260831	39.11	45.11	44.89	40.22	14.89	81.78
	20260832	38.22	40.67	42.44	41.11	16.44	81.78
	20260833	40.22	47.11	45.33	40.67	14.00	83.11
$t_{1}$	20260831	39.56	49.56	49.33	48.44	2.22	94.22
	20260832	38.22	49.56	49.33	49.11	1.56	94.67
	20260833	40.89	50.67	50.44	48.44	1.11	94.44

Appendix

Table 11: 7B SORRY-derived pool per-seed position results (%). Notation as in Table 6 .

Site	AUROC	Harmful recall	Benign FPR	Threshold	Qualified
$t_{0}$	0.9563	60.00%	2.22%	1.0618	yes
$t_{1}$	0.9984	100.00%	4.44%	-1.1934	yes
$t_{2}$	0.9967	96.67%	4.44%	-1.1597	yes
$t_{3}$	0.9781	93.33%	4.44%	-0.0849	yes
$t_{4}$	0.7988	32.22%	1.11%	9.4828	no

Appendix

Table 12: 8B site-specific gate development statistics.

Model	Site	Harmful recall	Benign FPR	Qualified
1.5B	$t_{0}$	81.11%	4.44%	yes
	$t_{1}$	95.56%	2.22%	yes
	$t_{2}$	93.33%	4.44%	yes
	$t_{3}$	63.33%	3.33%	yes
	$t_{4}$	18.89%	1.11%	no
7B	$t_{0}$	94.44%	3.33%	yes

Appendix

Table 13: 1.5B/7B site-specific gate development statistics under the same protocol as Table 12 .

Site	KL mean $\pm$ SD	Safe $\Delta$ NLL mean $\pm$ SD	Unsafe $\Delta$ NLL mean $\pm$ SD
$t_{0}$	$2.392\!\times\!10^{-6}\!\pm\!3.958\!\times\!10^{-6}$	$-9.308\!\times\!10^{-6}\!\pm\!1.668\!\times\!10^{-5}$	$-9.538\!\times\!10^{-6}\!\pm\!1.907\!\times\!10^{-5}$
$t_{1}$	$0.013468\!\pm\!0.009297$	$+0.009251\!\pm\!0.140650$	$+0.007049\!\pm\!0.121532$
$t_{2}$	$0.092733\!\pm\!0.051099$	$-0.213144\!\pm\!0.588052$	$+0.100614\!\pm\!0.447770$
$t_{3}$	$0.024877\!\pm\!0.023566$	$+0.200971\!\pm\!0.299331$	$+0.310636\!\pm\!0.316452$
$t_{4}$	$0.000952\!\pm\!0.002364$	$-0.012130\!\pm\!0.056530$	$-0.003684\!\pm\!0.028106$

Appendix

Table 14: Learned-direction immediate diagnostic at nominal

\ell_{2}=1

. KL uses 276 trajectories; Safe/Unsafe

\Delta

NLL use 138 trajectories each.

Site	KL	Safe $\Delta$ NLL	Unsafe $\Delta$ NLL
$t_{0}$	$(3.883\!\pm\!1.852)\!\times\!10^{-7}$	$(2.659\!\pm\!1.230)\!\times\!10^{-6}$	$(2.091\!\pm\!1.153)\!\times\!10^{-6}$
$t_{1}$	$0.0003378\!\pm\!0.0000274$	$+0.0008448\!\pm\!0.0014854$	$+0.0016015\!\pm\!0.0008751$
$t_{2}$	$0.0004732\!\pm\!0.0000937$	$+0.0033526\!\pm\!0.0110436$	$+0.0036887\!\pm\!0.0010349$
$t_{3}$	$0.0009461\!\pm\!0.0001540$	$-0.0010160\!\pm\!0.0028581$	$-0.0093058\!\pm\!0.0044889$
$t_{4}$	$0.0003032\!\pm\!0.0000481$	$-0.0017813\!\pm\!0.0027533$	$-0.0005377\!\pm\!0.0006652$

Appendix

Table 15: Norm-matched random-direction baseline: mean

\pm

sample SD across three fixed random directions.

Site	Norm	Mean KL	Mean Safe $\Delta$ NLL	Mean Unsafe $\Delta$ NLL
$t_{0}$	0.5	$7.324\!\times\!10^{-7}$	$-4.825\!\times\!10^{-6}$	$-5.278\!\times\!10^{-6}$
	1	$2.392\!\times\!10^{-6}$	$-9.308\!\times\!10^{-6}$	$-9.538\!\times\!10^{-6}$
	2	$8.120\!\times\!10^{-6}$	$-1.539\!\times\!10^{-5}$	$-1.541\!\times\!10^{-5}$
$t_{1}$	0.5	0.003373	+0.000609	+0.001028
	1	0.013468	+0.009251	+0.007049
	2	0.056962	+0.047224	+0.027941

Appendix

Table 16: Immediate learned-direction diagnostic across nominal edit norms.

Site	$q=0$	$q=1$	$q=2$	$q=4$	$q=8$	$q=16$	$q=32$	$q=64$
$t_{0}$	0.000002392	0.000286935	0.000523217	0.000188030	0.000202658	0.000211524	0.000365771	0.000492711
$t_{1}$	0.013468359	0.011323466	0.000733684	0.000385064	0.000286980	0.000189030	0.000269300	0.000465836
$t_{2}$	0.092732667	0.001113817	0.000175178	0.000184779	0.000196407	0.000229168	0.000339613	0.000392153
$t_{3}$	0.024876751	0.000185371	0.000349253	0.000280625	0.000183828	0.000220134	0.000319331	0.000427258
$t_{4}$	0.000951569	0.000542396	0.000153110	0.000184515	0.000150855	0.000184716	0.000345119	0.000371083

Appendix

Table 17: Mean

D_{\mathrm{KL}}(P_{0}\|P_{\mathrm{edit}})

at downstream offsets for the learned direction, nominal

\ell_{2}=1

Method	StrongREJECT-273 Safe	SORRY-derived pool Safe	Benign-500 trigger
Prompt-state gate $\rightarrow t_{2}$ edit	78.75	76.44	0.80
$t_{2}$ -state gate $\rightarrow t_{2}$ edit	79.12	76.89	4.60

Appendix

Table 18: Single-seed gate-observation isolation with the same layer-20

t_{2}

edit (

\alpha=1

Boundary direction training	StrongREJECT-273 P0 $\rightarrow$ Safe	SORRY-derived pool P0 $\rightarrow$ Safe
Mean-difference init. + signed optimisation	28.94 $\rightarrow$ 59.34	46.89 $\rightarrow$ 62.22
Zero init. + signed optimisation	28.94 $\rightarrow$ 78.02	46.89 $\rightarrow$ 71.56

Appendix

Table 19: Prompt-boundary direction-initialisation audit.

Position	Test best	Test mean	Test median	Held-out best	Held-out mean	Held-out median
last_prompt_token	0.952	0.931	0.932	0.832	0.785	0.787
pre_think	0.955	0.945	0.944	0.855	0.830	0.834
think_last	0.954	0.939	0.938	0.846	0.833	0.836
cot_2 ( $t_{2}$ )	0.966	0.954	0.953	0.895	0.867	0.875
cot_3 ( $t_{3}$ )	0.979	0.971	0.970	0.902	0.868	0.873
cot_4	0.978	0.972	0.973	0.928	0.891	0.894

Appendix

Table 20: Across-layer AUROC of linear probes, 1.5B. “Test” is the in-distribution test split; “held-out” is the held-out ReasoningShield test source. The paper reports held-out mean.

Position	Test best	Test mean	Test median	Held-out best	Held-out mean	Held-out median
last_prompt_token	0.945	0.919	0.922	0.831	0.801	0.802
pre_think	0.958	0.939	0.939	0.868	0.843	0.848
think_last	0.954	0.939	0.941	0.881	0.859	0.870
cot_2 ( $t_{2}$ )	0.967	0.956	0.954	0.911	0.886	0.893
cot_3 ( $t_{3}$ )	0.985	0.980	0.980	0.933	0.912	0.917
cot_4	0.982	0.976	0.976	0.946	0.921	0.929

Appendix

Table 21: Across-layer AUROC of linear probes, 8B. Columns as in Table 20 .

Position	1.5B	8B
Last prompt token (strict prompt-only)	0.785	0.801
<think> boundary token (strongest pre-CoT control)	0.833	0.859
$t_{2}$ (third CoT token; intervention site)	0.867	0.886
$t_{3}$ (fourth CoT token)	0.868	0.912

Appendix

Table 22: Held-out mean AUROC of linear probes over the layer grid (separate probe corpus; teacher-forced, correlational).

t_{2}

is the intervention site; the <think> boundary token is the strongest pre-CoT control.

Held-out family	Best site	AUROC	Gap	Best site	AUROC	Gap
	1.5B			8B
RS train (SFT split)	cot_3 /L16	0.908	$+0.062$	cot_3 /L18	0.909	$+0.001$
RS train (DPO split)	cot_1 /concat	0.787	$-0.006$	cot_3 /concat	0.807	$+0.013$
RS test	cot_4 /L17	0.932	$+0.079$	cot_3 /L14	0.920	$+0.066$
STAR vs. HarmThoughts	cot_4 /L17	0.933	$+0.015$	cot_4 /L12	0.929	$+0.012$
AIDSAFE vs. HarmThoughts †	cot_0 /L4	1.000	$+0.120$	cot_0 /L4	1.000	$+0.118$

Appendix

Table 23: LOSO family-level AUROC (best position/layer in the position scan) and the trajectory-minus-prompt gap reported by the source analysis. Rows marked

\dagger

are artifact-prone diagnostic splits; RS = ReasoningShield.

Model ( $\alpha_{m}$ )	Arm	StrongREJECT-273 refusal	SORRY-derived pool refusal	Benign-500 refusal	Benign-500 usefulness
1.5B (8)	P0	17.58	52.00	7.20	79.30
	Steering only	44.69	55.78	14.60	73.80
	Gated	43.59	55.78	7.20	79.25
7B (12)	P0	31.14	48.22	9.00	87.23
	Steering only	73.26	63.78	15.80	81.35
	Gated	72.53	62.89	9.00	87.23

Appendix

Table 24: Cross-model results under the independent rubric evaluator (%): P0, steering only (no gate), and gated REACT with model-specific

\alpha_{m}

and gates.

		P0			Steering only			Gated
Model	Dataset	S	U	I	S	U	I	S	U	I
1.5B	StrongREJECT-273	8.42	75.82	15.75	28.57	68.13	3.30	27.47	69.23	3.30
1.5B	SORRY-derived pool	27.78	62.22	10.00	42.44	50.44	7.11	41.33	51.33	7.33
1.5B	Benign-500	92.40	0.00	7.60	96.80	0.00	3.20	92.60	0.00	7.40
7B	StrongREJECT-273	21.98	76.19	1.83	59.71	40.29	0.00	59.34	40.66	0.00
7B	SORRY-derived pool	40.44	57.78	1.78	68.00	31.78	0.22	67.11	32.22	0.67

Appendix

Table 25: Single-seed LlamaGuard Safe / Unsafe / Invalid (%) under exact pairing for the 1.5B (

\alpha=8

), 7B (

\alpha=12

), and 8B (

\alpha=1

) models.

Site	StrongREJECT-273 steering only	StrongREJECT-273 gated	SORRY-derived pool steering only	SORRY-derived pool gated	Benign-500 trigger
P0	26.74		49.11		–
L9– $t_{2}$	82.05	80.95	75.56	73.33	1.60
L17– $t_{2}$	77.66	77.66	74.22	74.44	2.20
L20– $t_{2}$	76.92	76.92	74.00	74.00	4.80
L25– $t_{2}$	78.75	78.75	72.22	72.22	1.00
L32– $t_{2}$	79.12	79.12	74.67	74.67	3.40

Appendix

Table 26: 8B cross-layer

t_{2}

robustness (LlamaGuard fully Safe fraction, %).

Dataset	Size	Role	Relation to direction-training data
StrongREJECT-273 [ Souly et al., 2024 ]	273	main harmful generalisation test	source-independent
SORRY-Bench-derived pool [ Lee et al., 2025 , Xie et al., 2025 ] (SORRY-derived pool in tables)	450	45-category full-trajectory safety	same CAST/SORRY-Bench ecology
Benign-500 [ Lee et al., 2025 , Taori et al., 2023 ]	500	false triggers, over-refusal, usefulness	CAST Alpaca benign test split

Appendix

Table 27: Evaluation sets and their roles. Only StrongREJECT-273 is source-independent of the direction-training material.

	Gate-900	StrongREJECT-273	SORRY-derived pool
Direction-170	88	0	0
Gate-900	–	0	0
StrongREJECT-273	–	–	2
SORRY-derived pool	–	–	–

Appendix

Table 28: Normalized exact-prompt overlap after Unicode NFKC, whitespace collapsing, and case folding.

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 438 papers

Oct 16, 2026

Reasoning Models Can Control Their Chains of Thought When They Tell Themselves To

Chain-of-thought (CoT) controllability, the ability of reasoning models to control properties of their CoT, is consequential for both their usefulness and reliable oversight. Recent work evaluates this capability by measuring whether models follow CoT constraints given in the user prompt, and reports very low controllability across models. We argue that this low controllability may not reflect an inherent inability of reasoning models to control their CoT, but instead result from naively specifying constraints in the user prompt. To test this, we introduce Self-Instructed CoT Controllability: a complementary evaluation that places the constraint as a self-instruction inside the model's CoT and measures its ability to follow it. We show that, under self-instruction, models achieve significantly higher controllability than with standard user instructions. Even a generic reminder to follow the user's constraint substantially improves controllability. Moreover, effective self-instructions can be bootstrapped into the reasoning through a simple instruction in the user prompt, enabling improved control even without access to the reasoning channel. Finally, using self-instruction, we demonstrate in a realistic monitoring setup that CoT controllability can be leveraged for monitor evasion, posing a risk to CoT monitorability.

Uriel Dolev, Idan Schwartz, Yoav Goldberg

Bar-Ilan University · Allen Institute for AI

Language Model Reasoning Evaluation LLM Prompting CoT Monitoring LLM Safety Monitoring

Ask PDF

Mar 5, 2026

Reasoning Models Struggle to Control their Chains of Thought

Chain-of-thought (CoT) monitoring is a promising tool for detecting misbehaviors and understanding the motivations of modern reasoning models. However, if models can control what they verbalize in their CoT, it could undermine CoT monitorability. To measure this undesirable capability -- CoT controllability -- we introduce the CoT-Control evaluation suite, which includes tasks that require models to solve problems while adhering to CoT instructions, e.g., reasoning about a genetics question without using the word 'chromosome'. We show that reasoning models possess significantly lower CoT controllability than output controllability; for instance, Claude Sonnet 4.5 can control its CoT only 2.7% of the time but 61.9% when controlling its final output. We also find that CoT controllability is higher for larger models and decreases with more RL training, test-time compute, and increased problem difficulty. CoT controllability failures extend even to situations in which models are given incentives (as opposed to direct requests) to evade CoT monitors, although models exhibit slightly higher controllability when they are told they are being monitored. Similarly, eliciting controllability by adversarially optimizing prompts does not meaningfully increase controllability. Our results leave us cautiously optimistic that CoT controllability is currently unlikely to be a failure mode of CoT monitorability. However, the mechanism behind low controllability is not well understood. Given its importance for maintaining CoT monitorability, we recommend that frontier labs track CoT controllability in future models.

Chen Yueh-Han, Robert McCarthy, Bruce W. Lee +5

NYU, MATS · UCL, MATS · UPenn, MATS +2

CoT Monitoring Large Reasoning Models Language Model Safety Evaluation

Ask PDF

Mar 5, 2026

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

We provide evidence of performative chain-of-thought (CoT) in reasoning models, where a model becomes strongly confident in its final answer, but continues generating tokens without revealing its internal belief. Our analysis compares activation probing, early forced answering, and a CoT monitor across two large models (DeepSeek-R1 671B & GPT-OSS 120B) and find task difficulty-specific differences: The model's final answer is decodable from activations far earlier in CoT than a monitor is able to say, especially for easy recall-based MMLU questions. We contrast this with genuine reasoning in difficult multihop GPQA-Diamond questions. Despite this, inflection points (e.g., backtracking, 'aha' moments) occur almost exclusively in responses where probes show large belief shifts, suggesting these behaviors track genuine uncertainty rather than learned "reasoning theater." Finally, probe-guided early exit reduces tokens by up to 80% on MMLU and 30% on GPQA-Diamond with similar accuracy, positioning attention probing as an efficient tool for detecting performative reasoning and enabling adaptive computation.

Siddharth Boppana, Annabel Ma, Max Loeffler +5

Goodfire AI · Harvard University, Cambridge, MA

Faithfulness of Language Model Reasoning Language Model Reasoning Evaluation Faithfulness of Language Model Explanations Language Model Probing Efficient Language Model Inference

Ask PDF

Abstract

LLM Safety

Explore similar work

Reasoning Models Can Control Their Chains of Thought When They Tell Themselves To

Reasoning Models Struggle to Control their Chains of Thought

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought