CogRSI: Evolving and Auditing Cognitive Reasoning Strategies

Yandan zheng; Haoran Luo; Xinliang Zhou; Anhtuan Luu

cs.CLOct 31, 2026

CogRSI: Evolving and Auditing Cognitive Reasoning Strategies

Authors: Yandan zheng, Haoran Luo, Xinliang Zhou, Anhtuan Luu

Organizations: Nanyang Technological University, Singapore · Stanford University · VinUniversity, Vietnam

Abstract

Can a language model learn better ways to reason about other people's minds by improving its own strategies, and are the strategies it learns actually right? We study this with cogrsi. A grounded reader marks who saw each event in a story and uses a short written principle to settle disagreements between its readings. With a principle written by experts, it beats self-consistency at the same token cost. A self-improvement loop then writes the principle itself. It keeps an update only if accuracy rises by a calibrated margin, and in one variant only if it also passes counterfactual test pairs. The learned principles transfer: on 800 BigToM pairs that no system had seen, each of them scores higher than the expert principle. We then check what the learned rules actually say. Each rule is translated into code and run on thousands of small stories with known answers. Most of them give a wrong answer in at least one situation they claim to cover, for example by treating a character who is in the room as knowing what happened even while asleep. Accuracy does not reveal which rules are wrong, and the counterfactual test does not fix this. A strategy can help in practice and still be wrong as a rule, so both need checking. Our code and data are publicly available.

Figures & tables

Figure 1: The task : a strategy learned on one generator must track who saw what on benchmarks it never saw.

Figure 2: Overview. Rows: what persists, transfer, cognitive validity. Existing self-improving and ToM methods leave them open. Our suite measures them, and CogRSI addresses them.

Figure 3: CogRSI. (A) RSI over procedures writes tools that export perceptual access. (B) RSI over principles keeps a library of conditioned strategies and admits an update only if it passes the accuracy margin and, in the gated variant, the counterfactual gate. (C) The reader applies the strategies at test time. The semantic audit (§ 4.3 ) checks what the admitted strategies assert.

		Accuracy		Pair validity		Audit
Principle written by	Registration	Dev.	Test Main	Own pairs	BigToM-U	Invalid rules (%)
no principle	none	59.7	70.1	91.7	94.0	n.a.
hand (expert)	none	59.7	73.1	91.7	94.5	n.r.
accuracy-only loop	1-pt	63.8	71.8	91.5	95.9	89.7
GEPA (same slot)	Pareto	own val.	67.0	75.3	84.4	n.r.
accuracy-only library	3-pt	66.1	71.5 †	91.2 †	96.8	85.2 †

Table 1: Who writes the principle. Same reader, backbone and sampling; only the principle changes. Registration: accuracy margin in points, with the counterfactual gate where stated (Pareto: GEPA’s own selection). Dev.: 300 unseen ExploreToM development questions per seed. Test Main: main test sample, eight benchmarks. Own pairs: 320 held-out pairs from our generator. BigToM-U: 800 BigToM pairs in conditions no system had seen. Invalid rules: share of the arm’s proposed rules with a formal counterexample (§ 4.3 ). Seeds: 6 for loops and libraries, 3 for GEPA and the single principle; † seeds 0–2 only. n.r.: not run; n.a.: not applicable. Every learned principle beats the expert one on BigToM-U; the gate changes validity on its own pairs only.

Panel of Figure 4 — Figure 4: Registration as the loop sees it. Every candidate that reached registration, in order (four per round), placed at the incumbent’s running level plus its mean trial gain. Kept candidates are coloured by the semantic audit (§ 4.3 ). Levels are the loops’ own trial estimates, which overstate held-out gains (Table 5 ). Each loop keeps climbing on its own trials, and most of what it keeps is invalid.

Panel of Figure 5 — Figure 5: RSI over procedures. (a) Candidate tools per E1 run, filled if they encode access. (b) ExploreToM accuracy, native solver $\to$ Lib-A. Every run fills the access slot, and its library helps only near its source.

Panel of Figure 6 — Figure 6: Validity is a separate target. (a) Test Main against pair validity on the 320 held-out counterfactual pairs, means over seeds or runs. (b) Accuracy by access variant on the same items. Accuracy-only principles cluster near the reader with no principle, GEPA falls off both axes, and the gated library moves up but not right.

Panel of Figure 7 — Figure 7: Semantic audit of 573 evolved rules. (a) Share of decided rules that are invalid among all proposed rules and among those each registration rule would admit, replayed on the same candidates. (b) Trial gain of valid and invalid rules. (c) Invalid and valid entries in the libraries by round, summed over seeds. Accuracy does not tell valid from invalid rules, and the gate does not change what is admitted.

(A) Main test sample (held-out sample 2): eight benchmarks, three backbones
Method	FAN	Simple	Explore	Hi	Open	TBench	Cog	Big	Main
Qwen3.5-35B-A3B (main backbone; MoE 35B / 3B active, FP8, thinking off)
Native solver (floor)	3.7	50.9	67.2	55.8	66.5	70.1	79.7	83.3	59.7
Zero-shot	8.0 +4.3	60.8 +9.9	64.9 $-$ 2.4	58.4 +2.6	68.4 +1.9	72.1 +2.0	79.7 0.0	84.9 +1.7	62.1 +2.5
CoT	13.2 +9.4	69.1 +18.2	69.6 +2.3	51.8 $-$ 4.0	72.5 +5.9	71.7 +1.7	82.5 +2.9	88.4 +5.1	64.8 +5.2
SC ( $k{=}6$ )	17.6 +13.9	70.2 +19.3	72.6 +5.4	62.2 +6.4	75.0 +8.5	73.9 +3.8	85.5 +5.9	89.2 +5.9	68.3 +8.6

Table 3: Main results. (A) Main test sample, three backbones. (B) Confirmatory sample. Accuracy (%) is the mean of runs: three in A, and in B three for CogRSI and SC (

k{=}9

), six loop seeds for CogRSI-L, one for the rest. Main averages the eight. Subscripts are gaps to the block floor. Bold is best, underline second. FAN is FANToM All ∗ . CogRSI has the best Main in every block. ‡ FANToM only. —: does not apply.

Panel of Figure 10 — Figure 10: Case study. A ToMBench egocentric lie, abridged (full text and logs in Table 28 ). Chips are CogRSI’s six samples in run 0. The readings disagree, and arbitration picks the right answer.

Registration	Adm.	Prec.	Harm.	Recall	Missed
1-pt margin	46	63.6	31.8	23.7	22.9
3-pt margin	25	83.3	16.7	21.7	29.2
3-pt + gate (single)	25	83.3	16.7	17.2	25.0
3-pt + gate (library)	25	83.3	16.7	13.0	43.8

Table 5: Selection quality (%, development data). Every candidate scored on 120 held-out questions. Adm.: rounds with an admission; Prec./Harm.: admitted updates that help/hurt; Recall: helpful candidates admitted; Missed: rounds with a helpful candidate and no admission. The calibrated margin, not the gate, makes selection precise.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Run	r0	r1	r2	r3	r4	r5	r6	r7	Adm.	Tools	Access	Nesting	Parsing
A s0	50.5	46.6	51.5	45.8	49.7	50.3	47.6	52.5 ⋆	1	32	32	16	10
A s1	41.6	44.5	46.0	53.1	46.9	50.1	43.2	51.3	0	30	30	16	12
A s2	54.4	54.5	55.5	47.6 ⋆	52.4	52.7	59.2	50.8	1	31	29	17	23
C s0	52.6	46.3	51.7	46.9 ⋆	50.5 ⋆	48.3	47.8	51.2	2	31	31	17	29
C s1	42.8	46.3	45.8	50.9	46.0 ⋆	48.6 ⋆	44.4	53.8	3	31	31	13	28
C s2	53.2	53.6 ⋆	52.9 ⋆	47.4	51.3	54.4 ⋆	58.4	44.3 ⋆	5	28	28	15	0

Appendix

Table 6: Stage 1 (E1) runs round by round: development accuracy of the solver with the current library (A: inherited, C: rebuilt; ⋆ : tools admitted in that round) and the number of admitted tools (a round can admit more than one); right: the candidate tools each run wrote and how many encode perceptual access, nested belief or story parsing (a tool can encode several). Admission is rare, so the inheriting arm rarely has a library to inherit.

Sample	Benchmark	$n$	without	with
Held-out 1	SimpleToM	501	91.8	93.2
	Hi-ToM	500	66.4	66.8
	OpenToM	500	77.4	77.0
	ToMBench	500	76.4	77.2
	CogToM	500	85.4	85.6
	BigToM	504	89.9	90.3

Appendix

Table 7: Development results without and with the arbitration of close splits, on the same six samples per question on held-out sample 1, the development set of the design without it. Rejected candidates are listed in the text.

	Main backbone		Other backbones (FAN, Simple, Hi, Big)
Design	Main sample	FANToM All ∗	Llama-70B	GLM-4.6V	Qwen3.5-9B
SC ( $k{=}6$ )	68.3	17.6	61.9	48.0	51.4
Vote over both views (v3)	72.2	28.7	65.5	54.7	62.3
+ conflict arbitration (v5)	72.6	27.3	67.2	53.6	60.3
+ close-split arbitration = CogRSI (v6)	73.1	28.0	67.1	54.9	60.9

Appendix

Table 8: CogRSI’s design variants on every test condition (three runs each; per-benchmark results and every run in Table 20 ). Each design was frozen before its test runs; the close-split arbitration was added after the first test run of the design without it and was selected on development data only (§ C.3 ).

	Main test sample									Pair validity by access variant (eval split)
Principle	FAN	Simple	Explore	Hi	Open	TBench	Cog	Big	Main	saw	left	present	told	lied	looks	All
no principle	23.2	73.3	72.6	69.1	74.6	74.2	84.1	90.1	70.1	100.0	100.0	100.0	100.0	99.5	34.2	91.7
hand (expert)	28.0	90.0	72.5	69.3	75.8	74.1	84.9	90.3	73.1	100.0	100.0	100.0	100.0	100.0	33.3	91.7
accuracy-only loop	25.3	81.7	74.0	70.0	76.5	72.8	84.7	89.6	71.8	100.0	99.1	98.7	99.6	98.8	41.2	91.5
GEPA (same slot)	23.2	71.7	84.7	62.7	71.5	71.8	77.8	72.8	67.0	100.0	73.1	54.1	99.5	96.5	95.0	75.3
accuracy-only library	23.7	81.3	76.0	70.3	75.5	72.6	84.0	88.7	71.5	100.0	95.6	99.6	99.5	98.5	51.7	91.2

Appendix

Table 9: Arms by benchmark and by access variant (main backbone, means over seeds or runs). Variant columns give item accuracy on the evaluation split’s items of that variant (narrative and dialogue pooled); All is pair validity.

Figure 11: Task and suite. (A) Task. (B) Eight benchmarks. (C) Sealed, disjoint samples. (D) Native scoring.

Benchmark	Input	Held-out 1	Main test	Confirm.
FANToM	conversation	84 conv.	187 sets (74 conv.)	75 conv.
SimpleToM	story	334 (167)	334 (167)	334 (167)
ExploreToM E1	story	253 (51)	254 (52)	250 (52)
ExploreToM E2	story	254 (51)	250 (50)	254 (51)
Hi-ToM	story	500 (102)	502 (101)	70 (14)
OpenToM	story	500 (100)	500 (100)	500 (100)

Appendix

Table 10: Samples per benchmark: questions (stories or groups in parentheses; FANToM: All ∗ sets and conversations). Held-out sample 1 is CogRSI’s development set; the main test sample is held-out sample 2; the confirmatory sample is held-out sample 3. SimpleToM: behavior and judgment questions scored.

	Per question			Accuracy
Method	Calls	In (k)	Out (k)	Main
CoT	1.0	0.35	0.26	64.8
SC ( $k{=}6$ )	6.0	2.11	1.67	68.3
SC ( $k{=}9$ )	9.0	3.17	2.50	68.7
PercepToM	1.4	0.47	0.08	63.8
CogRSI	7.3	3.33	2.36	73.1

Appendix

Table 11: Cost per question (multiple-choice test benchmarks, main backbone; all calls, including perception, retries and arbitration) and overall accuracy (Table 3 ). SC (

k{=}9

) matches CogRSI’s tokens.

Method	Hi	Open	TBench	Cog	Big	Mean
PercepToM (filtered story)	48.2	68.4	72.0	79.7	84.8	70.6
SC ( $k{=}6$ , full story)	62.2	75.0	73.9	85.5	89.2	77.2
PercepToM + SC + arbitration	67.5	74.1	74.0	85.7	89.0	78.1
CogRSI (annotated + plain)	69.3	75.8	74.1	84.9	90.3	78.9

Appendix

Table 12: Annotate or filter? Accuracy on the five multiple-choice benchmarks of the main test sample, three runs. The third row gives PercepToM’s filtered reading our arbitration: where its answer and self-consistency’s differ, the evidential rule of § 4.1 decides.

Backbone	CogRSI	SC ( $k{=}9$ )	SC ( $k{=}6$ )	Tokens
Qwen3.5-35B (main)	69.4	60.9	59.8	1.00
Qwen3.5-122B	70.4	64.2	64.2	0.98
Llama-3.3-70B	67.1	62.0	61.9	1.15
Qwen3.5-9B	60.9	52.3	51.4	0.99
GLM-4.6V-Flash	54.9	47.9	48.0	1.09

Appendix

Table 13: Token-matched self-consistency on every backbone. Mean of FANToM All ∗ , SimpleToM, Hi-ToM and BigToM (three runs). On FANToM, Llama-3.3, Qwen3.5-9B and GLM use CogRSI without close-split arbitration, as in Figure 9 b. Tokens: SC (

k{=}9

)’s tokens per question relative to CogRSI’s (SimpleToM, Hi-ToM, BigToM; main backbone: all main benchmarks).

		Development (final cohort)			Test
Seed	Admitted	None	Hand	Loop	Main
0	4/8	61.7	61.3	63.0	70.9
1	5/8	60.0	62.3	64.0	72.5
2	2/8	52.3	51.7	60.0	72.1
3	3/8	58.3	59.0	61.7	72.4
4	6/8	62.7	63.0	62.7	71.4

Appendix

Table 14: Principle-level RSI, one row per loop seed. Admitted: rounds whose candidate passed the registration rule. Development: accuracy on a final cohort of 300 unseen ExploreToM development questions with no principle (the annotation format only), the hand-written principle of CogRSI and the loop’s final principle. Test: one run of the reader with the seed’s principle (Table 3 ). Principles verbatim in Appendix C.2 .

Method	What is kept	Learned from	Access	Rebuild
Voyager	code skills	env. feedback	–	$\times$
Reflexion	verbal memory	task feedback	–	$\times$
ADAS, DGM	agent code	benchmark score	–	$\times$
GEPA	prompt	rollout reflection	–	$\times$
SymbolicToM	– (fixed)	hand-designed	belief graphs	$\times$
SimToM	– (fixed)	hand-designed	filtered story	$\times$

Appendix

Table 15: CogRSI against representative self-improving systems and ToM methods. What is kept : the artifact carried to later rounds; Access : how perceptual access enters reasoning; Rebuild : whether inheritance is tested against a rebuild with the same budget. Only CogRSI inherits across rounds and keeps the whole story while marking who perceived each part.

Backbone	Architecture	Weights	GPUs
Qwen3.5-35B-A3B (main)	MoE, 256 experts, 40 layers	FP8	1 $\times$ H200 / A6000
Qwen3.5-122B-A10B	MoE, 256 experts, 48 layers	FP8	1 $\times$ H200
Llama-3.3-70B-Instruct	dense, 80 layers	AWQ 4-bit	1 $\times$ H200 / 2 $\times$ A6000
GLM-4.6V-Flash	dense, 40 layers, text-only serving	BF16	1 $\times$ A6000
Qwen3.5-9B	dense, 32 layers	BF16	1 $\times$ H200 / A6000

Appendix

Table 16: Backbones and hardware (NVIDIA H200 NVL 141 GB, RTX A6000 48 GB). All served with vLLM 0.30.0.

Method	ToMi	Para	Nego	Omni	MMToM	MuMA	SIQ	Mean
Prompting
Native solver (floor)	69.3	78.6	64.6	61.8	39.2	44.3	62.5	60.0
Zero-shot	66.2	74.4	66.3	64.5	41.4	44.9	63.2	60.1
CoT	67.9	66.3	68.7	59.4	53.1	66.5	61.3	63.3
SC ( $k{=}6$ )	66.1	67.1	71.7	61.1	57.7	70.1	63.2	65.3
SC ( $k{=}9$ )	66.2	66.9	72.5	61.7	60.1	72.8	63.2	66.2

Appendix

Table 17: Seven test-only benchmarks (main backbone, main test sample, accuracy %, mean of three runs; CogRSI-L: six loop seeds). They were added after the first design was frozen and never used for development. Best in bold, second underlined. Here CogRSI is level with self-consistency, and SimToM leads on the ToMi family.

Qwen3.5-35B-A3B-FP8 (MoE 35B / 3B active, FP8, thinking off)
Method	FAN	Simple	Explore	Overall
	All ∗	Beh.+Judg.	E1+E2	(3/3)
Native solver (floor)	4.3	51.9	61.3	39.2
Prompting
Zero-shot	7.8 +3.5	60.9 +9.0	60.1 $-$ 1.2	42.9 +3.8
CoT	14.4 +10.1	68.4 +16.5	62.0 +0.7	48.2 +9.1

Appendix

Table 18: Full sealed test sets (RSI study, design frozen before its runs). Accuracy (%) per dataset, full sealed test sets of FANToM, SimpleToM and ExploreToM (October 6–9; before CogRSI v3 existed); smoke subsets excluded. Mean over three runs (CogRSI and baselines; GEPA: three optimizer seeds; Stage-1 tools: three loop seeds; CogRSI-L: one run per principle-loop seed). FANToM: official All ∗ on short contexts. SimpleToM: mean of behavior and judgment. ExploreToM: mean of the E1 (physical access) and E2 (communication access) partitions. Subscripts: gap to the floor (the same native solver with an empty tool library). Best in bold, runner-up underlined. Overall averages the datasets every row has (count in the header). ‡ PercepToM on FANToM is scored from its per-question results like every other system; the official scorer’s printed summary for these runs is higher and cannot be reproduced from the per-question results it writes (for every other system the two agree exactly). Per-metric detail in Table 21 ; every run in Table 24 .

Qwen3.5-35B-A3B-FP8 (MoE 35B / 3B active, FP8, thinking off)
Method	FAN	Simple	Explore	Hi	Open	TBench	Cog	Big	Overall
	All ∗	Beh.+Judg.	E1+E2	Acc.	Acc.	Acc.	Acc.	Acc.	(8/8)
Native solver (floor)	3.9	51.8	67.6	49.5	65.9	76.1	80.4	84.9	60.0
Prompting
Zero-shot	6.8 +2.9	58.0 +6.2	70.4 +2.8	54.2 +4.7	66.7 +0.8	74.7 $-$ 1.4	80.4 0.0	86.0 +1.2	62.2 +2.1
CoT	15.0 +11.1	68.6 +16.8	67.8 +0.2	48.5 $-$ 1.0	71.0 +5.1	74.5 $-$ 1.7	81.9 +1.5	88.7 +3.8	64.5 +4.5

Appendix

Table 19: Development sample: held-out sample 1. Accuracy (%) per dataset, held-out sample 1, the development set of CogRSI’s arbitration (not a test sample); smoke subsets excluded. Mean over three runs (CogRSI and baselines; GEPA: three optimizer seeds; Stage-1 tools: three loop seeds; CogRSI-L: one run per principle-loop seed). FANToM: official All ∗ on short contexts. SimpleToM: mean of behavior and judgment. ExploreToM: mean of the E1 (physical access) and E2 (communication access) partitions. Subscripts: gap to the floor (the same native solver with an empty tool library). Best in bold, runner-up underlined. Overall averages the datasets every row has (count in the header). Test results: Table 3 .

System	Run	FAN	Simple	Simple	Explore	Explore	Hi	Open	TBench	Cog	Big
		All ∗	Behav.	Judg.	E1	E2
Native solver	0	3.2	68.3	33.5	54.7	79.2	54.8	66.4	70.3	79.6	83.3
	1	3.7	68.3	33.5	54.7	81.2	56.0	66.8	70.1	79.6	83.5
	2	4.3	68.3	33.5	52.0	81.6	56.6	66.4	69.9	79.8	82.9
Zero-shot	0	7.5	75.4	47.3	51.6	77.6	59.2	68.2	72.1	80.0	85.1
	1	8.0	75.4	44.3	50.8	79.2	57.6	68.2	72.3	79.6	84.7

Appendix

Table 20: Every run behind the main test sample (Table 3 ) (accuracy %; FANToM: All ∗ over 187 target sets). Runs of static baselines are independent executions; GEPA runs are optimizer seeds.

	FANToM		SimpleToM		ExploreToM
Method	All ∗	Belief	Behav.	Judg.	E1	E2
Native solver	4.3	52.9	65.3	38.6	50.0	72.5
Zero-shot	7.8 +3.5	63.5 +10.5	72.0 +6.7	49.8 +11.2	45.8 $-$ 4.2	74.3 +1.8
CoT	14.4 +10.1	68.2 +15.2	78.0 +12.7	58.9 +20.3	44.4 $-$ 5.6	79.5 +7.0
SC ( $k{=}6$ )	18.2 +14.0	71.2 +18.3	78.5 +13.2	58.3 +19.8	47.2 $-$ 2.9	82.3 +9.8
Retrieval	2.8 $-$ 1.5	41.7 $-$ 11.2	61.4 $-$ 3.9	38.1 $-$ 0.5	56.7 +6.7	73.7 +1.2

Appendix

Table 21: Per-metric detail for FANToM, SimpleToM and ExploreToM (same runs as Table 18 ). Subscripts: gap to the floor. † Fewer than three runs. ‡ PercepToM on FANToM is scored from its per-question results like every other system; the official scorer’s printed summary for these runs is higher and cannot be reproduced from the per-question results it writes (for every other system the two agree exactly).

	FANToM		SimpleToM			ExploreToM
System	All ∗	Belief	Behav.	Judg.	Mental	E1	E2
Native solver	4.3	52.9	65.3	38.6	97.2	50.0	72.5
Zero-shot	7.8	63.5	72.0	49.8	95.0	45.8	74.3
CoT	14.4	68.2	78.0	58.9	98.0	44.4	79.5
SC ( $k{=}6$ )	18.2	71.2	78.5	58.3	98.1	47.2	82.3
Retrieval	2.8	41.7	61.4	38.1	89.4	56.7	73.7

Appendix

Table 22: All systems, including rows kept out of Table 18 : the original PercepToM adapter (v4) and the E2 branches (parent library plus new tools). Mental-state questions all have gold answer “No”, so a constant “No” scores 100%; this column is reported separately. † Fewer than three runs. ‡ PercepToM on FANToM is scored from its per-question results like every other system; the official scorer’s printed summary for these runs is higher and cannot be reproduced from the per-question results it writes (for every other system the two agree exactly).

Qwen3.5-122B-A10B (FP8), larger-model spot check
Method	FANToM All ∗	SimpleToM	Hi-ToM	BigToM	Overall
Zero-shot	3.0	70.0	68.8	85.9	56.9
CoT	11.6	78.5	55.6	89.0	58.7
SC ( $k{=}6$ )	18.2	80.8	68.0	89.9	64.2
PercepToM	6.2	88.2	48.0	86.2	57.2
CogRSI	25.3	94.0	72.4	89.7	70.4

Appendix

Table 23: Other backbones. Same systems, prompts and test samples as Table 3 ; mean of three runs. SimpleToM: mean of behavior and judgment. Best per backbone in bold. † Fewer than three runs finished. On FANToM, CogRSI’s close-split arbitration does not apply (dialogue questions), so its FANToM cells reuse the runs of the variant without it (Appendix C.3 ).

System	Run	All ∗	Belief	Behav.	Judg.	Mental	E1	E2
Native solver	rep. 0	4.1	53.5	65.3	38.6	97.2	50.1	72.6
	rep. 1	4.5	52.5	65.3	38.6	97.2	50.3	72.3
	rep. 2	4.3	52.7	65.3	38.6	97.2	49.6	72.7
Zero-shot	rep. 0	7.5	63.1	71.7	49.3	95.0	46.0	74.3
	rep. 1	8.1	63.8	72.3	49.9	95.1	45.7	74.4
	rep. 2	7.9	63.5	71.8	50.2	94.8	45.8	74.3

Appendix

Table 24: Every run behind Tables 18 – 22 (accuracy %). Repeats of static baselines are independent executions, not controlled sampling seeds. – = not run.

	Construct	$n$	SC	PercepToM	Lib-A	CogRSI	CogRSI $-$ SC [95% CI]	within benchmark
Epistemic	Belief	3438	73.8	71.8	63.6	77.7	+3.9 [+2.5, +5.4]	+1.2 [+0.3, +2.1]
	Knowledge	3095	84.6	65.4	77.3	85.7	+1.1 [+0.2, +2.0]	$-$ 0.8 [ $-$ 1.8, +0.1]
Motivational	Desire	72	81.5	79.2	81.0	80.1	$-$ 1.4 [ $-$ 3.7, +0.5]	$-$ 1.1 [ $-$ 3.5, +0.9]
	Intention	118	88.1	89.0	85.9	88.1	0.0 [ $-$ 1.7, +1.8]	+0.2 [ $-$ 1.4, +2.0]
Affective	Emotion	151	78.1	76.2	75.3	75.7	$-$ 2.4 [ $-$ 5.4, +0.2]	$-$ 2.4 [ $-$ 5.4, +0.2]
Pragmatic	Non-literal	177	79.8	82.7	76.3	78.2	$-$ 1.7 [ $-$ 3.7, +0.2]	$-$ 1.7 [ $-$ 3.5, $-$ 0.1]

Appendix

Table 25: Accuracy by ATOMS construct ( Beaudoin et al., 2020 ) on all test questions of the main backbone (main test sample, three runs each). The difference to self-consistency is bootstrapped over stories/conversations (2,000 resamples); within benchmark centres each question’s difference on its benchmark’s mean difference and resamples stories within benchmarks. Bold intervals exclude zero. Constructs with fewer than 50 questions (perception, other) are omitted.

Benchmark	Strongest baseline	CogRSI $-$ best	Holm	CogRSI $-$ SC
FANToM All ∗	PercepToM + SC (belief) (23.7)	+4.3 [+0.2, +8.6]	–	+10.3 [+5.2, +15.6]
SimpleToM	PercepToM (87.5)	+2.5 [ $-$ 0.8, +6.0]	–	+19.9 [+16.1, +23.7]
ExploreToM	SimToM + SC (76.9)	$-$ 4.3 [ $-$ 8.3, $-$ 0.6]	–	$-$ 0.1 [ $-$ 2.1, +1.9]
Hi-ToM	SC ( $k{=}9$ ) (64.9)	+4.4 [+1.2, +7.6]	–	+7.1 [+4.2, +10.0]
OpenToM	SimToM + SC (76.5)	$-$ 0.7 [ $-$ 3.0, +1.5]	–	+0.8 [ $-$ 0.9, +2.7]
ToMBench	PercepToM + SC (74.3)	$-$ 0.1 [ $-$ 1.6, +1.3]	–	+0.2 [ $-$ 1.1, +1.5]

Appendix

Table 26: Paired differences (points) between CogRSI and the strongest baseline of each benchmark (its score in brackets) and self-consistency, main backbone, three runs per system. 95% intervals from 2,000 bootstrap resamples of stories or conversations; bold intervals exclude zero. Strongest baseline: the highest Table 3 score among all baselines, including SimToM + SC and PercepToM + SC. Holm: significant after Holm correction over the eight benchmarks (bootstrap

p

-values, familywise 0.05). An interval that includes zero is not evidence of equivalence.

Panel of Figure 12 — Figure 12: RSI over principles. (a) Incumbent’s accuracy on new development questions by round (mean and range over 6 seeds; round 0 admits the first principle). (b) 300 new development questions on seen stories, per seed (no, hand-written, loop-written principle). (c) Test accuracy per benchmark, Lib-A $\to$ CogRSI-L, main test sample. The loop’s rule beats ours on development questions in most seeds, not on test.

Panel of Figure 13 — Figure 13: Grounded reading. (a) Belief and knowledge accuracy. (b) Narrative accuracy by reading. (c) Conflicts settled correctly. Grounding helps epistemic questions, and two readings with the evidential rule beat one.

		Percept-ToMi (576 questions)					FANToM (62 dialogues)
Method	Perceivers	FB1	FB2	TB1	TB2	All	All ∗	Belief
SC ( $k{=}6$ )	model	84.8	92.1	56.6	5.6	59.8	—	—
	gold	85.7	92.1	56.2	7.2	60.4	—	—
PercepToM	model	89.0	1.9	56.9	47.2	48.8	2.2	82.3
	gold	87.8	72.7	57.3	19.0	59.3	8.2	90.6
CogRSI	model	89.9	92.6	56.6	2.8	60.5	26.0	79.3

Appendix

Table 27: Perception or reasoning? Each system with its own perception call (model) and with the gold perceivers of Percept-ToMi and Percept-FANToM in place of that call (gold; nothing else changes). FB/TB: false/true belief, first/second order. SC never calls perception, so its gold rows are a control. Mean of three runs (CogRSI-L: six loop seeds). Gold perceivers lift both readers most on true-belief questions. On FANToM, belief accuracy rises and All ∗ does not.

Panel of Figure 14 — Figure 14: Sensitivity of CogRSI’s fixed choices (narrative test benchmarks, macro mean, all runs). (a) Accuracy of a majority over the first $a$ annotated and $p$ plain samples of the vote-only runs; ours (4+2) outlined. (b) Close-split arbitration restricted to stricter thresholds (looser ones would need new arbiter calls). Robustness checks on test data, not selections. Mixes with both readings are best; ours scores 79.9 (best mix: 79.9), and our thresholds score highest.

Story. Miss Zhou is a professional in the competitive advertising industry. At an industry gathering, someone asks her about the details of an innovative advertising strategy her company is about to launch. Miss Zhou replies that her team develops a brand new marketing method that changes the industry landscape, but currently cannot reveal more information. She also emphasizes that this is a major breakthrough in the industry. After the industry gathering ends, Miss Zhou lets her team start planning the innovative strategy within this week. Question. Is what Miss Zhou says true? (A) Yes (B) No. Gold: B.
Method	Ans.	What decides it (from the run logs)
SC ( $k{=}6$ )	A $\times$	majority of six samples is A in all three runs
PercepToM	A $\times$	“target character not found” $\to$ zero-shot fallback, A (all runs)
Lib-A	A $\times$	inherited tools; A in all three seeds
Annotated	B $\surd$	run 0: three of four annotated samples answer B
Plain	A $\times$	run 0: both plain samples A; the 3–3 tie keeps the first sampled answer, A

Appendix

Table 28: Case study : a ToMBench item every baseline misses and CogRSI gets right by arbitration.

Adaptive Reasoning in Language Models

See all 49 papers

Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness
Mar 24, 2026Abhinaba Basu, Pavan ChakrabortyLanguage Model Reasoning Evaluation Structured Reasoning in Language Models
Indian Institute of Information Technology Allahabad (IIITA) · National Institute of Electronics and Information Technology (NIELIT)
Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking
Jul 1, 2026Chia-Hsuan Lee, Sihui Dai, Mingyang Zhou +4Reasoning Trace Analysis Overthinking in Language Models
Capital One
BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Mar 16, 2026Yuzhe TangCommonsense Reasoning Language Model Reasoning Evaluation
Georgia Institute of Technology

Mar 24, 2026J/K move · Enter open · S save

Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness

Abhinaba Basu, Pavan Chakraborty

Indian Institute of Information Technology Allahabad (IIITA) · National Institute of Electronics and Information Technology (NIELIT)

Language Model Reasoning Evaluation Structured Reasoning in Language Models CoT Faithfulness Faithfulness of Language Model Reasoning Faithfulness of Language Model Explanations RL for Language Model Reasoning+1

Open paper AskPDF arXiv

Language models increasingly show their work by writing step-by-step reasoning before answering. But are these steps genuinely used, or is the answer rigid - fixed before reasoning begins? We introduce the Step-Level Reasoning Capacity (SLRC) metric and prove it is a consistent causal estimator (Theorem 1). We propose LC-CoSR, a training method with Lyapunov stability guarantees that directly reduces rigidity. Evaluating 16 frontier models (o4-mini, GPT-5.4, Claude Opus, Grok-4, DeepSeek-R1, Gemini 2.5 Pro, and others) across six domains at N=133-500, we find reasoning falls into three modes. OpenAI's o4-mini shows 74-88% step necessity on five of six tasks (73.8-88.3%) - the highest SLRC in our study. The critical differentiator is RL-based reasoning training, not thinking tokens: Grok-4's reasoning mode shows lower faithfulness than its non-reasoning mode (1.4% vs 7.2% necessity). We discover a faithfulness paradox - high-SLRC models are more susceptible to sycophancy - and propose the Reasoning Integrity Score (RIS = SLRC x (1-Sycophancy)), which significantly predicts error detection (rho=0.66, p=0.026). LC-CoSR achieves 2.6x less negative reward than FARL and CSR baselines without external model dependencies.

Abstract

Adaptive Reasoning in Language Models

Explore similar work

Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness

Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models