Can a language model learn better ways to reason about other people's minds by improving its own strategies, and are the strategies it learns actually right? We study this with cogrsi. A grounded reader marks who saw each event in a story and uses a short written principle to settle disagreements between its readings. With a principle written by experts, it beats self-consistency at the same token cost. A self-improvement loop then writes the principle itself. It keeps an update only if accuracy rises by a calibrated margin, and in one variant only if it also passes counterfactual test pairs. The learned principles transfer: on 800 BigToM pairs that no system had seen, each of them scores higher than the expert principle. We then check what the learned rules actually say. Each rule is translated into code and run on thousands of small stories with known answers. Most of them give a wrong answer in at least one situation they claim to cover, for example by treating a character who is in the room as knowing what happened even while asleep. Accuracy does not reveal which rules are wrong, and the counterfactual test does not fix this. A strategy can help in practice and still be wrong as a rule, so both need checking. Our code and data are publicly available.
Figures & tables
Figure 1: The task : a strategy learned on one generator must track who saw what on benchmarks it never saw.
Figure 2: Overview. Rows: what persists, transfer, cognitive validity. Existing self-improving and ToM methods leave them open. Our suite measures them, and CogRSI addresses them.
Figure 3: CogRSI. (A) RSI over procedures writes tools that export perceptual access. (B) RSI over principles keeps a library of conditioned strategies and admits an update only if it passes the accuracy margin and, in the gated variant, the counterfactual gate. (C) The reader applies the strategies at test time. The semantic audit (§ 4.3 ) checks what the admitted strategies assert.
Accuracy
Pair validity
Audit
Principle written by
Registration
Dev.
Test Main
Own pairs
BigToM-U
Invalid rules (%)
no principle
none
59.7
70.1
91.7
94.0
n.a.
hand (expert)
none
59.7
73.1
91.7
94.5
n.r.
accuracy-only loop
1-pt
63.8
71.8
91.5
95.9
89.7
GEPA (same slot)
Pareto
own val.
67.0
75.3
84.4
n.r.
accuracy-only library
3-pt
66.1
71.5 †
91.2 †
96.8
85.2 †
Table 1: Who writes the principle. Same reader, backbone and sampling; only the principle changes. Registration: accuracy margin in points, with the counterfactual gate where stated (Pareto: GEPA’s own selection). Dev.: 300 unseen ExploreToM development questions per seed. Test Main: main test sample, eight benchmarks. Own pairs: 320 held-out pairs from our generator. BigToM-U: 800 BigToM pairs in conditions no system had seen. Invalid rules: share of the arm’s proposed rules with a formal counterexample (§ 4.3 ). Seeds: 6 for loops and libraries, 3 for GEPA and the single principle; † seeds 0–2 only. n.r.: not run; n.a.: not applicable. Every learned principle beats the expert one on BigToM-U; the gate changes validity on its own pairs only.
Figure 4: Registration as the loop sees it. Every candidate that reached registration, in order (four per round), placed at the incumbent’s running level plus its mean trial gain. Kept candidates are coloured by the semantic audit (§ 4.3 ). Levels are the loops’ own trial estimates, which overstate held-out gains (Table 5 ). Each loop keeps climbing on its own trials, and most of what it keeps is invalid.
Figure 5: RSI over procedures. (a) Candidate tools per E1 run, filled if they encode access. (b) ExploreToM accuracy, native solver → Lib-A. Every run fills the access slot, and its library helps only near its source.
Figure 6: Validity is a separate target. (a) Test Main against pair validity on the 320 held-out counterfactual pairs, means over seeds or runs. (b) Accuracy by access variant on the same items. Accuracy-only principles cluster near the reader with no principle, GEPA falls off both axes, and the gated library moves up but not right.
Figure 7: Semantic audit of 573 evolved rules. (a) Share of decided rules that are invalid among all proposed rules and among those each registration rule would admit, replayed on the same candidates. (b) Trial gain of valid and invalid rules. (c) Invalid and valid entries in the libraries by round, summed over seeds. Accuracy does not tell valid from invalid rules, and the gate does not change what is admitted.
Figure 9
(A) Main test sample (held-out sample 2): eight benchmarks, three backbones
Table 3: Main results. (A) Main test sample, three backbones. (B) Confirmatory sample. Accuracy (%) is the mean of runs: three in A, and in B three for CogRSI and SC ( k=9 ), six loop seeds for CogRSI-L, one for the rest. Main averages the eight. Subscripts are gaps to the block floor. Bold is best, underline second. FAN is FANToM All ∗ . CogRSI has the best Main in every block. ‡ FANToM only. —: does not apply.
Figure 11
Figure 10: Case study. A ToMBench egocentric lie, abridged (full text and logs in Table 28 ). Chips are CogRSI’s six samples in run 0. The readings disagree, and arbitration picks the right answer.
Registration
Adm.
Prec.
Harm.
Recall
Missed
1-pt margin
46
63.6
31.8
23.7
22.9
3-pt margin
25
83.3
16.7
21.7
29.2
3-pt + gate (single)
25
83.3
16.7
17.2
25.0
3-pt + gate (library)
25
83.3
16.7
13.0
43.8
Table 5: Selection quality (%, development data). Every candidate scored on 120 held-out questions. Adm.: rounds with an admission; Prec./Harm.: admitted updates that help/hurt; Recall: helpful candidates admitted; Missed: rounds with a helpful candidate and no admission. The calibrated margin, not the gate, makes selection precise.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Run
r0
r1
r2
r3
r4
r5
r6
r7
Adm.
Tools
Access
Nesting
Parsing
A s0
50.5
46.6
51.5
45.8
49.7
50.3
47.6
52.5 ⋆
1
32
32
16
10
A s1
41.6
44.5
46.0
53.1
46.9
50.1
43.2
51.3
0
30
30
16
12
A s2
54.4
54.5
55.5
47.6 ⋆
52.4
52.7
59.2
50.8
1
31
29
17
23
C s0
52.6
46.3
51.7
46.9 ⋆
50.5 ⋆
48.3
47.8
51.2
2
31
31
17
29
C s1
42.8
46.3
45.8
50.9
46.0 ⋆
48.6 ⋆
44.4
53.8
3
31
31
13
28
C s2
53.2
53.6 ⋆
52.9 ⋆
47.4
51.3
54.4 ⋆
58.4
44.3 ⋆
5
28
28
15
0
Appendix
Table 6: Stage 1 (E1) runs round by round: development accuracy of the solver with the current library (A: inherited, C: rebuilt; ⋆ : tools admitted in that round) and the number of admitted tools (a round can admit more than one); right: the candidate tools each run wrote and how many encode perceptual access, nested belief or story parsing (a tool can encode several). Admission is rare, so the inheriting arm rarely has a library to inherit.
Figure 15
Sample
Benchmark
n
without
with
Held-out 1
SimpleToM
501
91.8
93.2
Hi-ToM
500
66.4
66.8
OpenToM
500
77.4
77.0
ToMBench
500
76.4
77.2
CogToM
500
85.4
85.6
BigToM
504
89.9
90.3
Appendix
Table 7: Development results without and with the arbitration of close splits, on the same six samples per question on held-out sample 1, the development set of the design without it. Rejected candidates are listed in the text.
Main backbone
Other backbones (FAN, Simple, Hi, Big)
Design
Main sample
FANToM All ∗
Llama-70B
GLM-4.6V
Qwen3.5-9B
SC ( k=6 )
68.3
17.6
61.9
48.0
51.4
Vote over both views (v3)
72.2
28.7
65.5
54.7
62.3
+ conflict arbitration (v5)
72.6
27.3
67.2
53.6
60.3
+ close-split arbitration = CogRSI (v6)
73.1
28.0
67.1
54.9
60.9
Appendix
Table 8: CogRSI’s design variants on every test condition (three runs each; per-benchmark results and every run in Table 20 ). Each design was frozen before its test runs; the close-split arbitration was added after the first test run of the design without it and was selected on development data only (§ C.3 ).
Main test sample
Pair validity by access variant (eval split)
Principle
FAN
Simple
Explore
Hi
Open
TBench
Cog
Big
Main
saw
left
present
told
lied
looks
All
no principle
23.2
73.3
72.6
69.1
74.6
74.2
84.1
90.1
70.1
100.0
100.0
100.0
100.0
99.5
34.2
91.7
hand (expert)
28.0
90.0
72.5
69.3
75.8
74.1
84.9
90.3
73.1
100.0
100.0
100.0
100.0
100.0
33.3
91.7
accuracy-only loop
25.3
81.7
74.0
70.0
76.5
72.8
84.7
89.6
71.8
100.0
99.1
98.7
99.6
98.8
41.2
91.5
GEPA (same slot)
23.2
71.7
84.7
62.7
71.5
71.8
77.8
72.8
67.0
100.0
73.1
54.1
99.5
96.5
95.0
75.3
accuracy-only library
23.7
81.3
76.0
70.3
75.5
72.6
84.0
88.7
71.5
100.0
95.6
99.6
99.5
98.5
51.7
91.2
Appendix
Table 9: Arms by benchmark and by access variant (main backbone, means over seeds or runs). Variant columns give item accuracy on the evaluation split’s items of that variant (narrative and dialogue pooled); All is pair validity.
Table 10: Samples per benchmark: questions (stories or groups in parentheses; FANToM: All ∗ sets and conversations). Held-out sample 1 is CogRSI’s development set; the main test sample is held-out sample 2; the confirmatory sample is held-out sample 3. SimpleToM: behavior and judgment questions scored.
Per question
Accuracy
Method
Calls
In (k)
Out (k)
Main
CoT
1.0
0.35
0.26
64.8
SC ( k=6 )
6.0
2.11
1.67
68.3
SC ( k=9 )
9.0
3.17
2.50
68.7
PercepToM
1.4
0.47
0.08
63.8
CogRSI
7.3
3.33
2.36
73.1
Appendix
Table 11: Cost per question (multiple-choice test benchmarks, main backbone; all calls, including perception, retries and arbitration) and overall accuracy (Table 3 ). SC ( k=9 ) matches CogRSI’s tokens.
Method
Hi
Open
TBench
Cog
Big
Mean
PercepToM (filtered story)
48.2
68.4
72.0
79.7
84.8
70.6
SC ( k=6 , full story)
62.2
75.0
73.9
85.5
89.2
77.2
PercepToM + SC + arbitration
67.5
74.1
74.0
85.7
89.0
78.1
CogRSI (annotated + plain)
69.3
75.8
74.1
84.9
90.3
78.9
Appendix
Table 12: Annotate or filter? Accuracy on the five multiple-choice benchmarks of the main test sample, three runs. The third row gives PercepToM’s filtered reading our arbitration: where its answer and self-consistency’s differ, the evidential rule of § 4.1 decides.
Backbone
CogRSI
SC ( k=9 )
SC ( k=6 )
Tokens
Qwen3.5-35B (main)
69.4
60.9
59.8
1.00
Qwen3.5-122B
70.4
64.2
64.2
0.98
Llama-3.3-70B
67.1
62.0
61.9
1.15
Qwen3.5-9B
60.9
52.3
51.4
0.99
GLM-4.6V-Flash
54.9
47.9
48.0
1.09
Appendix
Table 13: Token-matched self-consistency on every backbone. Mean of FANToM All ∗ , SimpleToM, Hi-ToM and BigToM (three runs). On FANToM, Llama-3.3, Qwen3.5-9B and GLM use CogRSI without close-split arbitration, as in Figure 9 b. Tokens: SC ( k=9 )’s tokens per question relative to CogRSI’s (SimpleToM, Hi-ToM, BigToM; main backbone: all main benchmarks).
Development (final cohort)
Test
Seed
Admitted
None
Hand
Loop
Main
0
4/8
61.7
61.3
63.0
70.9
1
5/8
60.0
62.3
64.0
72.5
2
2/8
52.3
51.7
60.0
72.1
3
3/8
58.3
59.0
61.7
72.4
4
6/8
62.7
63.0
62.7
71.4
Appendix
Table 14: Principle-level RSI, one row per loop seed. Admitted: rounds whose candidate passed the registration rule. Development: accuracy on a final cohort of 300 unseen ExploreToM development questions with no principle (the annotation format only), the hand-written principle of CogRSI and the loop’s final principle. Test: one run of the reader with the seed’s principle (Table 3 ). Principles verbatim in Appendix C.2 .
Method
What is kept
Learned from
Access
Rebuild
Voyager
code skills
env. feedback
–
×
Reflexion
verbal memory
task feedback
–
×
ADAS, DGM
agent code
benchmark score
–
×
GEPA
prompt
rollout reflection
–
×
SymbolicToM
– (fixed)
hand-designed
belief graphs
×
SimToM
– (fixed)
hand-designed
filtered story
×
Appendix
Table 15: CogRSI against representative self-improving systems and ToM methods. What is kept : the artifact carried to later rounds; Access : how perceptual access enters reasoning; Rebuild : whether inheritance is tested against a rebuild with the same budget. Only CogRSI inherits across rounds and keeps the whole story while marking who perceived each part.
Backbone
Architecture
Weights
GPUs
Qwen3.5-35B-A3B (main)
MoE, 256 experts, 40 layers
FP8
1 × H200 / A6000
Qwen3.5-122B-A10B
MoE, 256 experts, 48 layers
FP8
1 × H200
Llama-3.3-70B-Instruct
dense, 80 layers
AWQ 4-bit
1 × H200 / 2 × A6000
GLM-4.6V-Flash
dense, 40 layers, text-only serving
BF16
1 × A6000
Qwen3.5-9B
dense, 32 layers
BF16
1 × H200 / A6000
Appendix
Table 16: Backbones and hardware (NVIDIA H200 NVL 141 GB, RTX A6000 48 GB). All served with vLLM 0.30.0.
Method
ToMi
Para
Nego
Omni
MMToM
MuMA
SIQ
Mean
Prompting
Native solver (floor)
69.3
78.6
64.6
61.8
39.2
44.3
62.5
60.0
Zero-shot
66.2
74.4
66.3
64.5
41.4
44.9
63.2
60.1
CoT
67.9
66.3
68.7
59.4
53.1
66.5
61.3
63.3
SC ( k=6 )
66.1
67.1
71.7
61.1
57.7
70.1
63.2
65.3
SC ( k=9 )
66.2
66.9
72.5
61.7
60.1
72.8
63.2
66.2
Appendix
Table 17: Seven test-only benchmarks (main backbone, main test sample, accuracy %, mean of three runs; CogRSI-L: six loop seeds). They were added after the first design was frozen and never used for development. Best in bold, second underlined. Here CogRSI is level with self-consistency, and SimToM leads on the ToMi family.
Table 18: Full sealed test sets (RSI study, design frozen before its runs). Accuracy (%) per dataset, full sealed test sets of FANToM, SimpleToM and ExploreToM (October 6–9; before CogRSI v3 existed); smoke subsets excluded. Mean over three runs (CogRSI and baselines; GEPA: three optimizer seeds; Stage-1 tools: three loop seeds; CogRSI-L: one run per principle-loop seed). FANToM: official All ∗ on short contexts. SimpleToM: mean of behavior and judgment. ExploreToM: mean of the E1 (physical access) and E2 (communication access) partitions. Subscripts: gap to the floor (the same native solver with an empty tool library). Best in bold, runner-up underlined. Overall averages the datasets every row has (count in the header). ‡ PercepToM on FANToM is scored from its per-question results like every other system; the official scorer’s printed summary for these runs is higher and cannot be reproduced from the per-question results it writes (for every other system the two agree exactly). Per-metric detail in Table 21 ; every run in Table 24 .
Table 19: Development sample: held-out sample 1. Accuracy (%) per dataset, held-out sample 1, the development set of CogRSI’s arbitration (not a test sample); smoke subsets excluded. Mean over three runs (CogRSI and baselines; GEPA: three optimizer seeds; Stage-1 tools: three loop seeds; CogRSI-L: one run per principle-loop seed). FANToM: official All ∗ on short contexts. SimpleToM: mean of behavior and judgment. ExploreToM: mean of the E1 (physical access) and E2 (communication access) partitions. Subscripts: gap to the floor (the same native solver with an empty tool library). Best in bold, runner-up underlined. Overall averages the datasets every row has (count in the header). Test results: Table 3 .
System
Run
FAN
Simple
Simple
Explore
Explore
Hi
Open
TBench
Cog
Big
All ∗
Behav.
Judg.
E1
E2
Native solver
0
3.2
68.3
33.5
54.7
79.2
54.8
66.4
70.3
79.6
83.3
1
3.7
68.3
33.5
54.7
81.2
56.0
66.8
70.1
79.6
83.5
2
4.3
68.3
33.5
52.0
81.6
56.6
66.4
69.9
79.8
82.9
Zero-shot
0
7.5
75.4
47.3
51.6
77.6
59.2
68.2
72.1
80.0
85.1
1
8.0
75.4
44.3
50.8
79.2
57.6
68.2
72.3
79.6
84.7
Appendix
Table 20: Every run behind the main test sample (Table 3 ) (accuracy %; FANToM: All ∗ over 187 target sets). Runs of static baselines are independent executions; GEPA runs are optimizer seeds.
FANToM
SimpleToM
ExploreToM
Method
All ∗
Belief
Behav.
Judg.
E1
E2
Native solver
4.3
52.9
65.3
38.6
50.0
72.5
Zero-shot
7.8 +3.5
63.5 +10.5
72.0 +6.7
49.8 +11.2
45.8 − 4.2
74.3 +1.8
CoT
14.4 +10.1
68.2 +15.2
78.0 +12.7
58.9 +20.3
44.4 − 5.6
79.5 +7.0
SC ( k=6 )
18.2 +14.0
71.2 +18.3
78.5 +13.2
58.3 +19.8
47.2 − 2.9
82.3 +9.8
Retrieval
2.8 − 1.5
41.7 − 11.2
61.4 − 3.9
38.1 − 0.5
56.7 +6.7
73.7 +1.2
Appendix
Table 21: Per-metric detail for FANToM, SimpleToM and ExploreToM (same runs as Table 18 ). Subscripts: gap to the floor. † Fewer than three runs. ‡ PercepToM on FANToM is scored from its per-question results like every other system; the official scorer’s printed summary for these runs is higher and cannot be reproduced from the per-question results it writes (for every other system the two agree exactly).
FANToM
SimpleToM
ExploreToM
System
All ∗
Belief
Behav.
Judg.
Mental
E1
E2
Native solver
4.3
52.9
65.3
38.6
97.2
50.0
72.5
Zero-shot
7.8
63.5
72.0
49.8
95.0
45.8
74.3
CoT
14.4
68.2
78.0
58.9
98.0
44.4
79.5
SC ( k=6 )
18.2
71.2
78.5
58.3
98.1
47.2
82.3
Retrieval
2.8
41.7
61.4
38.1
89.4
56.7
73.7
Appendix
Table 22: All systems, including rows kept out of Table 18 : the original PercepToM adapter (v4) and the E2 branches (parent library plus new tools). Mental-state questions all have gold answer “No”, so a constant “No” scores 100%; this column is reported separately. † Fewer than three runs. ‡ PercepToM on FANToM is scored from its per-question results like every other system; the official scorer’s printed summary for these runs is higher and cannot be reproduced from the per-question results it writes (for every other system the two agree exactly).
Method
FANToM All ∗
SimpleToM
Hi-ToM
BigToM
Overall
Qwen3.5-122B-A10B (FP8), larger-model spot check
Zero-shot
3.0
70.0
68.8
85.9
56.9
CoT
11.6
78.5
55.6
89.0
58.7
SC ( k=6 )
18.2
80.8
68.0
89.9
64.2
PercepToM
6.2
88.2
48.0
86.2
57.2
CogRSI
25.3
94.0
72.4
89.7
70.4
Appendix
Table 23: Other backbones. Same systems, prompts and test samples as Table 3 ; mean of three runs. SimpleToM: mean of behavior and judgment. Best per backbone in bold. † Fewer than three runs finished. On FANToM, CogRSI’s close-split arbitration does not apply (dialogue questions), so its FANToM cells reuse the runs of the variant without it (Appendix C.3 ).
System
Run
All ∗
Belief
Behav.
Judg.
Mental
E1
E2
Native solver
rep. 0
4.1
53.5
65.3
38.6
97.2
50.1
72.6
rep. 1
4.5
52.5
65.3
38.6
97.2
50.3
72.3
rep. 2
4.3
52.7
65.3
38.6
97.2
49.6
72.7
Zero-shot
rep. 0
7.5
63.1
71.7
49.3
95.0
46.0
74.3
rep. 1
8.1
63.8
72.3
49.9
95.1
45.7
74.4
rep. 2
7.9
63.5
71.8
50.2
94.8
45.8
74.3
Appendix
Table 24: Every run behind Tables 18 – 22 (accuracy %). Repeats of static baselines are independent executions, not controlled sampling seeds. – = not run.
Construct
n
SC
PercepToM
Lib-A
CogRSI
CogRSI − SC [95% CI]
within benchmark
Epistemic
Belief
3438
73.8
71.8
63.6
77.7
+3.9 [+2.5, +5.4]
+1.2 [+0.3, +2.1]
Knowledge
3095
84.6
65.4
77.3
85.7
+1.1 [+0.2, +2.0]
− 0.8 [ − 1.8, +0.1]
Motivational
Desire
72
81.5
79.2
81.0
80.1
− 1.4 [ − 3.7, +0.5]
− 1.1 [ − 3.5, +0.9]
Intention
118
88.1
89.0
85.9
88.1
0.0 [ − 1.7, +1.8]
+0.2 [ − 1.4, +2.0]
Affective
Emotion
151
78.1
76.2
75.3
75.7
− 2.4 [ − 5.4, +0.2]
− 2.4 [ − 5.4, +0.2]
Pragmatic
Non-literal
177
79.8
82.7
76.3
78.2
− 1.7 [ − 3.7, +0.2]
− 1.7 [ − 3.5, − 0.1]
Appendix
Table 25: Accuracy by ATOMS construct ( Beaudoin et al., 2020 ) on all test questions of the main backbone (main test sample, three runs each). The difference to self-consistency is bootstrapped over stories/conversations (2,000 resamples); within benchmark centres each question’s difference on its benchmark’s mean difference and resamples stories within benchmarks. Bold intervals exclude zero. Constructs with fewer than 50 questions (perception, other) are omitted.
Benchmark
Strongest baseline
CogRSI − best
Holm
CogRSI − SC
FANToM All ∗
PercepToM + SC (belief) (23.7)
+4.3 [+0.2, +8.6]
–
+10.3 [+5.2, +15.6]
SimpleToM
PercepToM (87.5)
+2.5 [ − 0.8, +6.0]
–
+19.9 [+16.1, +23.7]
ExploreToM
SimToM + SC (76.9)
− 4.3 [ − 8.3, − 0.6]
–
− 0.1 [ − 2.1, +1.9]
Hi-ToM
SC ( k=9 ) (64.9)
+4.4 [+1.2, +7.6]
–
+7.1 [+4.2, +10.0]
OpenToM
SimToM + SC (76.5)
− 0.7 [ − 3.0, +1.5]
–
+0.8 [ − 0.9, +2.7]
ToMBench
PercepToM + SC (74.3)
− 0.1 [ − 1.6, +1.3]
–
+0.2 [ − 1.1, +1.5]
Appendix
Table 26: Paired differences (points) between CogRSI and the strongest baseline of each benchmark (its score in brackets) and self-consistency, main backbone, three runs per system. 95% intervals from 2,000 bootstrap resamples of stories or conversations; bold intervals exclude zero. Strongest baseline: the highest Table 3 score among all baselines, including SimToM + SC and PercepToM + SC. Holm: significant after Holm correction over the eight benchmarks (bootstrap p -values, familywise 0.05). An interval that includes zero is not evidence of equivalence.
Figure 12: RSI over principles. (a) Incumbent’s accuracy on new development questions by round (mean and range over 6 seeds; round 0 admits the first principle). (b) 300 new development questions on seen stories, per seed (no, hand-written, loop-written principle). (c) Test accuracy per benchmark, Lib-A → CogRSI-L, main test sample. The loop’s rule beats ours on development questions in most seeds, not on test.
Figure 13: Grounded reading. (a) Belief and knowledge accuracy. (b) Narrative accuracy by reading. (c) Conflicts settled correctly. Grounding helps epistemic questions, and two readings with the evidential rule beat one.
Percept-ToMi (576 questions)
FANToM (62 dialogues)
Method
Perceivers
FB1
FB2
TB1
TB2
All
All ∗
Belief
SC ( k=6 )
model
84.8
92.1
56.6
5.6
59.8
—
—
gold
85.7
92.1
56.2
7.2
60.4
—
—
PercepToM
model
89.0
1.9
56.9
47.2
48.8
2.2
82.3
gold
87.8
72.7
57.3
19.0
59.3
8.2
90.6
CogRSI
model
89.9
92.6
56.6
2.8
60.5
26.0
79.3
Appendix
Table 27: Perception or reasoning? Each system with its own perception call (model) and with the gold perceivers of Percept-ToMi and Percept-FANToM in place of that call (gold; nothing else changes). FB/TB: false/true belief, first/second order. SC never calls perception, so its gold rows are a control. Mean of three runs (CogRSI-L: six loop seeds). Gold perceivers lift both readers most on true-belief questions. On FANToM, belief accuracy rises and All ∗ does not.
Figure 14: Sensitivity of CogRSI’s fixed choices (narrative test benchmarks, macro mean, all runs). (a) Accuracy of a majority over the first a annotated and p plain samples of the vote-only runs; ours (4+2) outlined. (b) Close-split arbitration restricted to stricter thresholds (looser ones would need new arbiter calls). Robustness checks on test data, not selections. Mixes with both readings are best; ours scores 79.9 (best mix: 79.9), and our thresholds score highest.
Story. Miss Zhou is a professional in the competitive advertising industry. At an industry gathering, someone asks her about the details of an innovative advertising strategy her company is about to launch. Miss Zhou replies that her team develops a brand new marketing method that changes the industry landscape, but currently cannot reveal more information. She also emphasizes that this is a major breakthrough in the industry. After the industry gathering ends, Miss Zhou lets her team start planning the innovative strategy within this week. Question. Is what Miss Zhou says true? (A) Yes (B) No. Gold: B.
Method
Ans.
What decides it (from the run logs)
SC ( k=6 )
A ×
majority of six samples is A in all three runs
PercepToM
A ×
“target character not found” → zero-shot fallback, A (all runs)
Lib-A
A ×
inherited tools; A in all three seeds
Annotated
B √
run 0: three of four annotated samples answer B
Plain
A ×
run 0: both plain samples A; the 3–3 tie keeps the first sampled answer, A
Appendix
Table 28: Case study : a ToMBench item every baseline misses and CogRSI gets right by arbitration.
Language models increasingly show their work by writing step-by-step reasoning before answering. But are these steps genuinely used, or is the answer rigid - fixed before reasoning begins? We introduce the Step-Level Reasoning Capacity (SLRC) metric and prove it is a consistent causal estimator (Theorem 1). We propose LC-CoSR, a training method with Lyapunov stability guarantees that directly reduces rigidity. Evaluating 16 frontier models (o4-mini, GPT-5.4, Claude Opus, Grok-4, DeepSeek-R1, Gemini 2.5 Pro, and others) across six domains at N=133-500, we find reasoning falls into three modes. OpenAI's o4-mini shows 74-88% step necessity on five of six tasks (73.8-88.3%) - the highest SLRC in our study. The critical differentiator is RL-based reasoning training, not thinking tokens: Grok-4's reasoning mode shows lower faithfulness than its non-reasoning mode (1.4% vs 7.2% necessity). We discover a faithfulness paradox - high-SLRC models are more susceptible to sycophancy - and propose the Reasoning Integrity Score (RIS = SLRC x (1-Sycophancy)), which significantly predicts error detection (rho=0.66, p=0.026). LC-CoSR achieves 2.6x less negative reward than FARL and CSR baselines without external model dependencies.
Abhinaba Basu, Pavan Chakraborty
Indian Institute of Information Technology Allahabad (IIITA) · National Institute of Electronics and Information Technology (NIELIT)
Reasoning language models frequently overthink: generating extended chains of behaviors such as hedging, approach abandonment, and self contradiction that consume tokens without improving answers. We show that these behaviors are not merely a consequence of length; even when controlling for response length, incorrect traces exhibit higher rates of unproductive self-reflection than correct ones. Addressing this requires identifying where self-reflection helps vs hurts, but obtaining these step-level annotations is costly. We observe that intermediate answer commitments within reasoning traces can provide a cheap proxy: by comparing each final answer candidate in the trace to the ground truth, we can determine whether subsequent reflection is productive without any additional supervision. Building on this insight, we propose DASH (Drift Aware advantage SHaping), which assigns segment-level credit based on whether each reasoning segment leads toward or away from correctness. On competition-level math benchmarks, DASH achieves the highest accuracy where overthinking is prevalent (AIME25: 50.8% vs. 45.4% GRPO) while reducing overthinking behaviors and achieving more productive self-correction than baselines.
Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.