Language-model systems in rule-governed domains often pair the model with a deterministic fact layer, but what that layer should compute is rarely measured. We measure it in the Chinese sexagenary system underlying BaZi chart analysis, where every fact has one table-checkable value: 682 items over twelve fact categories and 160 transit--natal collision items, each posed on a ladder of rungs that differ only in how much of the computation is done for the model. Answering directly, none of eleven open-weight models gets more than 4% of sexagenary days or 1% of collisions right, and supplying the attributes of every pillar does not change the collisions. A complete list of computed relations is not enough either: it leaves Qwen3-32B at 1%, with relations listed for other pairs misattributed to the queried pair, whereas a query-filtered layer from follow-up runs, stating only the queried relations pillar by pillar, gives 92%. On chess positions, two predictions locked in advance hold: attributes do not help, and unqueried relations are again misattributed. Three proprietary reasoning models compute 68--95% of collisions unaided, while reasoning in smaller open models overrides supplied facts. A fact layer should compute the interactions and return the ones the question asks about.
Figures & tables
Figure 1: The fact-layer ladder on one collision item (transit pillar given). Each rung supplies more of the computation to the same model; bars give exact accuracy over 80 items (orange: Qwen3-32B without thinking; blue: Qwen3.8-27B with thinking). Static attributes help neither model. The complete list ( +full ) contains every relation the answer needs, mixed with relations between natal pillars that the question does not concern, and Qwen3-32B answers 1% of items from it; stated per natal pillar by the query-filtered layer ( +rel. ), the same relations give 92%.
Abbr.
n
Steps
What is asked
StE
10
1
polarity, element of a stem
BrE
12
1
polarity, element of a branch
ERel
45
2
sheng / ke relation, acting side
TG
100
3
ten-god label vs. a day master
Hid
12
1
ordered hidden stems of a branch
NaY
60
1
NaYin of a pillar (free text)
Table 1: The twelve fact categories. Steps: table look-ups or derivation steps from the reference table to the gold answer.
No thinking
Thinking
Macro
DP
TrTG
Macro
DP
TrTG
Model
closed
table
tool
closed
table
closed
tool
closed
table
tool
closed
table
closed
tool
Majority class
23
23
23
5
5
3
3
23
23
23
5
5
3
3
Qwen3-0.6B
16
23
55
0
2
0
3
15
26
56
2
2
0
5
Qwen3-1.7B
25
36
70
2
2
0
25
31
46
68
2
2
0
18
Qwen3-4B
31
43
93
3
0
0
98
52
76
94
2
9
0
67
Table 2: Accuracy (%) on the 682-item benchmark. Macro: unweighted mean over the twelve categories; DP: day pillar; TrTG: transit ten god. No-thinking runs use greedy decoding; thinking runs use each model’s released sampling configuration with an 8k-token cap; “–”: not run or excluded (§ 4 ). Claude models (confirmatory set, § 5.5 ) reason at effort low .
Figure 2: Closed-book accuracy (%) by category (abbreviations as in Table 1 ), grouped by the steps an answer needs. Top row: majority-class answer; apricot and bold: at or below it. Look-ups rise with model generation and size; the day pillar and the transit ten god stay at or below the majority answer for every model except the two Qwen3.8 models with thinking.
Figure 3: Transit–natal collisions: share of items (%, n=80 per point) whose four relation sets are all exactly right, as more of the computation is supplied (8k cap). Thin lines: every open-weight model and mode; bold: Qwen3-32B without and Qwen3.8-27B with thinking. Shaded: +rel. ; right of the grey line: the reference table instead of facts. Static attributes have no consistent effect; removing only the natal–natal relations ( +full − natal ) recovers at most a third of what +rel. gains for Qwen3-8B and Qwen3-32B. Full results in Tables 6 and 12 .
Figure 4: BaZi collisions (a, transit pillar given) and chess (b) on the same ladder. Without thinking, the complete list (hollow) lifts 9 of 11 models to at least 21% in chess, while 7 of 11 stay below 2% in BaZi. *: +rel. beats +full after Holm correction (C2, chess only).
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Think
StE
BrE
ERel
TG
Hid
NaY
StR
BrR
Sea
DP
Abs
TrTG
Macro
Majority
10
17
11
10
8
3
80
54
20
5
49
3
23
Qwen3-0.6B
–
80
50
11
10
0
2
0
0
20
0
18
0
16
Qwen3-0.6B
✓
50
33
24
6
0
2
11
8
27
2
18
0
15
Qwen3-1.7B
–
90
75
11
12
8
3
0
53
22
2
27
0
25
Qwen3-1.7B
✓
80
75
22
8
0
3
53
37
43
2
52
0
31
Qwen3-4B
–
100
75
27
21
8
7
56
0
27
3
51
0
31
Appendix
Table 3: Per-category accuracy (%), closed condition. Abbreviations as in Table 1 .
Model
Think
StE
BrE
ERel
TG
Hid
NaY
StR
BrR
Sea
DP
Abs
TrTG
Macro
Majority
10
17
11
10
8
3
80
54
20
5
49
3
23
Qwen3-0.6B
–
90
83
9
12
33
10
0
0
18
2
15
0
23
Qwen3-0.6B
✓
80
75
22
7
58
8
4
4
23
2
27
0
26
Qwen3-1.7B
–
100
92
13
16
58
27
67
1
25
2
35
0
36
Qwen3-1.7B
✓
100
83
58
14
67
22
80
36
53
2
32
0
46
Qwen3-4B
–
100
100
22
18
100
28
36
6
50
0
50
0
43
Appendix
Table 4: Per-category accuracy (%), table condition. Abbreviations as in Table 1 .
Model
Think
StE
BrE
ERel
TG
Hid
NaY
StR
BrR
Sea
DP
Abs
TrTG
Macro
Majority
10
17
11
10
8
3
80
54
20
5
49
3
23
Qwen3-0.6B
–
100
92
18
99
92
52
40
35
28
93
6
3
55
Qwen3-0.6B
✓
90
67
40
85
42
32
44
33
95
92
44
5
56
Qwen3-1.7B
–
100
92
36
100
100
100
20
96
43
95
28
25
70
Qwen3-1.7B
✓
100
100
71
100
75
100
20
65
50
83
28
18
68
Qwen3-4B
–
100
83
78
100
100
100
100
100
98
100
58
98
93
Appendix
Table 5: Per-category accuracy (%), tool condition. Abbreviations as in Table 1 .
Transit pillar given
Only the date given
Model
Think
closed
+attrs
+full
table
+rel.
closed
+pillar
+attrs
+full
table
+rel.
Qwen3-0.6B
–
0
0
0
0
0
0
0
0
0
0
1
Qwen3-1.7B
–
0
0
0
0
42
0
0
0
0
0
32
Qwen3-4B
–
0
0
0
0
54
0
0
0
0
0
49
Qwen3-8B
–
0
0
1
0
81
1
1
0
0
0
78
Qwen3-14B
–
0
0
0
0
68
0
0
0
0
0
60
Appendix
Table 6: Transit–natal collisions: share of items (%, n=80 per cell) whose complete set of pairwise relations between the transit day and the four natal pillars is exactly right. +pillar : the computed transit pillar is supplied; +attrs : polarity, element and hidden stems of every pillar (including the transit pillar) are supplied; +full : the fact layer’s full output, including all relations; table : the reference table instead of facts; +rel. : only the transit pillar, its ten god and the relations between the transit day and each natal pillar (query-filtered fact layer).
Model
Th.
closed
table
+attrs
+full
− natal
+rel.
Qwen3-0.6B
–
0
0
0
0
0
0
Qwen3-1.7B
–
0
0
0
1
2
2
Qwen3-4B
–
0
0
0
21
26
71
Qwen3-8B
–
0
0
0
30
24
41
Qwen3-14B
–
0
0
0
32
45
42
Qwen3-32B
–
0
0
0
71
79
81
Appendix
Table 7: Chess set: share of items (%, n=160 ) whose relation sets between the new piece and all four pieces on the board are exactly right. Conditions mirror the collision ladder; − natal is +full without the relations between two pieces already on the board. Th.: thinking; low: Claude at effort low .
Figure 5: Chess set in detail. (a) Exact accuracy along the fact ladder; the shaded column is +rel. , and the reference table is shown apart because it is not a step of the ladder. Qwen3-4B without thinking gains most from filtering, and Qwen3-8B with thinking loses from it. Where the filtering gain is at least 10 points, removing only the relations between pieces already on the board recovers less than half of it, except for Qwen3-32B ( R=0.82 ), Qwen3.6 ( R=0.75 ) and Qwen3.8-Flash-Next ( R=0.88 ) without thinking. (b) Under +full , wrong slots whose relation type the list states receive more false positives than unlisted slots, for the 13 runs with at least 30 false positives. The axis is logarithmic, so connector length is the ratio. n.s.: not significant after Holm correction (one-sided Fisher exact test).
Figure 6: Qwen3 0.6B–32B: accuracy against parameter count in six categories, by condition (colour) and mode (dashed: no thinking; solid: thinking). Dotted line: majority-class answer. Memorised-table categories rise with size; the closed-book day pillar stays near zero.
8k cap
24k cap
Model
Cat.
Cond.
Acc.
Trunc.
Acc.
Trunc.
Med. tok.
Qwen3-32B
DP
closed
2
0
2
0
288
DP
table
29
2
35
0
1,926
TrTG
closed
0
0
0
0
695
TrTG
table
3
0
5
0
1,102
Qwen3.6-35B-A3B
DP
closed
2
55
6
1
8,227
Appendix
Table 8: Thinking-budget probe. Accuracy (%), number of truncated outputs, and median output tokens at the 24k cap. DP: 100 items; TrTG: 60 items.
closed
table
tool
Model
Th.
WD
DP
± 1
WD
WD
Qwen3-0.6B
–
9
0
33
15
100
✓
11
2
35
13
100
Qwen3-1.7B
–
20
2
35
11
100
✓
22
2
32
15
100
Qwen3-4B
–
34
3
67
51
100
Appendix
Table 9: Weekday control on the 100 day-pillar dates. WD: weekday accuracy (%; majority answer 21%, uniform 14%); DP: closed-book day-pillar accuracy on the same dates from the main runs; ± 1: share of wrong weekdays that are one day off (uniform 33%). Th.: thinking.
Day pillar, closed-book
table
Weekday, closed-book
Model
Acc.
corr. anchor
corr. count
no anchor
wrong only
trunc.
corr. count
anchor prec.
formula
Qwen3-4B
2
1
0
77
17
3
26
63
99
Qwen3-8B
1
0
0
71
25
3
51
75
97
Qwen3-14B
0
0
0
72
27
1
36
72
93
Qwen3-32B
2
0
0
82
16
0
39
77
99
Qwen3.6-35B-A3B
2
61
47
0
20
55
93
72
99
Appendix
Table 10: Thinking traces, graded step by step (%, 100 items per cell, 8k cap). corr. anchor : the trace states at least one correct date–day-pillar pair other than the target; corr. count : it contains the correct day count from such a pair, or the target’s Julian Day Number; no anchor , wrong only , trunc. : share of traces that state no date–pillar pair, only wrong ones, or end at the cap without an answer. Weekday: share of stated date–weekday pairs that are correct, and share of traces that name a weekday formula (Zeller, Kim Larsen or Doomsday).
P: closed
P: +attrs
D: closed
D: table
D: +attrs
8k
24k
8k
24k
8k
24k
8k
24k
8k
24k
Qwen3.6-35B-A3B
74 (0)
69 (0)
64 (0)
75 (0)
1 (43)
1 (1)
28 (37)
46 (0)
72 (0)
62 (0)
Qwen3.8-27B
82 (13)
96 (0)
68 (17)
94 (0)
2 (79)
89 (4)
26 (58)
100 (0)
69 (20)
95 (0)
Qwen3.8-Flash-Next
91 (3)
96 (0)
79 (12)
94 (0)
9 (75)
94 (0)
62 (31)
99 (0)
88 (7)
94 (0)
Appendix
Table 11: Collision accuracy (%, all four relation sets exact) with thinking at an 8k and a 24k output cap; number of truncated outputs (of 80) in parentheses. P: transit pillar given; D: only the date given.
Transit pillar given
Only the date given
Model
Th.
+full
− natal
+rel.
R
+full
− natal
+rel.
R
Qwen3-1.7B
–
0
0
42
0.00
0
0
32
0.00
Qwen3-4B
–
0
0
54
0.00
0
1
49
0.03
Qwen3-8B
–
1
2
81
0.02
0
4
78
0.05
Qwen3-14B
–
0
5
68
0.07
0
6
60
0.10
Qwen3-32B
–
1
10
92
0.10
0
1
90
0.01
Appendix
Table 12: Ablation of the filtering gain (H8, exploratory). Exact collision accuracy (%, n=80 ) under +full , +full − natal (relations between two natal pillars removed) and +rel. ; R=(a−natal−afull)/(arel−afull) , computed where +rel. exceeds +full by at least 10 points. Qwen3-0.6B has no such cell and is omitted.
Model
Err.
Hˉ
AUROC
Conf. err.
Qwen3-0.6B
91
1.15
0.62
17
Qwen3-1.7B
82
1.08
0.63
26
Qwen3-4B
78
0.87
0.76
34
Qwen3-8B
76
0.93
0.70
24
Qwen3-14B
63
0.66
0.79
32
Qwen3-32B
62
0.92
0.82
24
Appendix
Table 13: Sampling consistency, closed condition, no thinking (10 samples at T=0.7 per item). Err.: greedy error rate (%); Hˉ : mean answer entropy (bits); AUROC: entropy as a detector of greedy errors; Conf. err.: share of greedy errors on which all 10 samples agree (%).
Locked decision rule
Runs
H1
closed look-ups ≥95% (StE, BrE, Hid)
2/20
H1
BrR recall < precision
3/20
H2
rule 1: pooled look-up error cut ≥50% , McNemar sig.
10/18
H2
rule 2: DP and TrTG cut below look-up cut
17/18
H3
tool ≥95% in every category
5/20
H4
closed DP error ≥20% (no thinking)
11/11
Appendix
Table 14: Locked decision rules applied to the open-weight runs (exploratory; runs where the rule is satisfied / runs where it is decidable). H2 is undecidable when the closed look-up error is zero.
Check
Cases
Result
B1 stem relations, all ordered pairs
100
pass
B2 branch relations, all ordered pairs
144
pass
B3 day pillar, every date 1900-01-01–2100-12-31
73,414
pass
B4 ten god, all (day master, stem) pairs
100
pass
B5 collision items: gold relations, ten god, day pillar
160
pass
C1 coverage, every kind × colour × square (empty board)
768
pass
Appendix
Table 15: Exhaustive checks of the fact layers against answer-set programs (B: BaZi, C: chess). Each check covers every case of its domain. Mutation test: 12 of 12 single-rule mutants fail at least one check.
Residual
Context lines
Exact (%)
Condition
R
depth
on
off
direct
thinking
BaZi collisions
closed
43.0
4.5
0.0
0.0
0.1
21.1
table
43.0
4.5
0.0
0.0
0.1
40.2
+pillar
41.0
3.0
1.0
0.0
0.7
38.4
+attrs
41.0
3.0
1.0
4.0
0.1
35.4
Appendix
Table 16: Residual computation left by each condition (means over items) and exact accuracy (mean over open-weight models of at least 4B parameters, 8k cap). R : rule applications the model must perform; on / off : context lines that do or do not feed them.
Hugging Face repository
Revision
GPUs
Runs
Qwen/Qwen3-0.6B
c1899de289
1
no-think, sampled, think
Qwen/Qwen3-1.7B
70d244cc86
1
no-think, sampled, think
Qwen/Qwen3-4B
1cfa9a7208
1
no-think, sampled, think
Qwen/Qwen3-8B
b968826d9c
1
no-think, sampled, think
Qwen/Qwen3-14B
40c069824f
1
no-think, sampled, think
Qwen/Qwen3-32B
9216db5781
1
no-think, sampled, think
Appendix
Table 17: Model checkpoints. “sampled”: ten samples at T=0.7 without thinking (consistency analysis). Excluded thinking runs (Gemma-4, OLMo-3.1-Think) are listed in Appendix C .