CARAT: Do Materials LLMs Reason or Recite?

Jiajun Wu; Jian Yang; Zixiang Ni; Zhenzhu Li; Bin Chong

cs.AISep 29, 2026

CARAT: Do Materials LLMs Reason or Recite?

Authors: Jiajun Wu, Jian Yang, Zixiang Ni, Zhenzhu Li, Bin Chong

Organizations: Beihang University · Xi’an Jiaotong University · Imperial College London · Peking University

Abstract

When a materials LLM answers a question about crystal structure, does it reason from the structure or copy an answer already printed in its input? Accuracy cannot tell: a structural description often prints the very field it is scored against. CARAT holds question and gold answer fixed across eight matched views, names each structural relation separately in GraphSpace, and adds matched fine-tuning, answer masking, evidence injection, paired inference, and a rule that can withhold claims. First, on the benchmark's hardest families the grounded view is worth 17.3 points over formula inputs. Second, we turn that scrutiny on ourselves. GraphSpace beats a plain periodic graph by 19.3 points, but that margin is two effects at once: where the plain rendering carries everything the question needs it is 1.96 points, and where it omits those fields entirely, 46.7 points. The headline mostly measures what the baseline lacked, not how evidence is presented. Third, we attack our own benchmark. A rule that skips the link and reads the list directly answers four of seven hardened families, so we rebuilt it until eleven such shortcuts sat near chance. The frozen model quotes that link yet answers the same when we redirect it, on 95.6% of paired cases: it repeats the relation without using it. After matched supervision it reaches 99.8%, and deleting the link drops it to 23.4%, below the 27.0% the best shortcut reaches: both steps are learnable.

Figures & tables

Figure 1: Reasoning or recitation? CARAT varies structural evidence under matched questions and tests answer dependence through masking and injection.

Figure 2: The CARAT framework. Evidence curation, GraphSpace construction, matched SFT, and causal certification connect controlled inputs to scoped capability claims.

Answer printed (the answer field is verbatim in the evidence)
	Overall	Per-family answer-exact rate
View	Micro	CIF plan	EFS path	Suff.	Graph	Local	MOF	Symm.
GraphSpace full	0.956	1.000	0.531	1.000	1.000	1.000	1.000	1.000
$-$ citations	0.876	1.000	0.573	1.000	0.578	1.000	1.000	1.000
$-$ graph	0.805	1.000	0.479	1.000	0.265	1.000	1.000	1.000
Answer masked (symmetry-local block removed)

Table 1: Per-view accuracy by task family. 27B model, 1,024 matched questions, seed 42, strict answer-exact rate; best per column bold. Table 2 gives the macro-hard-slice accuracy.

Seed	GS full	Best-comp.	Formula	$\Delta$ vs formula (95% CI, pp)	$\Delta$ worst (pp)	Worst family
42	0.956	0.944	0.783	$+17.3\;[+15.8,+18.7]$	$-8.9$	EFS path
43	0.958	0.943	0.789	$+16.9\;[+15.5,+18.3]$	$-5.7$	EFS path
44	0.960	0.943	0.780	$+18.0\;[+16.5,+19.5]$	$-8.9$	EFS path

Table 2: Multi-seed certification matrix. Seeds 42, 43 and 44 on the same paired set of 1,024 questions. The EFS-path family breaches the two-point tolerance in every seed.

Figure 3: Answer visibility and evidence intervention. Each intervention uses its own paired cohort; full uncertainty estimates appear in Appendix Table 9 .

Condition	What it tests	$n$	Frozen	Pointer-SFT
$k{=}4$ , pointer shown	selection $+$ computation	882	32.3%	99.8%
$k{=}4$ , pointer deleted	residual shortcut	882	26.8%	23.4%
Pointer moved (twins)	selection alone	1,764	25.9%	99.9%
Single list (control)	computation alone	882	60.2%	99.8%
Twin agreement when only the pointer moves		882	95.6%	0.11%
Paired effect of showing the pointer (pp)		882	$+5.56$	$\mathbf{+76.42}$

Table 3: Citing a relation versus using it. Held-out pointer-forced families, GraphSpace view. Shortcut maximum

27.0\%

; Appendix F.6 gives the denominators and reads the twin-agreement row, over pairs that differ only in the pointer.

Figure 4: Grounded citation and unsupported completion in a held-out case. The schematic contrasts supplied Co neighbours with an unsupported O completion after masking.

Figure 5: Canonicalization of an archived Rb record. Schematic graphs illustrate duplicate-edge merging while preserving multiplicity, sites, coordinates, and lattice.

Figure 6: More training data does not stabilize the measured effects. Panel (a) shows material-cluster 95% intervals; panel (b) pools 90,000 predictions per training size.

Figure 7: Generalization and the limits of certification. (a) Fluorine-holdout accuracy, material-cluster 95% intervals. (b) Worst-family regression per seed; all three breach the

-2

pp tolerance.

Property	Result	Verdict
Canonical order	$14{,}262$ / $14{,}262$	Pass
Round-trip decode	$14{,}262$ / $14{,}262$	Pass
Scalability	$170{,}290$ / $170{,}290$	Pass
Cross-material	$100\%$	Pass
No redundancy	$+11.8$ pp	Strong

Table 4: GraphSpace audits.

Figure 8: Answer visibility across views and external transfer. (a) Per-view accuracy by answer-visibility cohort. (b) External-benchmark macro accuracy for four task families under three views.

Figure 9: Layer-48 space-group probe accuracy by answer visibility. (a) probe mean and 95% CI vs. the majority baseline; (b) per-fold accuracy. Rows above the dashed rule print the answer.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Token cost, surface edits, and query phrasing. Style whiskers are material-cluster 95% intervals. Shading marks the pre-specified two-point equivalence band.

Figure 11: Evaluator symmetry-tolerance sensitivity. Stars mark the gold-label tolerances.

Figure 12: Thinking-SFT contrasts and alignment checks. Whiskers show available 95% intervals; dashed lines mark each check’s own threshold.

Figure 13: Probe decodability and reader-head ablation. Probe markers distinguish layers 56 and 63; only the pooled ablation row has a stored 95% interval.

Figure 14: Activation patching across three two-hop recipient views. The dashed 0.30 line is only the recovery-magnitude component of the joint diagnostic.

Figure 15: Refusal and confidence diagnostics. Confidence points separate correct and incorrect answers; group-specific confidence uncertainty is unavailable.

Figure 16: Scorer acceptance and LLM-judge agreement. The judge panel separates agreement between judges from agreement with gold, and marks a 0.75 reference.

Figure 17: Professional-tool and SFT references on EFS paths. MACE-MP uses 64 unique frame–task pairs, whereas SFT views use 225 prompts across three seeds.

Figure 18: Geometry controls and multi-turn specialist scores. Dialogue values are teacher-forced joint-state accuracies, not final-answer accuracies.

Figure 19: Recovery status and the proposed MOF task suite. Neither panel establishes an admitted structure-generation or module-reasoning capability.

Audit	Checks	Claim boundary
Scoring	Gold replay; null policies	Validates tested scorer contracts
Shortcuts	Answer masking; identity edits	Separates readback from evidence use
Mechanism	Probes; activation patching	No localized causal site established
Generalization	Element, symmetry, size, phrasing	Restricted to evaluated holdouts
Reliability	Refusal; confidence; judge agreement	Task- and judge-dependent evidence
Tool reference	MACE-MP comparison	Different evaluation denominators

Appendix

Table 5: CARAT audit categories and claim boundaries.

Input	Acc.	$\Delta$
Anonymous	$0.9937$	—
Identity shown	$0.9905$	$-0.0032$
Formula hidden	$0.9921$	$-0.0016$
Sites relabelled	$0.9953$	$+0.0016$
Labels shuffled	$0.4968$	$-0.4968$

Appendix

Table 6: Memorization controls.

View / group	S4c accuracy	n
Grounded (all cases)	68.5%	499
cites both orbit_link & sibling	72.1%	165
cites only orbit_link	67.5%	329
cites neither	20.0%	5
Plain-graph (all cases)	46.3%	367
cites orbit_link (any family)	n/a, rate = $0.000$	—

Appendix

Table 7: Dominant-neighbour accuracy by cited evidence. Citation groups are observational; the neither-cited subgroup contains only five cases.

View	Group	Accuracy
graphspace_full	Print-through	$100.0\%$
graphspace_minus_graph	Print-through	$100.0\%$
graphspace_no_evidence_refs	Print-through	$100.0\%$
graphspace_minus_symmetry_local	Answer-masked	$\mathbf{71.2\%}$
raw_cif	Alternative	$84.9\%$
raw_lattice_sites	Alternative	$66.7\%$

Appendix

Table 8: Answer masking in symmetry tasks. Each view uses 132 cases (seed 42); answer-printing views are excluded from capability claims.

Variant	Operation	S4c acc	$\Delta$	95% CI
V0 grounded	(baseline)	68.5%	—	—
V1	remove orbit-link tags	66.3%	$-2.2$	$[-3.8,-0.6]$
V2	also remove sibling references	63.4%	$-5.0$	$[-7.4,-3.0]$
V3	also remove typed neighbour shell	25.1%	$\mathbf{-45.6}$	$\mathbf{[-51.7,\,-39.6]}$
V0 plain-graph	(baseline)	45.1%	—	—
V4	inject typed edges into plain-graph	74.3%	$\mathbf{+27.7}$	$\mathbf{[+21.8,\,+33.9]}$

Appendix

Table 9: Typed-evidence masking and injection. Deltas and 95% bootstrap intervals use intervention-matched cases, 379 for masking and 354 for injection; baseline rows cover all cases.

View	Baseline	Thinking-SFT	Paired $\Delta$	95% CI
gs (frozen baseline)	34.53%	37.68%	$+3.16$ pp	$[+0.79,\,+5.46]$
V1 mask orbit_link	33.47%	37.37%	$+3.89$ pp	$[+1.54,\,+6.11]$
V4 pg $+$ orbit_link	29.42%	35.26%	$+5.84$ pp	$[+3.27,\,+8.41]$
V2 mask sibling_site_ref	31.74%	20.58%	$-11.16$ pp	$[-13.31,\,-9.01]$
V3 mask sibling_neighbours_typed	11.89%	4.00%	$-7.89$ pp	$[-9.46,\,-6.28]$

Appendix

Table 10: Thinking-SFT versus matched frozen-baseline references. Every row is scored over all 1,900 cases, non-parsing counted as errors, with material-clustered paired bootstrap intervals.

Arm	heads	acc	$\Delta$ S4c drop	95% CI
intact_baseline	0	1.0000 (341/341)	$+0.00$	—
top5_orbit_readers	5	0.9531 (325/341)	$+4.69$	$[+2.64,+7.04]$
Random heads (seed 42, max)	5	—	$+2.93$	—

Appendix

Table 11: Reader-head ablation on 341 dominant-neighbour cases. Teacher-forced answer scores compare the top five reader heads with equal-size random controls, under paired case bootstraps.

Release	n	Compress $\times$	Warnings
Pilot	$262$	$1.10$	$0$
Final canonical	$2{,}000$	$1.08$	$0$
Full (v1)	$2{,}000$	—	$0$
Full (v2)	$2{,}000$	$1.36$	$0$
Hardened v2	$2{,}000$	$1.05$	$0$
Hardened v3 (NL rewrite)	$2{,}000$	$1.05$	$0$

Appendix

Table 12: Canonicalization audit across eight releases. Compression is the raw-to-canonical edge ratio on full-edge cases; dashes indicate unavailable full-edge inputs.

Axiom	Result	Sample
A1 ′ hierarchical idempotency	PASS 100%	170,290 rows
A2 ′ permutation invariance (sampled)	PASS 100%	120 sampled
A3 ′ information conservation	PASS 100%	170,290 rows

Appendix

Table 13: Hierarchical canonicalization audit across all releases.

Class	Pass/Total	Rate
A elemental substitution	33/40	82.5%
C symmetry variant	30/30	100%
B ′ synthetic supercell	30/30	100%

Appendix

Table 14: Cross-material canonicalization under a physical-equivalence rubric.

CE	Removed field	Lemma	Witness
CE-1	Wyckoff labels	L2 space group	v4 anti-readback pass
CE-2	Element, typed neighbours	L4 element	V3 $-45.6$ ; V4 $+27.7$
CE-3	Edge multiplicity	L3 distance mset	$52\to 6$ rows
CE-4	Boundary clamp	L1 site order	$14{,}262/14{,}262$ pass

Appendix

Table 15: Boundary counterexamples, affected lemmas, and empirical witnesses.

View	Answer-exact	Field acc.	Macro-hard-slice
GraphSpace full	0.956	0.978	0.956
$-$ citations	0.876	0.938	0.926
$-$ graph	0.805	0.913	0.899
$-$ symmetry	0.922	0.956	0.944
Raw CIF	0.761	0.903	0.899
Plain graph	0.821	0.899	0.891

Appendix

Table 16: The same eight views under all three metrics (seed 42,

n{=}1{,}024

Hard slice	Task family	Items
CIF generation plan	CIF plan reconstruction	148
EFS path	EFS path reasoning	72
Insufficient or adv.	Evidence sufficiency	174
Insufficient or adv.	EFS path reasoning	24
MOF	MOF module property	136
Standard	Graph construction	204

Appendix

Table 17: Task families, hard slices, and item counts in the 1,024-question benchmark.

Family	What is asked	Items	Majority	List rule
S1	Count distinct Wyckoff orbits	6,555	0.101	—
S2	$k$ -th neighbour, then its own CN	23,993	0.272	0.949
S3	Element with max mean coordination	7,021	0.069	—
S4a	Anchor CN via its sibling	10,197	0.316	0.881
S4b	Anchor second shell via its sibling	9,769	0.224	0.996
S4c	Anchor dominant element via its sibling	10,197	0.505	0.932

Appendix

Table 18: Healthy-pool families, item counts, and majority baselines. Release v4, test partition.

Training view	Paired $n$	$\Delta$	95% CI
GraphSpace full	229,485	$+0.1933$	$[+0.1913,+0.1953]$
$-$ citations	229,528	$+0.1973$	$[+0.1952,+0.1993]$
$-$ graph	227,106	$+0.1678$	$[+0.1659,+0.1696]$
$-$ symmetry	229,498	$+0.1993$	$[+0.1974,+0.2014]$
Raw CIF	229,530	$+0.1363$	$[+0.1346,+0.1380]$
Plain graph	229,491	$+0.2027$	$[+0.2007,+0.2048]$

Appendix

Table 19: Healthy-pool paired difference by training view , pooled over seeds 42/43/44.

Family	Answer	Items	Majority	List rule	Model
S2	CN of the $k$ -th neighbour	23,993	27.2	94.9	—
S4a	CN via the sibling	10,197	31.6	88.1	31.8
S4b	Second shell via sibling	9,769	22.4	99.6	18.6
S4c	Dominant element via sibling	10,197	50.5	93.2	68.4

Appendix

Table 20: A pointer-blind list rule vs. majority baseline and model , four sibling/two-hop families.

Family	Arm	$n$	Accuracy	95% CI	$>$ max
F1 multi-candidate ( $k{=}4$ )	pointer shown	882	32.3%	$[29.2,\,35.3]$	yes
F1 multi-candidate ( $k{=}4$ )	pointer deleted	882	26.8%	$[23.9,\,29.5]$	no
F2 pointer swap	pointer shown	1,764	25.9%	$[24.2,\,27.6]$	no
F3 single-candidate control	pointer shown	882	60.2%	$[55.7,\,64.7]$	—

Appendix

Table 21: Pointer-forced families, frozen checkpoint, GraphSpace view. Every case counts, and a non-parsing generation counts as wrong. Intervals are material-clustered bootstraps over 2,000 resamples. The last column compares the lower bound against

27.0\%

, the highest of the eleven pointer-blind rules measured in Appendix F.4 .

Family	Arm	$n$	Frozen	Pointer-SFT
F1 multi-candidate ( $k{=}4$ )	pointer shown	882	32.3%	99.8%
F1 multi-candidate ( $k{=}4$ )	pointer deleted	882	26.8%	23.4%
F2 pointer swap	pointer shown	1,764	25.9%	99.9%
F3 single-candidate control	pointer shown	882	60.2%	99.8%

Appendix

Table 22: The same held-out pointer-forced test under two different models, GraphSpace view. Frozen is the main union checkpoint ckpt-14182. Pointer-SFT is the base Qwen3.6-27B trained answer-only on pointer-forced cases from disjoint materials. These are two models with different training histories, not two stages of one run. Every case counts, and a non-parsing generation counts as wrong. The highest of eleven measured pointer-blind rules reaches

27.0\%

Group	Family	$n$	$\Delta$	95% CI
Both views carry	S1 orbit count	19,401	$+0.0764$	$[+0.0717,+0.0814]$
the required	S2 $k$ -th nbr. CN	71,163	$-0.0158$	$[-0.0182,-0.0133]$
inputs	S3 argmax elem. coord	20,862	$+0.0842$	$[+0.0791,+0.0895]$
	S4d multiattr match	28,971	$+0.0218$	$[+0.0201,+0.0237]$
pooled		140,397	$\mathbf{+0.0196}$	—
Plain view cannot	S4a anchor CN	30,107	$+0.5915$	$[+0.5853,+0.5974]$

Appendix

Table 23: The headline view gap split by information parity. Same artefact as Table 19 : the GraphSpace-full training view over seeds 42, 43 and 44, paired on case id. The two groups sum to the 229,485 items and the

+0.1933

quoted in Section 3 . Family intervals are the pooled paired bootstrap. Group rows are weighted by item count and carry no re-bootstrapped interval.

Explore similar work

OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields
May 28, 2026Wanhao Liu, Jiaqing Xie, Qian Tan +10Materials Science Materials
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Fudan University +4
Visualizing Graph-to-Answer Mechanism Recovery in Materials-Science Hypothesis Generation
Aug 4, 2026Shashwat Sourav, Subhadeep Pal, Markus J. Buehler +4Hypothesis Synthesis
Department of Physics, Washington University in St. Louis · Oak Ridge National Laboratory · Lawrence Berkeley National Laboratory +6
Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis
Apr 15, 2026Zipeng Ling, Shuliang Liu, Seonil Son +4Reasoning Traces Chain-of-Thought Reasoning
The Hong Kong University of Science and Technology (Guangzhou) · University of Alberta · Alberta Machine Intelligence Institute +3

May 28, 2026, cs.AIJ/K move · Enter open · S save

OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields

Wanhao Liu, Jiaqing Xie, Qian Tan, Weida Wang+9

University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Fudan University+4

Materials Science Materials Multimodal Reasoning Benchmarks Scientific Discovery

Open paper AskPDF arXiv

As multimodal language models play an increasingly important role in scientific research, materials science offers a critical testbed due to its interdisciplinary, multimodal, and application-driven nature. However, existing materials benchmarks mainly focus on property prediction, knowledge QA, or characterization understanding, leaving the broader reasoning process from materials knowledge to application underexplored. To fill this gap, we present OmniMatBench, a human-calibrated multimodal reasoning benchmark for materials science. OmniMatBench contains 3,171 expert-curated QA and calculation problems across 19 materials-science subfields, spanning fundamental materials knowledge, structural and engineering materials, materials processing and manufacturing, and functional and applied materials. We evaluate 13 open-source and closed-source MLLMs and find that the best model achieves only a 0.372 overall score, revealing a substantial gap in current materials-science reasoning. Further analysis shows strong variation across subfields, fixed reasoning heuristics, uneven materials knowledge, and limited high-level knowledge application under formula-, retrieval-, and code-assisted settings. OmniMatBench provides crucial insights into the capabilities and limitations of current MLLMs and establishes a foundation for reliable AI assistants in materials-science research.

Abstract

Explore similar work

OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields

Visualizing Graph-to-Answer Mechanism Recovery in Materials-Science Hypothesis Generation

Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis