cs.AIOct 30, 2026
SaveFrontierMat: Benchmarking Language Models at the Frontier of Materials R&D
Organizations: Nanyang Technological University, Singapore · Peking University · Shanghai Jiao Tong University · Xi’an Jiaotong University ·
Abstract
To accelerate materials discovery, language models must be able to address problems at the research frontier. Yet textbook- and examination-derived benchmarks primarily assess established knowledge, leaving performance on frontier research problems largely untested. We introduce FrontierMat, a benchmark of 330 problems drawn from frontier academic and industrial research in fields including aerospace, semiconductors, embodied intelligence and nuclear fusion. Every problem is validated by leading experimentalists and industry experts in its field. Each problem presents a real study and asks the model to infer a withheld outcome or draw a conclusion from selected experimental evidence. Problems are annotated by functional domain, material system and the scale of the governing mechanism; 130 require cross-scale reasoning. The strongest general-purpose model scores only 53.7%, and on matched problems, cross-scale problems score 8.7 points below single-scale ones. Half of the problems are solved by at most one model, and models fail chiefly by endorsing what is plausible rather than what was observed. A domain-adaptation study with meta-weighted continued pretraining and scale-modular self-distillation raises an open-weight model from 36.7% to 55.4%, matching the strongest proprietary model, with improvements on public materials benchmarks and no decline in aggregate performance on general benchmarks. FrontierMat provides a testbed for measuring progress toward language models that support frontier materials R&D.
Figures & tables
Figure 1: FrontierMat leaderboard of six evaluated models. Bars split each overall score by the scale of the governing mechanism. Raw cross-scale scores are higher because these problems are more often single-answer; § 5.2 compares matched problems.
Figure 2: Overview of FrontierMat. (a) A unified pipeline constructs and verifies problems from frontier academic and industrial research. (b) An abridged example separates the study setting given to the model from the experimental result withheld for scoring. The schematic and reasoning chain illustrate how yarn architecture connects to macroscopic strength; full problem details appear in Appendix G .
| Benchmark | Problem source | Answer is | Frontier research | Scale labels | R&D practitioners | Difficulty from |
| Teaching material | ||||||
| MaScQA ( Zaki et al., 2024 ) | Entrance examination | Textbook knowledge | ✗ | ✗ | ✗ | Exam level |
| MaterialBENCH ( Yoshitake and others, 2024 ) | Two textbooks | Textbook knowledge | ✗ | ✗ | ✗ | Course level |
| MatSciBench ( Zhang and others, 2025 ) | Ten textbooks | Textbook knowledge | ✗ | ✗ | ✗ | Tiers by LLM response length |
| Generated from text | ||||||
| MSQA ( Cheung et al., 2025 ) | Papers (2000–2021) | Methods and results in source papers | ✗ | ✗ | ✗ | – |
Table 1: Comparison of FrontierMat with related benchmarks. Answer is : what the correct answer draws on; FrontierMat targets experimental outcomes and conclusions drawn from study evidence. Frontier research : problems come from current research rather than established curricula. Scale labels : each problem records the scale of the governing mechanism and whether it crosses scales. R&D practitioners : researchers in active academic and industrial R&D contributed research problems and assessed their relevance to current work; ✗: not reported (expert authoring or review alone does not count). Difficulty from : how problem difficulty is established or controlled.
Figure 3: Composition of all 330 problems in FrontierMat. Left: problems by scale of the governing mechanism (inner ring) and, within each scale, by functional domain (outer ring). Right: problems by functional domain (inner ring) and, within each domain, by material system (outer bars; percentages are shares within the domain).
| Scale of the governing mechanism | Domain group | |||||||
| Model | Reasoning | All | Micro | Meso | Macro | Cross | Structural | Functional |
| # problems | 330 | 88 | 24 | 88 | 130 | 159 | 171 | |
| Proprietary frontier models | ||||||||
| GPT-6-Astra | xhigh | 53.7 | 51.1 | 47.9 | 39.2 | 66.3 | 55.1 | 52.3 |
| GPT-6-Sol | xhigh | 51.4 | 48.3 | 50.0 | 34.1 | 65.4 | 50.0 | 52.6 |
| Claude Opus 5.5 | max | 47.3 | 43.2 | 35.4 | 27.3 | 65.8 | 47.8 | 46.8 |
Table 2: Score (%) on FrontierMat (scoring rule in § 4.1 ). Reasoning : the highest reasoning setting each model offers. Domain group : structural problems (mechanical, protective) and functional problems (thermal, electrical, electromagnetic, multifunctional trade-off); per-domain scores in Appendix D . The two Qwen rows are means over 16 samples with identical prompts and decoding (§ 6 ); cross-scale and single-scale columns differ in answer format and construction channel (§ 5.2 ). Bold: best per column; underlined: second best.
Figure 4: Where and how models fail on FrontierMat. (a) Responses to the 232 multi-answer problems. (b) Scores by scale on 202 problems with answer format and construction channel held fixed (bars: mean of five general-purpose models, dots: individual models). (c) Problems by the number of general-purpose models solving them.
| Materials | General | |||
| Model | MSQA | MatSciB. | MaScQA | Know. |
| GPT-4 (CoT) | – | – | 62.0 ⋆ | – |
| Gemini-2.0-Flash | 72.9 ⋆ | – | – | – |
| Claude-3.7-Sonnet | 70.2 | – | – | – |
| DeepSeek-R1 | – | 75.2 ⋆ | – | – |
| GPT-5 | – | 75.0 | – | – |
Table 3: Public materials benchmarks and general ability (%). Upper rows are reported by the original papers under their own settings, with ⋆ for each paper’s best. General Know. averages six benchmarks ( Table 12 ).
| Qwen3.5-0.8B | Held-out | FrontierMat |
| CPT + SFT (natural mixture) | 35.9 | 18.0 |
| + meta-weighted CPT | 39.4 (+3.5 † ) | 18.3 (+0.3) |
| + single-module self-distillation | 59.0 | 20.1 |
| + scale-modular self-distillation | 59.4 (+23.5 † ) | 20.5 (+2.5 † ) |
Table 4: Ablations on Qwen3.5-0.8B. Variants independently modify the natural-mixture CPT + SFT baseline. Held-out problems (513) use exact match and FrontierMat its standard score. Small figures are gains, and † marks (paired bootstrap).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Items | 330 |
| Single-answer / multi-answer | 98 / 232 |
| Sources | |
| Partner R&D groups (academic / industry) | 13 (9 / 4) |
| Scale of the governing mechanism | |
| Micro / meso / macro | 88 / 24 / 88 |
| Cross-scale | 130 |
Table 5: Statistics of FrontierMat. All items are multiple choice. Scale is the level of the structure and mechanism that governs the answer (§ 3.2 ).
| Field | Content |
| id | Stable identifier |
| prompt | Study setting, related results, options, answer format |
| answer_format | Single-answer or multi-answer |
| gold † | Letter, or alphabetically ordered letter set |
| domain | Functional domain (§ 3.2 ) |
| material_system | Class of material (§ 3.2 ) |
Table 6: Schema of a problem record.
| Model | Setting |
| GPT-6-Astra | reasoning effort xhigh; median 71 s, 2.2k output tokens |
| GPT-6-Sol | reasoning effort xhigh; median 39 s, 1.4k tokens |
| Claude Opus 5.5 | extended thinking (max); median 121 s, 5.0k tokens |
| DeepSeek V4 Pro | reasoning mode |
| GLM-5.3 | reasoning mode |
| Qwen3.5-35B-A3B | official post-trained release; same prompts and decoding as FrontierMat-Model; 16 samples |
Table 7: Model configurations.
| Subset | GPT-6 Astra | GPT-6 Sol | Claude Opus 5.5 | DeepSeek V4 Pro | GLM- 5.3 | Qwen3.5 (official) | FrontierMat- Model |
| All | 53.7 | 51.4 | 47.3 | 47.6 | 45.0 | 36.7 | 55.4 |
| Mechanical | 53.9 | 48.1 | 46.6 | 48.5 | 46.2 | 36.4 | 56.6 |
| Thermal | 50.7 | 50.0 | 48.0 | 51.0 | 47.0 | 34.4 | 49.8 |
| Multifunctional trade-off | 44.6 | 44.1 | 36.4 | 32.2 | 33.9 | 32.2 | 48.9 |
| Protective | 61.1 | 59.3 | 53.7 | 51.9 | 40.7 | 32.5 | 60.6 |
| Electromagnetic | 65.2 | 60.9 | 52.2 | 58.7 | 54.3 | 42.3 | 62.2 |
Table 8: Scores (%) by functional domain.
| Bucket | Pool (B tok.) | Reference share (%) |
| Papers, hi | 4.11 | 40.0 |
| Papers, mid | 5.74 | 31.1 |
| Reviews, hi | 0.52 | 5.0 |
| Reviews, mid | 0.24 | 2.4 |
| Synthetic (extractions, summaries) | 0.13 | 1.5 |
| English web / encyclopaedia | – | 6.0 / 1.4 |
Table 9: CPT data pools and reference mixture for the 10.94B-token budget. Materials buckets are reweighted once before CPT using the in-context procedure below. These proportions describe the corpus mixture, not the learned weights within materials data. Papers are hi if their citation count is at or above the median for their publication year and mid otherwise (including papers without citation data); reviews, books and chapters form the review buckets. Replay (bottom, 20% in total) draws on FineWeb-Edu, Ultra-FineWeb, FineWiki, Wikipedia, FineWeb2-HQ, Stack-Edu, StarCoderData, OpenCodeInstruct, CodeContests, OpenCodeReasoning, FineMath, InfiWebMath, OpenMathReasoning, NuminaMath-1.5, LibreTexts, Cosmopedia and Pressbooks.
| Part | Samples | Content |
| General capability | 99.9K | mathematics, code and science reasoning (OpenThoughts3, LIMO, s1K, Codeforces-CoTs, Nemotron-Science); Chinese reasoning distillation; instruction following (Nemotron-IF, SmolTalk2); dialogue (Firefly, SmolTalk2); structured output |
| Materials tasks | 11.0K | extraction from papers (EN 3.6K, ZH 3.5K), summarisation 3.8K |
| General supplement | 9.9K | constrained instructions, rewriting, summarisation, Python, long context |
| Materials questions | 9.0K | single-choice 3.2K, multi-select 4.8K, calculation 1.0K |
| Format-aligned items | 3.4K | recomposed, true/false, numeric-variant and newly written items from an internal set disjoint from all evaluation sets |
Table 10: SFT data composition.
| Comparison | Held-out | FrontierMat |
| Meta-weighted vs. natural CPT | +3.5 [1.1, 5.9] | +0.3 [ 0.6, 1.1] |
| Scale-modular vs. CPT + SFT | +23.5 [20.1, 26.9] | +2.5 [0.5, 4.6] |
| Scale-modular vs. single-module | +0.4 [ 2.0, 2.9] | +0.4 [ 1.2, 2.0] |
Table 11: Differences (points) in the 0.8B ablations, with 95% paired-bootstrap intervals (held-out: exact match; FrontierMat: score).
| Qwen3.5- 35B-A3B | FrontierMat-Model | |
| Public materials benchmarks | ||
| MSQA (binary) | 65.2 | 73.8 |
| MatSciBench (text) | 58.4 | 72.4 |
| MaScQA | 94.8 | 94.9 |
| General benchmarks | ||
| MMLU | 84.8 | 82.5 |
Table 12: Scores (%) of the official post-trained Qwen3.5-35B-A3B and FrontierMat-Model on public materials benchmarks and on six general benchmarks, under identical evaluation settings.
| CPT | |
| Data | 10.94B tokens ( Table 9 ) |
| Learning rate | , cosine to |
| Batch, length | 256 4,096 |
| Steps | 10,433 |
Table 13: Hyperparameters for CPT.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,432 papers