OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery

Jiarui Yao; Ruida Wang; Tong Zhang

cs.AIOct 27, 2026

OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery

Authors: Jiarui Yao, Ruida Wang, Tong Zhang

Organizations: University of Illinois Urbana-Champaign

Abstract

Large language models (LLMs) have shown increasing promise in research-level problem solving, yet a comprehensive collection of such problems is still missing, and evaluating them on research-level problems remains challenging. We introduce OpenFrontiers, a cross-disciplinary resource that combines a structured corpus of open research problems with a retrospective benchmark of recently resolved problems in mathematics. The corpus records problem statements, context, progress, and other information for 12,151 problems. The benchmark contains 200 problems, 110 resolved in 2026 and 90 still open, each with a Lean~4 statement checked by automated probes and two independent reviews; a subset HQ-50 further filters based on quality score, etc. Our evaluation protocol grades formal proofs with two independent checkers in an isolated sandbox under closed-book, frozen-retrieval, and live-web conditions. Evaluating seven LLM agents on the HQ-50 subset under a standardized budget, we find that closed-book rediscovery is rare, at most 7 of 40 resolved problems, whereas live-web success mostly reflects retrieval of published solutions. By linking a broad research-problem corpus to a formally graded evaluation subset with documented solutions, OpenFrontiers provides a foundation for measuring research-level rediscovery and studying progress toward solving currently open scientific problems. The dataset and benchmark could be found in https://huggingface.co/collections/FlippyDora/openfrontiers.

Figures & tables

Figure 1: Overview of OpenFrontiers: registered sources are fetched and normalized into a provenance-first corpus (Section 3 ); curation yields the 200 Lean-stated items of OpenFrontiers-Bench (Section 4 ); models are evaluated under three access conditions and graded by two Lean checkers (Section 5 ).

Quantity	Value
Problem / catalogue items	12,151 / 4,751
Sources registered / contributing	71 / 62
Collections with $\geq$ 100 records	7
Distinct source names	6,294
Records from problem / workshop lists	8,349 / 3,516
Top-level fields / sub-fields	10 / 32

Table 1: Key figures of the corpus (problem partition unless stated; snapshot 2026-10-10; source records, not deduplicated conjectures).

Panel of Figure 2 — Figure 2: Corpus statistics for the 12,151 problem records: (a) status; (b) top-10 MSC 2020 top-level classes by records with at least one code in the class (8,011 records carry a code); (c) posed decade for the 3,389 records with a known posed year (91 records before 1950 omitted).

Rediscovery-110	Jan–Jun	Jul–Sep	total
Tier A / B / C	26/8/0	59/13/4	85/21/4
Erdős Problems	22	18	40
Kourovka Notebook	8	8	16
Formal Conjectures / Wikipedia	0	13	13
Other sources (Green’s list: 2)	4	37	41
AI produced the key mathematics	14	41	55

Table 2: Composition of OpenFrontiers-Bench. Rediscovery-110 columns split by the located solution-evidence month of 2026. AI classes classify disclosures in the solution materials; “none disclosed” does not imply that no AI was used.

Panel of Figure 3 — Figure 3: Located solution-evidence month of the 110 Rediscovery-110 items, by disclosed AI involvement (monthly totals above the bars).

		A: closed-book		B: frozen retrieval		C: live web
Model	lineage, cutoff	all	clean	all	clean	all	clean
Native protocol (the agent ends the session)
Claude Sonnet 5.5	Anthropic, 2026-06	0	0/30	1	1/30	6	3/30
GPT-6.1-sol	OpenAI, 2026-06	2	2/15	2	2/15	15	6/15
GPT-6-luna	OpenAI, 2026-06	0	0/15	0	0/15	4	3/15
Kimi-K3	–, 2026-06	2	2/33	2	2/33	15	12/33

Table 3: Results on the 40 solved items of HQ-50 (pass@1, 3 h per attempt). all : items solved, i.e. a Lean proof with the reference answer accepted by both checkers, out of 40; clean : solved / eligible after removing items whose key mathematics came from the model’s lineage and items with solution evidence on or before the model’s cutoff (release date where none is published; – means no lineage exclusion). C mostly measures retrieval of known solutions (Section 5.4 ). Open items are reported separately.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Prot.	Cond.	Median min	.lean	End reasons
Claude Sonnet 5.5	nat.	A	0.3	3	agent stopped 49, agent error 1
		B	0.4	18	agent stopped 48, wall clock 2
		C	0.4	19	agent stopped 48, wall clock 2
Claude Sonnet 5.5	cont.	A	5.1	33	stalled 41, wall clock 4, submitted 4, agent stopped 1
		B	17.7	47	stalled 45, submitted 3, agent stopped 2
		C	35.9	46	stalled 27, submitted 23

Appendix

Table 7: Per-run details on HQ-50 (50 items per condition, one attempt each; 180-minute budget). Median min = median attempt wall-clock minutes; .lean = attempts that left a formal submission; Open acc. = open items accepted by both checkers, before review.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,425 papers

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
May 9, 2026Guijin Son, Seungone Kim, Catherine Arnett +73Mathematical Reasoning Benchmarks Benchmark Design
Advancing Mathematics Research with AI-Driven Formal Proof Search
May 21, 2026George Tsoukalas, Anton Kovsharov, Sergey Shirobokov +18Large Language Model-Guided Theorem Proving Automated Theorem Proving
1Google DeepMind1 · 2Aarhus University · 3Google
LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics
Feb 27, 2026Antoine Peyronnet, Fabian Gloeckle, Amaury HayatMathematical Reasoning Benchmarks Automated Theorem Proving
1Ecole Normale Superieure de Rennes · 2Ecole des Ponts Paris · 3Korean Institute for Advanced Study

May 9, 2026J/K move · Enter open · S save

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

Guijin Son, Seungone Kim, Catherine Arnett, Hyunwoo Ko+72

Mathematical Reasoning Benchmarks Benchmark Design LLM Mathematical Reasoning

Open paper AskPDF arXiv

Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challenging target for measuring LLM reasoning. Whereas olympiad-style problems measure step-by-step reasoning alone, research-level problems use such reasoning to advance the frontier of mathematical knowledge itself, emerging as a compelling alternative. Yet research-level math benchmarks remain scarce because such problems are difficult to source (e.g., Riemann Bench and FrontierMath-Tier 4 contain 25 and 50 problems, respectively). To support reliable evaluation of next-generation frontier models, we introduce Soohak, a 439-problem benchmark newly authored from scratch by 64 mathematicians. Soohak comprises two subsets. On the Challenge subset, frontier models including Gemini-3-Pro, GPT-5, and Claude-Opus-4.5 reach 30.4%, 26.4%, and 10.4% respectively, leaving substantial headroom, while leading open-weight models such as Qwen3-235B, GPT-OSS-120B, and Kimi-2.5 remain below 15%. Notably, beyond standard problem solving, Soohak introduces a refusal subset that probes a capability intrinsic to research mathematics: recognizing ill-posed problems and pausing rather than producing confident but unjustified answers. On this subset, no model exceeds 50%, identifying refusal as a new optimization target that current models do not directly address. To prevent contamination, the dataset will be publicly released in late 2026, with model evaluations available upon request in the interim.

Abstract

LLM Evaluation

Explore similar work

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

Advancing Mathematics Research with AI-Driven Formal Proof Search

LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics