Rebuilding a Field's Open-Problem Catalog from the Literature: A Two-Layer Knowledge Graph, and Why Status Must Stay Proposed

Madhulatha Mandarapu; Sandeep Kunkunuru

cs.DBOct 29, 2026

Rebuilding a Field's Open-Problem Catalog from the Literature: A Two-Layer Knowledge Graph, and Why Status Must Stay Proposed

Authors: Madhulatha Mandarapu, Sandeep Kunkunuru

Abstract

A research field records its open problems in surveys, keynotes and community reports. Those are prose, written every few years, and nobody can ask them who works on a problem now or whether it was settled last year. We argue the problems should be data, and we build the system that makes them so: a hand-curated problem layer carried in unchanged, under an evidence layer rebuilt from a bibliographic index, joined by typed, provenance-carrying links in one graph database and queryable from one language. A language model does four jobs -- per-paper extraction, catalog grounding, per-paper status verdict, and idea generation -- each constrained to a candidate set so out-of-set answers are refused rather than stored. We report a study on two curated catalogs, database research (1,053 problems) and AI research (1,000 problems), each with a prior catalog-only graph. The rebuilt graph answers every question the catalog-only graph answered -- 47 of 47 on databases, 34 of 35 on AI -- plus questions the catalog-only graph refused: recent workers, untouched problems, methods tried, and outside papers bearing on a problem (336 problems, 770 works on databases). The capability we most wanted does not survive its own audit. Of 41 solves and refutes verdicts hand-checked against the papers, 9 are correct, 30 partly correct and 2 wrong; no refutes verdict (0 of 34) disproves an open problem, and only 1 of 7 solves closes the central question. We report that failure in full and draw the design conclusion: a status change must be proposed with its evidence and never applied, making the human reviewer part of the architecture. We close with the data-engineering problems this raises: entity resolution across curated and harvested layers, verifying model status claims, freshness under source quotas, and evaluation without ground truth.

Figures & tables

Panel of Figure 1 — Fig. 1: The two layers and the bridge between them. No node label appears in both layers; a shared label is a build-time error (Section VII ). Every bridge edge carries how it was made, and EVIDENCE_FOR additionally carries the verdict, the quote it rests on and the model’s confidence.

Stage	Kind	Offered set
1 resolve author ids	model + web	candidates from the index
2 harvest works	index	—
3 expand (refs, citers)	index	—
4 extract per paper	model	free text, then canonicalised
5 profile people	model + web	—
6 map the domain	model + web	—

TABLE I: Pipeline stages. “Offered set” is what the model may name; anything else is refused.

	Databases	AI
Catalog problems	1,053	1,000
Catalog papers	2,679	4,932
Catalog people	4,454	15,102
Seed researchers	40	40
V1 nodes	20,211	29,707
V1 edges	53,721	62,842

TABLE II: The two fields. V1 is the catalog-only graph; V3 adds the evidence layer. Catalog commits and bench files are recorded in the artifact.

Field	Question	V1	V3
DB	Papers after 2015 bearing on the problems	787 catalog papers	873 evidence works, 480 new to the catalog
DB	Which catalog papers a bibliographic index verifies	DOI on 1,800 of 2,679	2,278 matched; 386 have neither a DOI nor a match
DB	References no bibliographic source verifies	15	the same 15 (corroborated independently)
DB	Researchers with an ORCID	0 rows	2,181 via their matched author
DB	Name variants of one researcher	3 separate nodes	all three resolve to one author
DB	Open problems nobody works on	72	38, counting evidence works as work

TABLE III: Questions the catalog-only graph answered in part, refused, or listed as out of scope, re-asked against the rebuilt graph with the same parameters. Full per-question tables, including the ones not attempted and why, are in the artifact.

	Databases	AI
Catalog problems	1,053	1,000
…with at least one evidence work	579	592
Evidence works	1,773	1,969
EVIDENCE_FOR links	1,843	2,013
…with works from outside the catalog	336	187
…those works	770	300

TABLE IV: Evidence reaching the catalog’s problems.

	Databases	AI
Catalog papers	2,679	4,932
…matched	2,278	4,630
by DOI	—	0
by arXiv id	—	3,026
by title and year	—	1,604
…unmatched	401	302

TABLE V: Bridging the curated layer to the harvested one. On AI, exactly 1 catalog paper carries a DOI, so the DOI rule — the most reliable one — fires 0 times.

Verdict	Databases	AI
advances	893	1,062
restates	403	235
unrelated	351	416
solves	4	3
refutes	2	32
none (not reached)	190	265

TABLE VI: Verdicts produced by the status check. none is a link the check never reached, because it reads at most eight papers per problem.

Field	Verdict	$n$	correct	partly	wrong
AI	solves	3	0	3	0
AI	refutes	32	8	23	1
DB	solves	4	1	3	0
DB	refutes	2	0	1	1
All		41	9	30	2

TABLE VII: Hand check of every solves and refutes verdict both runs produced (

n=41

, not a sample). correct : the paper does what the verdict says to a claim or question in the problem statement. partly : it bears on the problem but advances it, settles a sub-case, or contradicts an adjacent or rival claim. wrong : it does not do what the verdict says.

Pattern	$n$
“methods fail” read as refuting an open “can it be done?”	9
settles a sub-case only	6
evidence from an adjacent or restricted setting	6
answers or advances a sub-question, labelled refutes	5
refutes a rival explanation or prior result, not the problem	4
source paper of the phenomenon, labelled refutes	1

TABLE VIII: Failure patterns across the 41 hand-checked verdicts.

Kind	Extracted	Placed
applications	4,665	38.6%
subjects	7,564	31.6%
problems	6,885	28.5%
methods	14,198	25.4%
metrics	5,198	12.9%
datasets, tools	4,431	5.9%

TABLE IX: Share of extracted concepts placed at same or narrower against a 13,551-class computer-science vocabulary (AI run). 9 answers were refused for naming a class not offered.

	Databases	AI
Model calls	2,808	3,698
…through a batch API	2,799	3,690
Input tokens (M)	23.6	28.6
Output tokens (M)	6.5	9.6
Web searches	431	417
Snapshot size (MB)	16.9	16.8

TABLE X: Model cost of one full build. Tokens are from the run’s usage log. No dollar figure is computed: the pipeline carries no price table, and quoting one would be a number with no artifact behind it.

KG Construction

Knowledge graph construction turns text, tables, images, and databases into structured entities and relationships that machines can query and reason over.

See all 197 papers

OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery
Oct 27, 2026Jiarui Yao, Ruida Wang, Tong ZhangLLM Evaluation Large Language Model-Guided Scientific Discovery
University of Illinois Urbana-Champaign
OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences
Oct 8, 2026Zhiyi Li, Sihan Hu, Tianning Xiao +4AI for Science Scientific Reasoning in Language Models
Department of Modern Physics, University of Science and Technology of China, Hefei, Anhui 230026, China · CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China · Hefei National Laboratory, University of Science and Technology of China, Hefei 230088, China +3
ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
Oct 1, 2026Sohyeon Kim, Yoonho Lee, Bo Liu +11AI Agents for Scientific Discovery Agentic Search
Stanford University · Seoul National University · University of Washington +3

Oct 27, 2026J/K move · Enter open · S save

OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery

Jiarui Yao, Ruida Wang, Tong Zhang

University of Illinois Urbana-Champaign

LLM Evaluation Large Language Model-Guided Scientific Discovery Large Language Model-Guided Theorem Proving

Open paper AskPDF arXiv

Large language models (LLMs) have shown increasing promise in research-level problem solving, yet a comprehensive collection of such problems is still missing, and evaluating them on research-level problems remains challenging. We introduce OpenFrontiers, a cross-disciplinary resource that combines a structured corpus of open research problems with a retrospective benchmark of recently resolved problems in mathematics. The corpus records problem statements, context, progress, and other information for 12,151 problems. The benchmark contains 200 problems, 110 resolved in 2026 and 90 still open, each with a Lean~4 statement checked by automated probes and two independent reviews; a subset HQ-50 further filters based on quality score, etc. Our evaluation protocol grades formal proofs with two independent checkers in an isolated sandbox under closed-book, frozen-retrieval, and live-web conditions. Evaluating seven LLM agents on the HQ-50 subset under a standardized budget, we find that closed-book rediscovery is rare, at most 7 of 40 resolved problems, whereas live-web success mostly reflects retrieval of published solutions. By linking a broad research-problem corpus to a formally graded evaluation subset with documented solutions, OpenFrontiers provides a foundation for measuring research-level rediscovery and studying progress toward solving currently open scientific problems. The dataset and benchmark could be found in https://huggingface.co/collections/FlippyDora/openfrontiers.

Abstract

KG Construction

Explore similar work

OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery

OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research