cs.DBOct 29, 2026

Rebuilding a Field's Open-Problem Catalog from the Literature: A Two-Layer Knowledge Graph, and Why Status Must Stay Proposed

Authors: Madhulatha Mandarapu, Sandeep Kunkunuru

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A research field records its open problems in surveys, keynotes and community reports. Those are prose, written every few years, and nobody can ask them who works on a problem now or whether it was settled last year. We argue the problems should be data, and we build the system that makes them so: a hand-curated problem layer carried in unchanged, under an evidence layer rebuilt from a bibliographic index, joined by typed, provenance-carrying links in one graph database and queryable from one language. A language model does four jobs -- per-paper extraction, catalog grounding, per-paper status verdict, and idea generation -- each constrained to a candidate set so out-of-set answers are refused rather than stored. We report a study on two curated catalogs, database research (1,053 problems) and AI research (1,000 problems), each with a prior catalog-only graph. The rebuilt graph answers every question the catalog-only graph answered -- 47 of 47 on databases, 34 of 35 on AI -- plus questions the catalog-only graph refused: recent workers, untouched problems, methods tried, and outside papers bearing on a problem (336 problems, 770 works on databases). The capability we most wanted does not survive its own audit. Of 41 solves and refutes verdicts hand-checked against the papers, 9 are correct, 30 partly correct and 2 wrong; no refutes verdict (0 of 34) disproves an open problem, and only 1 of 7 solves closes the central question. We report that failure in full and draw the design conclusion: a status change must be proposed with its evidence and never applied, making the human reviewer part of the architecture. We close with the data-engineering problems this raises: entity resolution across curated and harvested layers, verifying model status claims, freshness under source quotas, and evaluation without ground truth.

Figures & tables

Fig. 1: The two layers and the bridge between them. No node label appears in both layers; a shared label is a build-time error (Section VII ). Every bridge edge carries how it was made, and EVIDENCE_FOR additionally carries the verdict, the quote it rests on and the model’s confidence.
StageKindOffered set
1 resolve author idsmodel + webcandidates from the index
2 harvest worksindex—
3 expand (refs, citers)index—
4 extract per papermodelfree text, then canonicalised
5 profile peoplemodel + web—
6 map the domainmodel + web—
TABLE I: Pipeline stages. “Offered set” is what the model may name; anything else is refused.
 DatabasesAI
Catalog problems1,0531,000
Catalog papers2,6794,932
Catalog people4,45415,102
Seed researchers4040
V1 nodes20,21129,707
V1 edges53,72162,842
TABLE II: The two fields. V1 is the catalog-only graph; V3 adds the evidence layer. Catalog commits and bench files are recorded in the artifact.
FieldQuestionV1V3
DBPapers after 2015 bearing on the problems787 catalog papers873 evidence works, 480 new to the catalog
DBWhich catalog papers a bibliographic index verifiesDOI on 1,800 of 2,6792,278 matched; 386 have neither a DOI nor a match
DBReferences no bibliographic source verifies15the same 15 (corroborated independently)
DBResearchers with an ORCID0 rows2,181 via their matched author
DBName variants of one researcher3 separate nodesall three resolve to one author
DBOpen problems nobody works on7238, counting evidence works as work
TABLE III: Questions the catalog-only graph answered in part, refused, or listed as out of scope, re-asked against the rebuilt graph with the same parameters. Full per-question tables, including the ones not attempted and why, are in the artifact.
 DatabasesAI
Catalog problems1,0531,000
…with at least one evidence work579592
Evidence works1,7731,969
EVIDENCE_FOR links1,8432,013
…with works from outside the catalog336187
…those works770300
TABLE IV: Evidence reaching the catalog’s problems.
 DatabasesAI
Catalog papers2,6794,932
…matched2,2784,630
by DOI—0
by arXiv id—3,026
by title and year—1,604
…unmatched401302
TABLE V: Bridging the curated layer to the harvested one. On AI, exactly 1 catalog paper carries a DOI, so the DOI rule — the most reliable one — fires 0 times.
VerdictDatabasesAI
advances8931,062
restates403235
unrelated351416
solves43
refutes232
none (not reached)190265
TABLE VI: Verdicts produced by the status check. none is a link the check never reached, because it reads at most eight papers per problem.
FieldVerdictnncorrectpartlywrong
AIsolves3030
AIrefutes328231
DBsolves4130
DBrefutes2011
All419302
TABLE VII: Hand check of every solves and refutes verdict both runs produced ( n=41n=41 , not a sample). correct : the paper does what the verdict says to a claim or question in the problem statement. partly : it bears on the problem but advances it, settles a sub-case, or contradicts an adjacent or rival claim. wrong : it does not do what the verdict says.
Patternnn
“methods fail” read as refuting an open “can it be done?”9
settles a sub-case only6
evidence from an adjacent or restricted setting6
answers or advances a sub-question, labelled refutes5
refutes a rival explanation or prior result, not the problem4
source paper of the phenomenon, labelled refutes1
TABLE VIII: Failure patterns across the 41 hand-checked verdicts.
KindExtractedPlaced
applications4,66538.6%
subjects7,56431.6%
problems6,88528.5%
methods14,19825.4%
metrics5,19812.9%
datasets, tools4,4315.9%
TABLE IX: Share of extracted concepts placed at same or narrower against a 13,551-class computer-science vocabulary (AI run). 9 answers were refused for naming a class not offered.
 DatabasesAI
Model calls2,8083,698
…through a batch API2,7993,690
Input tokens (M)23.628.6
Output tokens (M)6.59.6
Web searches431417
Snapshot size (MB)16.916.8
TABLE X: Model cost of one full build. Tokens are from the run’s usage log. No dollar figure is computed: the pipeline carries no price table, and quoting one would be a number with no artifact behind it.
More in this topic

KG Construction

Knowledge graph construction turns text, tables, images, and databases into structured entities and relationships that machines can query and reason over.

See all 197 papers

Explore similar work

CardsList
Oct 27, 2026

OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery

Large language models (LLMs) have shown increasing promise in research-level problem solving, yet a comprehensive collection of such problems is still missing, and evaluating them on research-level problems remains challenging. We introduce OpenFrontiers, a cross-disciplinary resource that combines a structured corpus of open research problems with a retrospective benchmark of recently resolved problems in mathematics. The corpus records problem statements, context, progress, and other information for 12,151 problems. The benchmark contains 200 problems, 110 resolved in 2026 and 90 still open, each with a Lean~4 statement checked by automated probes and two independent reviews; a subset HQ-50 further filters based on quality score, etc. Our evaluation protocol grades formal proofs with two independent checkers in an isolated sandbox under closed-book, frozen-retrieval, and live-web conditions. Evaluating seven LLM agents on the HQ-50 subset under a standardized budget, we find that closed-book rediscovery is rare, at most 7 of 40 resolved problems, whereas live-web success mostly reflects retrieval of published solutions. By linking a broad research-problem corpus to a formally graded evaluation subset with documented solutions, OpenFrontiers provides a foundation for measuring research-level rediscovery and studying progress toward solving currently open scientific problems. The dataset and benchmark could be found in https://huggingface.co/collections/FlippyDora/openfrontiers.

Jiarui Yao, Ruida Wang, Tong Zhang

University of Illinois Urbana-Champaign

LLM EvaluationLarge Language Model-Guided Scientific DiscoveryLarge Language Model-Guided Theorem Proving
AskPDF
Oct 8, 2026

OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences

The next frontier for artificial general intelligence is tackling unresolved scientific problems, calling for benchmarks that assess progress beyond established knowledge. We introduce OpenProblemBench, a benchmark of 82 unresolved problems drawn from the mathematics and theoretical physics literature. Each problem supplies the research context, assumptions, and prior progress needed to investigate the question. We select problems whose proposed solutions admit comparatively clear checks of their decisive mathematical or computational claims. Four evaluator models independently assess the correctness, completeness, and degree of progress of each submission without reference solutions. Across seven evaluated configurations, GPT-6-Astra achieves the highest mean judged solve rate of 14.0%, compared with 5.5-6.7% for the evaluated full-size open models and 2.4-3.7% for Flash models. Case comparisons connect stronger outcomes to changes in problem representation, general arguments that extend beyond finite evidence, and proofs of the steps needed to complete a solution. By grounding evaluation in questions arising from the research literature, OpenProblemBench provides a setting for investigating the capabilities and limitations of AI as a contributor to foundational theoretical science.

Zhiyi Li, Sihan Hu, Tianning Xiao +4

Department of Modern Physics, University of Science and Technology of China, Hefei, Anhui 230026, China · CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China · Hefei National Laboratory, University of Science and Technology of China, Hefei 230088, China +3

AI for ScienceScientific Reasoning in Language ModelsMathematical Reasoning Benchmarks
AskPDF
Oct 1, 2026

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.

Sohyeon Kim, Yoonho Lee, Bo Liu +11

Stanford University · Seoul National University · University of Washington +3

AI Agents for Scientific DiscoveryAgentic SearchAI Agent BenchmarksInformation RetrievalIR EvaluationScientific Document Retrieval
AskPDF