ShadowMiner v1 - An Experience Report on Implementing and Measuring a Problem-and-Hypothesis Discovery Engine

Jinhyuk Choi

cs.AIOct 3, 2026

ShadowMiner v1 - An Experience Report on Implementing and Measuring a Problem-and-Hypothesis Discovery Engine

Authors: Jinhyuk Choi

Organizations: Inforience Inc. Republic of Korea

Abstract

ShadowMiner v1 is a system that automatically discovers research problems and generates hypotheses from AI papers. It is a nine-stage pipeline. It structures documents into a knowledge graph and finds graph gaps in it - structural blind spots in research. These graph gaps are included in the LLM generation prompt. Each generated hypothesis is then verified by checking whether it is already covered by existing research, scoring its quality, and checking that the facts it relies on are accurately drawn from its sources. This report does not propose a new generation or evaluation technique. It describes our experience of implementing and applying ideas from prior work, and measuring whether each one actually contributed.

Figures & tables

System	Generate / Execute / Feedback	Autonomy	Learning	Diversity mechanism
ShadowMiner	O / $\times$ / $\times$	Tool–Analyst	Prompts only	None (parallel 4-fold generation and deduplication only)
Earlier hypothesis-generation papers (2023–2025, 11 papers, for reference)	O / mostly $\times$ / $\times$	Tool–Analyst	Prompts	Mostly none
EvoNarrator [12]	O / $\times$ / $\times$	Analyst	Prompts (graph constraints at inference time)	Macro-patterns of the evolution graph
GoR [15]	O / $\times$ / $\times$	Tool–Analyst	SFT (graph $\rightarrow$ ideas)	None
Agentic-Ideation [1]	O / $\times$ / (self-critique)	Analyst–Scientist	SFT (agent trajectories)	None
EvoGens [11]	O / $\times$ / own scorer	Analyst	Prompts	Evolution (mutation + crossover)

Table 1: Positions of the systems on axes 1–6 (four columns: generate/execute/feedback, autonomy, learning, diversity mechanism).

Figure 1: Nine-stage pipeline, from document-importance re-ranking to the citation-faithfulness check. Stages 2, 6, 8, and 9 call an LLM throughout; Stages 3 and 4 call an LLM only in part, during normalization; the rest are rule- or embedding-based.

#	Stage	LLM calls	Description
1	Document-importance re-ranking	None	Adds a small weight from external scholarly metadata (citation count, field-disruption index) to search relevance to set the order of the documents used as evidence
2	Knowledge-graph extraction	One per document (parallel)	Calls an LLM once per document to extract concepts and relations that fit a fixed schema
3	Graph storage and retrieval	Partly, during normalization	Accumulates the extracted facts in a store and builds two kinds of graph, narrow and wide
4	Problem normalization	Partly	Merges problems that differ only in wording so that they are not counted as separate limitations
5	Graph-gap detection and description	None	Uses rules on the wide graph to find graph gaps and turns each into one described line that is passed to Stage 6
6	Problem and hypothesis generation	Parallel calls (default 4)	Sends the narrow graph and the hints to the LLM in parallel with the same prompt and merges the results into problems and hypotheses

Table 2: Name, LLM use, and one-line description of each of the nine stages.

Figure 2: The four kinds of graph gaps. In each panel, thin solid lines are triples extracted from the papers, and the thick red dashed line is the relation being searched for, which is not in the graph. (a) Matrix gap: Methods 2 and 3, which address the same Task 1, are evaluated on Data 3, but Method 1 is not (support 2). (b) Task transfer: Methods 2, 3 and 4 address Task 2 in addition to Task 1, but Method 1 does not (support 3). (c) ABC transfer: Methods 2 and 3, which use the same Data 1, address Task 2, but Method 1 does not (support 2). (d) Unresolved limitation: Methods 1 and 2 are limited by Problem P, and no method is recorded as claiming to solve it (support 2). The methods, tasks, datasets, and problem in the panels are placeholders.

Pattern	Grouping criterion	What only this method lacks
(1) Matrix gap (application of Zwicky’s morphological analysis [40])	Methods that address the same task	Datasets used by the other methods in the same group
(2) Task transfer	Methods that address the same task	Additional tasks addressed by the other methods in the same group
(3) ABC transfer (application of Swanson’s ABC linking [36])	Methods that use the same dataset	Tasks addressed by the other methods in the same group

Table 3: Three matching patterns of graph-gap detection — the criterion for grouping methods, and what only this method lacks.

Topic	Hypotheses	Appeared	Actual appearance rate	Problem-level appearance
efficient_inference	12	2	0.167	5/6
rag_hallucination	12	3	0.250	3/6
alignment	10	3	0.300	2/6
agents	11	6	0.545	6/6
multimodal	12	8	0.667	4/6
code_generation	12	7	0.583	5/6

Table 4: Actual appearance results by topic (all nine topics). “Problem-level appearance” is the proportion of cases in which the problem statement itself — the stage above the hypothesis, not the hypothesis — later matched actual research.

Figure 3: Actual appearance rate for the nine topics (efficient_inference 0.167 to multimodal 0.667). The thick red dashed line is the overall actual appearance rate (0.388), the light shading is the topic-level bootstrap 95% CI ([0.282, 0.500], the interval we adopt), and the dotted lines are the hypothesis-level Wilson CI (independence assumed, optimistic). The variation across topics is the direct cause of the difference in width between the two CIs.

Appearance rate	Value	Condition
Conservative lower bound (arxiv-confirmed matches only)	0.262 (27/103)	Only arxiv, whose recorded date is its first public date
Actual appearance rate	0.388 (40/103)	All four sources; contamination boundary = generation model release date
Reference upper bound (freeze date moved to 2024-06-01)	0.506 (44/87)	Memorization possible; separate run

Table 5: Comparison of three values of the actual appearance rate.

Label basis	Exists appearance rate	New appearance rate	Difference	Significance
Original three-member panel	0.538 (14/26)	0.338 (25/74)	+0.201	$z=1.80$ , $p=0.071$
Independent relabel (o3)	0.750 (3/4)	0.372 (29/78)	+0.378	$z=1.51$ , $p=0.130$

Table 6: Exists/new appearance-rate difference by the judge that assigned the novelty labels.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

#	Technique	What was taken	Source	Implementation	Measured result	Adoption
—	Graph-gap detection and description — matrix gap, task transfer, ABC transfer, unresolved limitations	Search for graph gaps by rules rather than by an LLM	Swanson’s ABC linking (literature-based discovery [36]) and Zwicky’s morphological analysis [40]; the idea of an explicit gap-analysis step has the same aim as the Analyse_Gap tool of Agentic-Ideation [1]	Four rule-based patterns, no LLM, computation of support, with a minimum support for adoption (2 for the matrix gap, ABC transfer, and unresolved limitations; 3 for task transfer); the result is inserted into the Stage 6 prompt as described graph gaps (DGGs, for reference)	Baseline (with graph-gap hints) 13/36 = 0.361 vs. ablated (hints removed) 13/32 = 0.406, a difference of +4.5pp, not statistically significant ( $p=0.702$ ); the direction differed by topic (one topic dropped sharply, one rose sharply, one unchanged)	Kept as a default component of ShadowMiner
1	Experiment-execution (code-execution verification) loop	A closed loop that runs experiments for a hypothesis and verifies the result by measurement	AutoSOTA [5], VERITAS [38], SciDisco [33], HypoForge [19], and others	Not started	—	The experiment-execution loop is not considered; only hypothesis generation and evaluation are addressed
2	Critic feedback $\rightarrow$ regeneration loop	For a hypothesis below the threshold, feed back per-axis comments and regenerate once	IDEAgent [21] (repair success rate 93%, refine 82%), ResearchAgent [30] (converges after three regenerations)	Citation-faithfulness grade as the first-priority criterion and critic axes as the secondary criterion; one regeneration per hypothesis and a per-topic budget cap; hard-duplicate check; replace the original only on improvement	Test on three topics, 34 hypotheses: targeted 35.3% of cases but replaced only 8.3% (1 of 12); the one replacement did not appear in a later paper	Implemented (default off) — not adopted
3	Parallel multi-hypothesis generation + diversity	Turn one call into N parallel calls; add merging, deduplication, and a diversity metric	Semantic diversity $\times$ 2.6 with ordinary personas [6]; a one-line instruction to “stand out from other responses that might be generated for this same task” raised semantic diversity significantly at about $\times$ 1.1 token cost [4]; the Yield metric (the size of the largest set of hypotheses that pass the quality threshold and are sufficiently different from one another) is from IDEAgent [21]; generation-by-generation evolving candidate pools are from EvoGens [11]	gen_mode option (a single call; parallel makes N calls independently at the same time; sequential makes N calls one after another, each told the earlier results and instructed to differ) + merging and deduplication + Yield metric	Single call: groundedness 3.00, appearance rate 0.278 (5/18). Parallel (adopted): groundedness 3.03, appearance rate 0.387 (12/31) — more hypotheses in total (31 vs. 18) at about the same groundedness. Sequential: groundedness 3.05, appearance rate 0.208 (5/24), waste rate higher than parallel in every topic, 1.0–1.8 $\times$ slower	Parallel adopted (made the default; gen_n=4 , duplicate threshold 0.75), sequential dropped
4	Critic multi-provider ensemble	Move away from single-provider self-scoring and score with different providers/models	Prior cases showing the need for ensemble scoring: novelty-judgment benchmark [29], HindSight [17], InnoEval [23], The AI Scientist [3] (five-reviewer ensemble)	Extend critic_models to a list of "provider:model" , automatically skip unavailable providers, and turn the fallback off for judge calls (so a judge is not replaced by another model on failure)	Switched the operating panel to [glm:glm-5.1, openai:gpt-5-mini, glm:glm-4.6] , removing scoring by the generation model ( glm-4.7 ) itself (two of the three panel members are of the same GLM family, one is OpenAI)	Code completed, then actually switched on and operated with this setting
5	Explicit falsification observation (six-field contract)	Specify in structured form the conditions under which a hypothesis can be falsified	Lit2Test [25]	Design only, not started	—	Postponed

Appendix

Table 10

Hypotheses included	Exists appearance rate	New appearance rate	Difference	Significance
(A) All hypotheses, with no exclusions (100)	0.538 (14/26)	0.338 (25/74)	+0.201	$z=1.80$ , $p=0.071$
(B) Of (A), excluding the two-judge hypotheses (89)	0.522 (12/23)	0.318 (21/66)	+0.204	$z=1.74$ , $p=0.082$
(C) Of (A), excluding those affected by the freeze-date leak discussed in Section 5.2 (92)	0.571 (12/21)	0.338 (24/71)	+0.233	$z=1.93$ , $p=0.054$
(D) Of (C), excluding the two-judge hypotheses (82)	0.579 (11/19)	0.317 (20/63)	+0.261	$z=2.06$ , $p=0.039$

Appendix

Table 11

Item	Value
Hypothesis-generation model	glm-4.7 (Stage 6)
Hypothesis-generation mode	parallel ( gen_n=4 ; duplicate threshold: embedding cosine similarity $\geq$ 0.75) (Stage 6)
Critic judging panel	glm-5.1 / gpt-5-mini / glm-4.6 (three judges, mixed providers) (Stage 8)
Citation-faithfulness and factuality check	On (Stage 9)
Actual-appearance evaluation judge model	o3 (three-vote majority)
Narrow graph / wide graph	1-hop / 2-hop (Stage 3)

Appendix

Table 7: Settings applied to the actual-appearance experiment of Section 5.2 (nine topics).

Research Idea Generation

Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.

See all 60 papers

Hypothesis-Driven Deep Research with Large Language Models: A Structured Methodology for Automated Knowledge Discovery
May 11, 2026Michael ChinDeep Research Agents AI-Assisted Scientific Research
Independent Researcher
SGHA: Evidence-Grounded Research Problem Discovery with Local Language Models
Aug 18, 2026Sarvesh Gharat, Junpei KomiyamaAI Agents for Scientific Discovery Research Idea Generation
C-MInDS, IIT Bombay · Machine Learning Department, MBZUAI
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Jul 29, 2026Peter Kirgis, Sayash Kapoor, Andrew Schwartz +21AI-Assisted Scientific Research AI Agent Evaluation
Princeton University · Cornflower Labs · UK AI Security Institute +8

May 11, 2026J/K move · Enter open · S save

Hypothesis-Driven Deep Research with Large Language Models: A Structured Methodology for Automated Knowledge Discovery

Michael Chin

Independent Researcher

Deep Research Agents AI-Assisted Scientific Research Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

Current AI-powered research systems adopt a direct search-then-summarize paradigm that treats hypotheses as end products of scientific discovery. We argue this leaves a critical gap: hypotheses can serve a far more powerful role as organizational instruments that structure the research process itself. We propose the Hypothesis-Driven Deep Research (HDRI) methodology - the first framework using hypotheses to organize general-purpose deep research across arbitrary domains, rather than merely validating claims within specific domains. This transforms research from reactive information retrieval into proactive, verifiable, and iterative knowledge discovery. HDRI is formalized with six core principles and an eight-stage pipeline. A central innovation is the gap-driven iterative research mechanism - a closed-loop quality assurance system that automatically identifies informational and logical gaps, triggering targeted supplementary investigation. We further introduce a fact reasoning framework with traceable reasoning chains and quantified confidence propagation, a subject locking mechanism to prevent entity confusion, and a multi-dimensional quality assessment scheme. The methodology is realized in the INFOMINER system. Experiments demonstrate improvements of 22.4% in fact density, 90% subject matching accuracy, 0.92 multi-source verification confidence, and 14% completeness gain from gap-driven supplementation. Five case studies validate its practical applicability, achieving an average quality rating of 4.46/5.0.

Abstract

Research Idea Generation

Explore similar work

Hypothesis-Driven Deep Research with Large Language Models: A Structured Methodology for Automated Knowledge Discovery

SGHA: Evidence-Grounded Research Problem Discovery with Local Language Models

Can AI agents conduct open-ended AI research? Early evidence from two case studies