Where Research Ideation Begins: Forward-Citation Supervision for Actionable Limitation Generation

Liyang Zhao; Yong Cao; Haibin Ling

cs.AIOct 16, 2026

Where Research Ideation Begins: Forward-Citation Supervision for Actionable Limitation Generation

Authors: Liyang Zhao, Yong Cao, Haibin Ling

Organizations: Westlake University · University of Tübingen

Abstract

Automated research ideation has advanced rapidly, with multi-agent workflows that generate whole research ideas end to end. These frameworks leave largely unexamined the step that seeds a good idea: identifying an actionable limitation from literature. In this paper, we focus on this first step in the research ideation pipeline. Starting from a pilot study, we find that as large language models (LLMs) scale from 0.6B to 32B parameters, they become far better at recognizing which limitation later work addressed, but no better at generating one, suggesting that the bottleneck lies in the supervision signal rather than model capacity. We therefore construct ActLim-Bench from forward citations, yielding 18,279 evidence-linked supervision triples with human verification. On this signal, we propose a three-stage post-training pipeline: supervised fine-tuning, verifier-filtered rejection-sampling fine-tuning, and reinforcement learning with the verifier as an online dense reward. Experimental results show that our method achieves the highest recall score compared to existing methods among all frontier models. We further show that our generated limitations are preferred in 64.9% of non-tied human comparisons and improve downstream ideation systems. We will release our data and code upon publication.

Figures & tables

Figure 1: Overview of our framework. Using forward-citation supervision as the training signal, we align general LLMs into domain experts for actionable limitation generation.

Split	Years	Papers	Triples	Targets
Train	2017–23	4,304	12,925	10,954
Dev	2017–23	300	901	763
Test	2024–25	1,512	4,453	3,774
Total	2017–25	6,116	18,279	15,491

Table 1: Statistics of ActLim-Bench . Triples are evidence-linked

(P,\ell,Q)

records; targets are paper-level limitations after within-paper semantic aggregation. The Computer Vision transfer set is held separately.

Figure 2: Benchmark construction and verifier-guided training. We construct ActLim-Bench from verified forward citations and train the model through supervised fine-tuning, verifier-filtered rejection-sampling fine-tuning, and set-level optimization with graded rewards.

Figure 3: Graded one-to-one matching. Predictions are aligned with observed targets at category, facet, or specific granularity. The one-to-one constraint prevents broad or repeated predictions from claiming multiple targets.

Zero-shot
System	NLP Test			CV Transfer
	GRec@3 I	SpecRec@3 I	GRec@3 B	GRec@3 I	SpecRec@3 I	GRec@3 B
Qwen3-8B †	0.204	0.079	0.210	0.198	0.070	0.204
Qwen3-32B †	0.181	0.058	0.188	0.176	0.051	0.183
Qwen-72B-Chat	0.235	0.076	0.242	0.226	0.066	0.233
Claude-3-Opus	0.251	0.089	0.258	0.243	0.078	0.250

Table 2: Main results on the held-out NLP test set and zero-shot transfer to CV. All trainable models use NLP-only training. Independent-evaluator metrics are primary; GRec@3 B shows that the ranking is preserved under the benchmark evaluator. Bold marks the overall best result and underline the strongest zero-shot result in each column. † indicates a Qwen3-based system, whose pretraining window may overlap evaluation papers. Baseline adaptations and more details are reported in the supplement.

Supervision	GRec@3 I	SpecRec@3 I	GRec@3 B
Author-stated	0.193	0.072	0.198
Reviewer-written	0.219	0.081	0.224
Controlled perturbations	0.236	0.087	0.242
Forward-citation	0.353	0.172	0.362

Table 3: Matched supervision-source comparison. All conditions use the same source papers, number of examples, token budget, target-length distribution, training recipe, and three seeds. Values are three-seed means; all standard deviations are at most 0.007 and are reported in the supplement.

Strategy	Use of judgment	GRec@3 I	GRec@3 B
Self-rerank@10	None	0.371	0.390
DPO	Binary preference	0.365	0.382
Verifier-guided expansion	Item filter	0.410	0.428
Set-level GRPO (matched)	Response reward	0.425	0.449
Full: expansion + GRPO	Filter+response reward	0.452	0.485

Table 4: Ablation study on Verifier. DPO, expansion, and matched set-level GRPO share the Stage-1 initialization, source papers, generated-item and verifier-scoring volume. Self-rerank@10 is an inference-only reference; the final row combines the two complementary verifier-guided stages.

Figure 4: Judgment–generation asymmetry. Within the Qwen3 family, 8-way selection accuracy rises strongly with model size, whereas source-only limitation generation shows no consistent positive trend. Earlier-release zero-shot systems are complementary references and are excluded from the same-family regressions. The horizontal dashed line in panel (a) marks chance at $1/8$ .

Pairwise preference for ActLim-8B
Opponent	W/T/L	$p_{\mathrm{Holm}}$
Qwen3-32B	74/9/37	0.0023
GPT-4-Turbo	70/9/41	0.0228

Table 5: Human evaluation. The upper panel reports ActLim-8B wins, ties, and losses over 120 blind comparisons against each zero-shot system, with Holm-adjusted

p

-values. The lower panel reports absolute ratings over 150 outputs per system. Valid–Act. denotes outputs judged both faithful(Faith.) and actionable(Act.).

Supplied signal	RetAlign
None	0.302
Zero-shot limitation	0.348
ActLim-8B limitation	0.480
Gold forward-citation limitation (oracle)	0.556

Table 6: Downstream evaluation with a fixed proposer. RetAlign is macro-averaged over 200 papers. The oracle supplies a verified future-pursued limitation and is shown only as an empirical upper reference. Human ratings use 100 paired examples and three raters.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Years	Papers	Triples	Targets
Train	2017–23	4,304	12,925	10,954
Dev	2017–23	300	901	763
Test	2024–25	1,512	4,453	3,774
Total	2017–25	6,116	18,279	15,491
CV (transfer)	2024–25	1,737	3,961	3,562

Appendix

Table 7: ActLim-Bench . The CV set is held separately and is not counted in the total. NLP Test averages 2.50 targets per paper and the CV set 2.05, so absolute recall is compared within domain rather than across domains.

Check	Method	Rule	Removed
Version collapse	DOI + title	exact	3,417
Metadata normalization	rule-based	—	0
Abstract near-duplicates	BGE-M3	$\cos>0.94$	1,286
Citation-graph overlap	manual	inspection	92
Paraphrase overlap	BGE-M3	$\cos>0.92$	604
Split disjointness	ID join	exact	2,158

Appendix

Table 8: Leakage controls, applied in the order listed. Counts are papers for the first four rows and triples for the last two.

Criterion	Pass	$n/400$	$\kappa$
Attribution ( $A$ )	0.94	376	0.81
Consistency ( $C$ )	0.93	372	0.77
Specificity ( $S$ )	0.90	360	0.69
Response ( $R$ )	0.92	368	0.75
Evidence support after aggregation	0.96	384	0.84
All criteria	0.87	348	0.74

Appendix

Table 9: Triple-level audit,

n=400

. Joint precision 0.87, 95% Wilson CI

[0.833,0.899]

. Train 0.88 (176/200) and Test 0.86 (172/200) combine to 348/400. The joint rate is bounded above by the weakest criterion (0.90) and below by

1-\sum_{i}(1-p_{i})=0.65

Attribution type	Share	$n$	Pass
Direct	82%	328	0.95
Method-family	18%	72	0.89
Combined	100%	400	0.94

Appendix

Table 10: Attribution by sub-case. The share-weighted rate

0.82\times 0.95+0.18\times 0.89=0.939

recovers the overall attribution pass rate in Table 9 . These are pass rates on criterion

A

, not joint precision over all criteria.

Stratum	$n$	Population share	Valid
Accepted pairs	300	5.7%	0.910
Near-threshold rejections	300	12.4%	0.150
Other verification rejections	200	17.9%	0.040
Extraction-stage rejections	100	64.0%	0.005

Appendix

Table 11: Pipeline audit strata. “Valid” is the audited rate of pairs carrying at least one fully valid limitation, reported to three decimals because the extraction stratum dominates the population and a rate printed to two decimals does not reproduce the recall estimate below.

Illustrative example, direct attribution.
$P$ — a 2024 long-document summarization model.
$e_{P}$ : “We evaluate on English news and scientific abstracts, and leave multi-document settings to future work.”
$Q$ — a 2025 multi-document summarizer.
$e_{Q}$ : “ [ $P$ ] operates on a single document and cannot resolve redundancy across sources; we extend the encoder with a cross-document salience layer to address this.”
Target $\ell$ : cannot resolve cross-document redundancy because salience is computed within a single document.
Judgments : $A$ direct; $C$ pass, since $e_{P}$ confirms the scope; $S$ pass, since the salience computation is named; $R$ pass, since the cross-document layer is the main contribution of $Q$ .

Appendix

Table 12: An illustrative example showing how the rubric applies to a direct-attribution case. It is written to explain the criteria and is not a record from the dataset.

Role	Model	Sees $Q$ ?	Guides training?	Final metric?
$V_{\mathrm{data}}$	DeepSeek-V3.2-Exp	Yes	No	No
$V_{\mathrm{train}}$	DeepSeek-V3.2-Exp	No	Yes	No
$V_{\mathrm{B}}$	DeepSeek-V3.2-Exp	No	No	secondary
$V_{\mathrm{I}}$	GLM-5.2	No	No	primary

Appendix

Table 13: The first three roles share one frozen DeepSeek model, accessed through the deepseek-chat endpoint, with role-specific prompts; only

V_{\mathrm{data}}

sees citing papers.

V_{\mathrm{I}}

uses GLM-5.2 ( zai-org/GLM-5.2 ) and takes no part in construction, filtering, reward, or model selection, which is why the main paper reports it as the primary metric and

V_{\mathrm{B}}

as a same-benchmark check. Matching calls for

V_{\mathrm{train}}

V_{\mathrm{B}}

, and

V_{\mathrm{I}}

use temperature

0.0

, top-

p

1.0

, and a 512-token cap; outputs still unparseable after three retries are assigned the none label, which occurred for 0.31% of calls. Construction calls for

V_{\mathrm{data}}

were made on 2025-11-04. Training and evaluation calls for

V_{\mathrm{train}}

V_{\mathrm{B}}

, and

V_{\mathrm{I}}

ran from 2026-02-03 to 2026-04-28.

System	Identifier	Release	Cutoff
Qwen3-8B	Qwen3-8B	2025-04	not documented
Qwen3-32B	Qwen3-32B	2025-04	not documented
Qwen-72B-Chat	Qwen-72B-Chat	2023-11	not documented
Claude-3-Opus	20240229	2024-03	2023-08
Gemini-1.0-Ultra	gemini-1.0-ultra	2024-02	not documented
GPT-4-Turbo	0125-preview	2024-01	2023-12

Appendix

Table 14: Release dates and documented knowledge cutoffs. Cutoffs are reported only where the provider publishes one: Anthropic’s Claude 3 model card states August 2023, and OpenAI’s model documentation states December 2023 for the GPT-4 Turbo preview snapshot. Google did not publish cutoffs for the Gemini 1.0 generation, and Alibaba does not publish cutoffs for the Qwen families, so those entries are left undocumented rather than estimated. Exposure control therefore rests on the two systems with published pre-2024 cutoffs, and Qwen3 rows are read as same-family capability references.

Verifier	Human
Verifier	none	cat.	facet	spec.
none	132	14	3	1
category	17	108	21	4
facet	5	19	113	13
specific	1	4	17	128

Appendix

Table 15: Verifier–human confusion matrix,

n=600

. Quadratic-weighted

\kappa=0.88

; Spearman

\rho=0.88

; exact agreement 80.2%; agreement within one level 97.0%. Disagreements are concentrated on adjacent levels and are near-symmetric, so the verifier is not systematically generous or severe.

Setting	Stage 1	Stage 2	Stage 3
Tuning	full	full	full
Learning rate	$1{\times}10^{-5}$	$7{\times}10^{-6}$	$2{\times}10^{-6}$
Batch size (eff.)	64	64	—
Epochs / steps	3 ep	2 ep	1,200 st
Max sequence length	4,096	4,096	4,096
Optimizer	AdamW	AdamW	AdamW

Appendix

Table 16: Optimization settings; GPU-hours are summed over three seeds. Hardware: 8

\times

A100-80GB for Stages 1–2 and 16

\times

A100-80GB for Stage 3; rollout generation uses vLLM 0.8.2. Seeds are 13, 42, 2024.

Arm	Items	Calls	Updates	GPU-h
DPO	9,000	9,000	1,406	14
Verifier-guided expansion	9,000	9,000	876	11
Set-level GRPO (matched)	9,000	9,000	1,125	26

Appendix

Table 17: Budget accounting per seed.

System	Backbone	Calls	Retrieval	Cutoff
+ CoT	Qwen3-8B	1	none	—
+ LimGen-style	Qwen3-8B	1	none	—
+ Self-Refine	Qwen3-8B	5	none	—
Pre-pub. RAG	Qwen3-8B	1	S2 corpus	date( $P$ )
MARG-style	Qwen3-8B	9	none	—
BAGELS-RAG	Qwen3-8B	3	S2 corpus	date( $P$ )

Appendix

Table 18: Baseline alignment. “Cutoff” is the latest publication date a retrieved document may carry.

$\|\mathcal{G}_{P}\|$	$n$	Qwen3-32B	GPT-4-Turbo	ActLim-8B
1	280	0.232	0.328	0.560
2	602	0.196	0.278	0.492
3	430	0.168	0.242	0.428
$\geq 4$	200	0.141	0.208	0.368
All	1,512	0.187	0.268	0.470

Appendix

Table 21: GMatch@3 I by target count, which removes the output-count ceiling affecting papers with

|\mathcal{G}_{P}|>3

. Within the

\geq 4

stratum the counts are

4{:}92

5{:}55

6{:}28

7{:}14

8{:}8

9{:}3

. The advantage is present in every stratum, so it does not arise from the mix of target counts.

Model	Params	Selection	Generation
Qwen3-0.6B	0.6B	0.148	0.186
Qwen3-1.7B	1.7B	0.236	0.207
Qwen3-4B	4B	0.301	0.213
Qwen3-8B	8B	0.352	0.210
Qwen3-14B	14B	0.347	0.176
Qwen3-32B	32B	0.368	0.188

Appendix

Table 22: Eight-way selection accuracy (chance

=0.125

) and generation GRec@3 B , plotted in the main paper. Selection is evaluated on the 1,421 retained unambiguous sets and generation on the full 1,512-paper Test split; generation values for the lower block are the GRec@3 B column of the main results table. The single rank inversion at 14B is what gives

\rho=0.94

rather than

1.00

Source $P$ — DoRA: Weight-Decomposed Low-Rank Adaptation ( Liu et al. 2024 ) . Citing $Q$ — MAP: Revisiting Weight Decomposition for Low-Rank Adaptation ( Si et al. 2025b ) . Observed target $\ell$ — the direction component is defined by column-wise normalization, a choice made without a principled basis for separating magnitude from direction.
Qwen3-32B CATEGORY 0.2 The decomposition is somewhat heuristic and its design choices are not fully justified.
Stage 1: SFT FACET 0.6 The magnitude–direction decomposition is introduced without an argument for why this particular split is the right one.
Stage 2: +RSFT FACET 0.6 Direction is obtained by column-wise normalization of the weight matrix , a specific choice that the paper adopts without comparison to alternatives.
ActLim-8B SPECIFIC 1.0 Column-wise normalization is used to define the direction component, so the magnitude–direction split rests on a convention rather than a principled criterion , and the reported gains cannot be attributed to the decomposition itself.

Appendix

Table 23: One source paper across the three cumulative stages. The match level rises from CATEGORY 0.2 to SPECIFIC 1.0 as the criticism moves from the design in general, to the decomposition, to the specific operation that defines it and the reason that operation is unjustified.

Source paper, citing paper, and observed target	GPT-4-Turbo (zero-shot)	ActLim-8B
Method assumption. ShortGPT ( Men et al. 2024 ) $\rightarrow$ PIP ( Cao et al. 2025 ) . $\ell$ : layer importance is judged by an isolated input–output similarity metric, which ignores adversarial robustness.	CATEGORY 0.2 The pruning criterion is simple and may not capture every aspect of layer importance.	SPECIFIC 1.0 Importance is scored by input–output similarity at each layer in isolation , so layers that matter only under adversarial or distribution-shifted input are indistinguishable from redundant ones and are pruned away.
Data and domain dependence. LLMLingua-2 ( Pan et al. 2024 ) $\rightarrow$ LongCodeZip ( Shi et al. 2025 ) . $\ell$ : compression is token-level and domain-agnostic, so it breaks code structure.	CATEGORY 0.2 The method is evaluated on natural language and may transfer imperfectly to other kinds of text.	SPECIFIC 1.0 Compression scores tokens independently of syntax , so on source code it deletes delimiters and identifiers that the remaining tokens depend on , leaving fragments that no longer parse.
Method assumption. MDLM ( Sahoo et al. 2024 ) $\rightarrow$ Masks Can Be Distracting ( Piskorz et al. 2025 ) . $\ell$ : a strong locality bias makes predictions over-rely on immediate neighbours.	NONE 0.0 Additional benchmarks and stronger baselines would strengthen the evaluation.	SPECIFIC 1.0 Denoising conditions each masked position primarily on its immediate neighbours , so the model under-uses long-range context and degrades where the informative tokens are far from the mask.
Safety and robustness. Safety Alignment Should Be Made More Than Just a Few Tokens Deep ( Qi et al. 2024 ) $\rightarrow$ Matching Ranks Over Probability ( Vega and Singh 2025 ) . $\ell$ : SFT-style data augmentation leaves deeper alignment vulnerabilities that remain easy to bypass.	CATEGORY 0.2 Alignment may not hold under all adversarial conditions.	FACET 0.6 The proposed defense is supervised fine-tuning on augmented refusal data , which reshapes early-token behaviour without constraining the model’s deeper generation dynamics.
Efficiency (CV transfer set). MAR ( Li et al. 2024 ) $\rightarrow$ MARché ( Jiang et al. 2025 ) . $\ell$ : attention and FFN are recomputed redundantly over all tokens at every step.	CATEGORY 0.2 Generation is computationally expensive relative to simpler baselines.	SPECIFIC 1.0 Attention and feed-forward blocks are recomputed over the full token set at every step rather than reusing per-token state, so cost scales with sequence length even where token representations are unchanged .

Appendix

Table 24: Paired outputs on five source papers. The zero-shot outputs are source-faithful but name no component to act on; ActLim-8B identifies the mechanism and, in four of the five cases, the operating condition recorded in the observed target. The safety case is scored FACET 0.6 rather than SPECIFIC 1.0 because the output stops short of naming the bypass behaviour the citing paper targets. The final row is generated by a model trained on NLP only and evaluated on a Computer Vision source paper.

Opponent	W/T/L	$n_{\mathrm{eff}}$	raw $p$	$p_{\mathrm{Holm}}$
Qwen3-32B	74/9/37	111	0.00057	0.0023
GPT-4-Turbo	70/9/41	111	0.00758	0.0228
Stage 1: Verified SFT	66/10/44	110	0.04476	0.0895
Stage 2: +RSFT	59/11/50	109	0.44367	0.4437

Appendix

Table 25: Complete pairwise preference results for ActLim-8B, 120 blind comparisons per opponent. Ties are excluded, giving

n_{\mathrm{eff}}=W+L

. Raw

p

-values are exact two-sided binomial tests against

p=0.5

, Holm-corrected over this family of four comparisons.

System	Actionable	Faithful	Valid–Act.
Qwen3-32B	0.61	0.92	0.58
GPT-4-Turbo	0.72	0.91	0.67
Stage 1: Verified SFT	0.79	0.87	0.71
Stage 2: +RSFT	0.84	0.88	0.76
ActLim-8B	0.88	0.89	0.80

Appendix

Table 26: Absolute ratings over 150 outputs per system, majority vote across three annotators, rounded to two decimals. Valid–Actionable is the intersection of the two criteria and lies between

\max(0,a+f-1)

and

\min(a,f)

in every row. The joint rate rises monotonically across training stages while faithfulness alone does not, since a generic criticism can be faithful without being actionable.

Supplied signal	RetAlign
None	0.302
Zero-shot limitation	0.348
ActLim-8B limitation	0.480
Gold forward-citation limitation	0.556

Appendix

Table 27: RetAlign, macro-averaged over the same 200 papers in every condition. ActLim-8B recovers

(0.480-0.302)/(0.556-0.302)=70.1\%

of the gap between direct ideation and the gold-limitation reference.

Condition	Rel.	Spec.	Nov.	Feas.
Paper only	3.6	3.3	3.3	3.5
+ ActLim-8B limitation	4.1	3.9	3.4	3.6
Wilcoxon $W$	1,284.5	1,402.0	2,318.0	2,201.5
raw $p$ (two-sided)	0.0016	0.0043	0.412	0.287

Appendix

Table 28: Human ratings on a 1–5 scale over 100 paired examples with three raters, drawn as the first 100 of the 200-paper subset under the same seed. Relevance and specificity both improve. Inter-rater agreement is Krippendorff’s

\alpha=0.61

over the four ordinal scales.

Research Idea Generation

Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.

See all 60 papers

Structure Liberates: How Constrained Sensemaking Produces More Novel Research Output
May 1, 2026James Mooney, Zae Myung Kim, Young-Jun Lee +1AI Agents for Scientific Discovery Scientific Hypothesis Generation
University of Minnesota
LDC: Learning to Generate Research Idea with Dynamic Control
Dec 19, 2024Ruochen Li, Liqiang Jing, Chi Han +2Controllable Text Generation Research Idea Generation
University of Texas at Dallas · UIUC · Stony Brook University
IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas
Oct 6, 2026Ziyu Chen, Yilun Zhao, Jiashuo Sun +3Research Idea Generation Large Language Model-Guided Scientific Discovery
The Unversity of Chicago · Yale University · University of Illinois Urbana-Champaign +1

May 1, 2026J/K move · Enter open · S save

Structure Liberates: How Constrained Sensemaking Produces More Novel Research Output

James Mooney, Zae Myung Kim, Young-Jun Lee, Dongyeop Kang

University of Minnesota

AI Agents for Scientific Discovery Scientific Hypothesis Generation Research Idea Generation Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

Scientific discovery is an extended process of ideation -- surveying prior work, forming hypotheses, and refining reasoning -- yet existing LLM research agents treat this phase as a brief preamble despite its central role in research. We introduce SciSense, a sensemaking-grounded framework that operationalizes ideation as a structured sequence of eight cognitive stages (Pirolli and Card, 2005). We construct SciSense-Traj, a dataset of 120K citation-conditioned research trajectories in two modes: Hindsight, where an LLM reconstructs the ideation path from a paper's cited works to the paper itself, and Foresight, where the LLM proposes novel directions from the same citations. We distill these into SciSense-LM, a family of sensemaking LLMs spanning 3B to 70B parameters. Contrary to the assumption that looser supervision promotes exploration, Hindsight-trained models improve trajectory quality by 2.0% over Foresight-trained models while also producing more novel and diverse outputs, across all eight model families we test. The advantage propagates downstream: coding agents conditioned on Hindsight trajectories produce research artifacts with higher executability, scientific grounding, and utility than those conditioned on Foresight trajectories. This suggests that outcome-anchored ideation reduces the cognitive burden on downstream agents, freeing them to explore more creatively. SciSense offers both a practical tool for augmenting LLM-driven research workflows and a principled testbed for studying how planning shapes scientific discovery.

Abstract

Research Idea Generation

Explore similar work

Structure Liberates: How Constrained Sensemaking Produces More Novel Research Output

LDC: Learning to Generate Research Idea with Dynamic Control

IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas