OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources

Martin Docekal; Martin Fajcik; Pavel Smrz

cs.CLMay 3, 2024

OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources

Authors: Martin Docekal, Martin Fajcik, Pavel Smrz

Organizations: Brno University of Technology

Abstract

This paper introduces OARelatedWork: a dataset for related work generation from open-access sources. It is the first large-scale multi-document summarization dataset for related work generation, containing whole related work sections and full texts of cited papers. Its validation and test splits are constructed so that every source paper is available in full text, enabling controlled evaluation of full-text related work generation. The dataset includes 94 450 papers from multiple domains. With OARelatedWork, we aim to shift the field from generating parts of related work sections from abstracts only to generating entire related work sections from all available content. We (i) benchmark a wide spectrum of models, highlighting that synthesizing massive full-text contexts remains challenge for tested cost-effective Large Language Models (LLMs): under our statement-level judge, GPT-4o-mini's evidence-grounded True rate drops from 92.9% with abstracts to 83.8% with full texts. We (ii) empirically analyze human writing behavior through a human evaluation over 40 papers and 408 factual statements, revealing that authors frequently introduce abstractive claims ungrounded in localized source texts; consequently, tested LLMs actually surpass human baselines in strict, evidence-grounded factuality. Finally, we (iii) conduct a fine-grained meta-evaluation, revealing that standard reference-based metrics are inadequate for evaluating such long-form structured outputs, and introduce a robust statement-level evaluation framework to address this gap.

Figures & tables

Figure 1: The task is to generate a whole related work section from cited papers and the rest of the target paper. We also try variants using abstracts instead of full-texts.

Dataset	Samples (Tr/Va/Te)	Tgt. Words	Cited Words	Docs	Para.
S2ORC Chen et al. (2021)	126k / 5k / 5k	148.0	1.1k	5.0	1.0
Delve Chen et al. (2021)	73k / 3k / 3k	181.0	0.6k	3.7	1.0
Multi-XSci. Lu et al. (2020)	30k / 5k / 5k	116.4	0.8k	4.4	1.0
STRoGeNS Nishimura et al. (2024)	86k / - / -	514.3	3.0k	16.6	4.2
OARelatedWork (Ours)
Train	91k	530.3	26.2k	11.4	4.5

Table 1: Comparison of our dataset with existing related-work generation datasets.

Model

Input

citations

True

Wrong Cit.

Unverifiable

False

References

Target Paper

Abs.

Cont.

Abs.

Cont.

LEAD

✓

99.1_{98.8}^{99.3}

0.2_{0.1}^{0.4}

0.6_{0.5}^{0.9}

0.0_{0.0}^{0.0}

Greedy Oracle

✓

98.1_{97.6}^{98.5}

0.1_{0.0}^{0.2}

1.1_{0.8}^{1.4}

0.6_{0.4}^{0.9}

✓

96.3_{95.7}^{96.7}

0.7_{0.5}^{0.9}

2.3_{1.9}^{2.7}

0.3_{0.2}^{0.5}

✓

96.0_{95.3}^{96.5}

0.4_{0.3}^{0.6}

2.4_{2.0}^{2.8}

0.4_{0.3}^{0.5}

Table 2: Test set results on multiple input variants. Inputs with Greedy Oracle (GO) are texts pre-extracted using Greedy Oracle extractor. Factuality metrics (True, Wrong Cit., Unverifiable, False) include 95% bootstrap confidence intervals formatted as

Mean_{Lower}^{Upper}

. Missing percentages reflect judge output errors.

Attribute	Model Pair (A vs. B)	TWR (A)	95% CI
Preference	GPT-4o-mini vs. PRIMERA	0.825	[0.725, 0.913]
	Human vs. PRIMERA	0.456	[0.338, 0.581]
	GPT-4o-mini vs. Human	0.800	[0.700, 0.894]
Relevance	GPT-4o-mini vs. PRIMERA	0.756	[0.656, 0.844]
	Human vs. PRIMERA	0.488	[0.381, 0.600]
	GPT-4o-mini vs. Human	0.788	[0.694, 0.869]

Table 3: Pairwise human document level evaluation results based on True Win Rate (TWR; ties award 0.5 points). Winning models are highlighted in bold. When a tie occurs (the neutral baseline of 0.500 falls within the 95% Bootstrap Confidence Interval), the entire row is bolded.

Figure 2: Violin plot of bootstrap distributions of Kendall’s

\tau

correlations of automatic metrics with human judgment on OARelatedWork meta-evaluation dataset for statement-level Faithfulness. Thick line is showing 95% confidence interval.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: An example of hierarchy parsing showing two major steps: Internal Numbering and creation of Anchor. Orange boxes contain notes with obstacles for the parser. See that we are able to guess section numbers for headlines with missing numbers by exploiting writing habits and format. The introduction is usually the first section, and we can see that the capital letters format is used for all (known) top-level headlines. The headline * H4 23 y 6 will be removed in the cleaning step.

Figure 4: Illustration of input format for PRIMERA. The red boxes are global tokens. We put the special tokens <unstructured> or <structured> to identify whether we want to generate subsections. The length prompt is the number of tokens (in text form) of a reference target sequence. GO means content filtered by the Greedy Oracle model. In experiments, the GO is also used for filtering full-text of cited documents.

Panel of Figure 5 — Figure 5: Histograms showing lengths of input types and lengths of outputs on train set. These numbers are obtained from PRIMERA tokenized inputs/outputs. GO for target paper is not created from abstract. GO summaries of cited documents are created for each document separately. We cut the top part of histograms to fit them on the page, thus the maximum is not there.

Panel of Figure 6 — Figure 6: Field of study domain shift between all papers in used corpus and dataset splits. As a single paper may have multiple fields of study, the counts do not add up to the total number of papers. The tpm is task (project management) .

Family	Specific Checkpoint / Version	Access Type
GPT	GPT-5.4	Prop. API
GPT	GPT-5.4-mini	Prop. API
Gemini	gemini-3-1-pro-preview	Prop. API
Gemini	gemini-3-1-flash-lite-preview	Prop. API
Llama	meta-llama/Llama-3.3-70B-Instruct	Open (vLLM)
Gemma	google/gemma-4-31B-it	Open (vLLM)

Appendix

Table 4: Details of the LLM evaluators used in the statement-level factuality assessment. Open-weight models were executed locally using the vLLM engine, while proprietary models were accessed via their respective official APIs.

Aspect	$\boldsymbol{\kappa}$
Preference	0.5099
Relevance	0.5452
Language	0.1707
Faithfulness	0.4301
Statement (all)	0.6169
Statements (randomly selected)	0.6556

Appendix

Table 5: Inter-annotator agreement. We use quadratic weighted Cohen’s

\kappa

for preference-based judgments (Preference/Relevance/Faithfulness/Language) because annotators could select no preference . We use standard Cohen’s

\kappa

for statement labels (Statements).

Model	N	True	W. Cite	False	Unver.
All Statements
PRIMERA	140	58.6	5.7	30.7	5.0
Human	121	73.6	0.8	6.6	19.0
GPT-4o-mini	147	78.2	2.7	5.4	13.6
Randomly Selected
PRIMERA	115	61.7	5.2	28.7	4.4

Appendix

Table 6: Fine-grained factuality evaluation. Metrics (aside from the total statement count,

N

) are reported as percentages (%). "W. Cite" denotes statements that are true but have the wrong citation, and "Unver." denotes unverifiable statements. The best performing model (highest True and lowest False) is in bold.

Figure 8: Violin plot showing 95% confidence interval of Kendall’s

\tau

correlations of automatic metrics with human judgment on LongEval’s PubMed dataset.

Evaluator Model	Subset	Class-Specific F1 Scores				Macro F1
Evaluator Model	Subset	True	W. Cite	Unverifiable	False	Macro F1
Human 82	All ( $N=408$ )	0.74	0.72	0.46	0.57	0.62
	Random ( $N=341$ )	0.76	0.74	0.45	0.49	0.61
	Adversarial ( $N=67$ )	0.59	0.67	0.49	0.73	0.62
Human 83	All	0.76	0.42	0.43	0.46	0.52
	Random	0.76	0.40	0.41	0.47	0.51

Appendix

Table 7: Detailed performance breakdown of evaluator models across the All, Random, and Adversarial meta-annotated statement subsets. Class-specific F1 scores highlight model calibration on minority classes such as "True, but wrong citation" (W. Cite) and "Unverifiable".

Text Summarization

Text summarization develops methods for condensing documents while preserving the information, perspective, and evidence readers need.

See all 142 papers

RWGBench: Evaluating Scholarly Positioning in Related Work Generation
May 30, 2026Anzhe Xie, Weihang Su, Jiaxin Mao +4LLM Evaluation Automated Evaluation
Tsinghua University Beijing, China · Renmin University of China Beijing, China
GRASP: Graph-Reasoning Aided Survey Planning for High-Fidelity Related Work Generation
Jul 4, 2026Haoming Li, Jessica OuyangCitation Network Analysis Graph-Based Planning
Department of Computer Science University of Texas at Dallas Richardson, TX 75080
Assembling the CREW: A Collaborative Multi-agent Reinforcement Learning Framework for Automated Related Work Generation
Sep 14, 2026Hai-Dang Dang, Bao-Yen Pham, Bao Nguyen +2Multi-Agent LLM Systems RL for LLM Agents
Hanoi University of Science and Technology · The Chinese University of Hong Kong · Hanoi University of Industry

May 30, 2026J/K move · Enter open · S save

RWGBench: Evaluating Scholarly Positioning in Related Work Generation

Anzhe Xie, Weihang Su, Jiaxin Mao, Yiqun Liu+3

Tsinghua University Beijing, China · Renmin University of China Beijing, China

LLM Evaluation Automated Evaluation Generative AI Evaluation

Open paper AskPDF arXiv

Large language models have shown strong fluency in scientific writing, yet the evaluation of related work generation (RWG) remains limited. Existing RWG evaluations largely inherit summarization-oriented metrics, using lexical or semantic similarity to reference sections as proxies for quality. However, related work writing is fundamentally a citation-level scholarly positioning task: it requires selecting, organizing, and framing prior work to clarify how a target paper relates to, differs from, and contributes beyond existing research.As a result, models may generate coherent and semantically-relevant text while exhibiting academically critical failures, such as inappropriate citation selection or misplaced references, that conventional metrics do not capture.To this end, we introduce RWGBench, a benchmark that evaluates RWG from the perspective of citation decision-making rather than text similarity. RWGBench is constructed from a large-scale collection of 40,108 computer science papers and a retrieval corpus of 1.09 million documents, with a carefully curated test set comprising 100 papers and their corresponding published related work sections.We propose a multi-dimensional evaluation framework that assesses citation selection, contextual appropriateness, organization, and discourse structure.Experiments reveal systematic limitations in current systems that are obscured by standard evaluations, while Oracle studies further disentangle retrieval-level and generation-level bottlenecks. Human evaluation further shows that our citation-centric metrics align substantially better with expert judgment than surface-level text metrics. RWGBench offers a citation-centric testbed for developing and evaluating related work generation systems that are better aligned with scholarly writing practices.

Abstract

Text Summarization

Explore similar work

RWGBench: Evaluating Scholarly Positioning in Related Work Generation

GRASP: Graph-Reasoning Aided Survey Planning for High-Fidelity Related Work Generation

Assembling the CREW: A Collaborative Multi-agent Reinforcement Learning Framework for Automated Related Work Generation