MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts

Weiyue Li; Ruizhi Qian; Yi Li; Yongce Li; Yunfan Long; Jiahui Cai; Yan Luo; Mengyu Wang

cs.CLApr 7, 2026

MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts

Authors: Weiyue Li, Ruizhi Qian, Yi Li, Yongce Li, Yunfan Long, Jiahui Cai, Yan Luo, Mengyu Wang

Abstract

Large language models (LLMs) are widely explored for reasoning-intensive research tasks, yet resources for testing whether they can infer scientific conclusions from structured biomedical evidence remain limited. We introduce $\textbf{MedConclusion}$ , a large-scale dataset of $\textbf{5.7M}$ PubMed structured abstracts for biomedical conclusion generation. Each instance pairs the non-conclusion sections of an abstract with the original author-written conclusion, providing naturally occurring supervision for evidence-to-conclusion reasoning. MedConclusion also includes journal-level metadata such as biomedical category and SJR, enabling subgroup analysis across biomedical domains. As an initial study, we evaluate diverse LLMs under conclusion and summary prompting settings and score outputs with both reference-based metrics and LLM-as-a-judge. We find that conclusion writing is behaviorally distinct from summary writing, strong models remain closely clustered under current automatic metrics, and judge identity can substantially shift absolute scores. MedConclusion provides a reusable data resource for studying scientific evidence-to-conclusion reasoning. Our code and data are available at: https://github.com/Harvard-AI-and-Robotics-Lab/MedConclusion.

Figures & tables

	Dataset					Study Design
Name	Data Size	Broad Biomed.	Struc. Abs.	Gold References	Journal Metadata	Summary Contrast	Judge Robust.
Gao et al. (2024)	17.4K	✓	✓	✓	✗	✗	✗
Shieh et al. (2019)	195.7K	✗	✓	✓	✗	✗	✗
Tang et al. (2022)	57.1K	✗	✗	✓	✗	✗	✗
Tang et al. (2023)	200.2K	✗	✗	✓	✗	✗	✗
Bastan et al. (2022)	633K	✓	✗	✓	✗	✗	✗

Table 1: Comparison of MedConclusion with conclusion-centric prior work. Checks denote properties that are central and explicitly emphasized by each resource or study. For multi-part resources, total dataset size sums all released components used in the paper.

Figure 1: Overview of MedConclusion and the evaluation pipeline. Left: an example MedConclusion instance, including article metadata, subject categories, and a structured abstract, where the non-conclusion sections are used as model input and the author-written conclusion serves as the gold reference. Right: the non-conclusion abstract is paired with prompts and given to diverse LLM families to generate conclusions, which are then compared against the ground-truth conclusion using both rule-based reference metrics and multi-dimensional LLM-as-a-judge metrics.

Model	Semantic Sim. $\uparrow$	Writing Style Sim. $\uparrow$	Non-Contradiction Rate $\uparrow$	Numeric Consistency $\uparrow$	Formality Sim. $\uparrow$
General-purpose Models
GPT-5.4	73.22	71.21	84.61	88.24	89.80
Gemini 3.1 Pro	71.87	70.13	82.02	86.92	89.49
Gemini 3 Flash	71.33	69.87	81.76	86.45	89.17
Gemma-3-27B	71.03	69.18	81.55	84.13	89.36
DeepSeek-V3.2	69.47	68.21	80.31	86.22	88.59

Table 2: LLM-as-Judge evaluation scores for conclusion generation. Models are grouped by primary capability. Bold and underline denote the best and second-best scores, respectively.

Model	WC Ratio $\downarrow$	SC Ratio $\downarrow$	Embed. Sim. $\uparrow$	ROUGE- 1 $\uparrow$	ROUGE- 2 $\uparrow$	ROUGE- L $\uparrow$	BLEU $\uparrow$	PPL Orig. $\downarrow$	PPL Gen. $\downarrow$
General-purpose Models
GPT-5.4	2.19	1.71	0.77	0.34	0.10	0.21	0.04	70.26	40.39
Gemini 3.1 Pro	2.28	1.75	0.73	0.33	0.10	0.21	0.04	70.26	34.54
Gemini 3 Flash	2.12	1.69	0.74	0.34	0.10	0.21	0.04	70.26	33.87
Gemma-3-27B	2.49	2.01	0.77	0.32	0.09	0.20	0.04	70.26	30.46
DeepSeek-V3.2	1.73	1.38	0.76	0.35	0.11	0.23	0.05	70.26	47.14

Table 3: Rule-based evaluation scores for conclusion generation, same order as Table 2 .

Mode	Model	Semantic Sim. $\uparrow$	Writing Style Sim. $\uparrow$	Non-Contradiction Rate $\uparrow$	Numeric Consistency $\uparrow$	Formality Sim. $\uparrow$
Prompt without Formatting Restriction
{tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) A;	GPT-5.4	73.22	71.20	84.61	88.24	89.80
	Gemini 3 Flash	71.33	69.87	81.76	86.45	89.17
{tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) B;	GPT-5.4	72.11	62.60	83.96	66.24	88.47
	Gemini 3 Flash	71.04	61.55	82.44	58.09	88.11
Prompt with Length and Writing Style Restriction

Table 4: LLM-as-Judge evaluation scores across prompt settings and generation modes. {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) A; and {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) C; are generating conclusions, whereas {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) B; and {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) D; are generating summaries.

Model	Semantic Sim. $\uparrow$	Writing Style Sim. $\uparrow$	Non-Contradiction Rate $\uparrow$	Numeric Consistency $\uparrow$	Formality Sim. $\uparrow$
Judge: GPT-5.4-mini
GPT-5.4	73.22	71.20	84.61	88.24	89.80
Gemini 3.1 Pro	71.87	70.13	82.02	86.92	89.49
Gemini 3 Flash	71.33	69.87	81.76	86.45	89.17
Judge: Gemini 3 Flash
GPT-5.4	84.30	71.49	97.51	98.18	92.50

Table 5: LLM-as-Judge evaluation scores: GPT-5.4-mini vs Gemini 3 Flash as judge.

Figure 2: Scatter plots of journal-level SJR scores versus evaluation metrics for GPT-5.4 under the {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) A; setting (outliers removed). Each point represents one journal, aggregated by mean score. The top row shows reference-based metrics and the bottom row shows LLM-judge dimensions. Pearson ( $r$ ) and Spearman ( $\rho$ ) correlations are annotated in each panel ( ${*}p<0.05$ , ${**}p<0.01$ , ${***}p<0.001$ ).

Panel of Figure 3 — Figure 3: Radar chart comparison of the top 5 and bottom 5 biomedical categories ranked by both mean Semantic Similarity and mean ROUGE-L for GPT-5.4 under setting {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) A; . Each axis is normalized to $[0,1]$ .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 6 — Table 6: Example datapoint used for prompt construction and evaluation. In the structured abstract, the non-conclusion sections are highlighted separately from the CONCLUSION section to indicate that the former are used as model input, while the conclusion serves as the ground-truth reference for evaluation.

Panel of Table 7 — Table 7: Prompts for conclusion generation. The highlighted variants distinguish the constrained writing setting ( {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) C; ), which enforces sentence and word count targets and asks the model to match the abstract’s writing style, from the unconstrained setting ( {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) A; ), which instead only requires a formal academic style without length constraints.

Panel of Table 8 — Table 8: Prompts for summary generation. The highlighted variants distinguish the constrained writing setting ( {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) D; ), which requires the summary to follow the abstract’s writing style and satisfy sentence and word count targets, from the unconstrained setting ( {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) B; ), which only specifies a formal academic style and leaves length unrestricted.

Panel of Table 9 — Table 9: Prompt for LLM judge

Panel of Table 10 — Table 10: Example from Experimental and Cognitive Psychology . ROUGE-1: 0.480, ROUGE-2: 0.130, ROUGE-L: 0.272, BLEU: 0.068, Perplexity: 41.1, Semantic Sim.: 88, Writing Style Sim.: 84, Non-Contradiction Rate: 96, Numeric Consistency: 100, Formality Sim.: 92.

Panel of Table 11 — Table 11: Example from Endocrine and Autonomic Systems . ROUGE-1: 0.358, ROUGE-2: 0.065, ROUGE-L: 0.232, BLEU: 0.020, Perplexity: 46.5, Semantic Sim.: 88, Writing Style Sim.: 82, Non-Contradiction Rate: 96, Numeric Consistency: 100, Formality Sim.: 94.

Panel of Table 12 — Table 12: Example from Advanced and Specialized Nursing . ROUGE-1: 0.198, ROUGE-2: 0.000, ROUGE-L: 0.123, BLEU: 0.004, Perplexity: 15.7, Semantic Sim.: 88, Writing Style Sim.: 82, Non-Contradiction Rate: 96, Numeric Consistency: 100, Formality Sim.: 90.

Panel of Table 13 — Table 13: Example from Environmental Science . ROUGE-1: 0.375, ROUGE-2: 0.231, ROUGE-L: 0.300, BLEU: 0.039, Perplexity: 15.8, Semantic Sim.: 88, Writing Style Sim.: 82, Non-Contradiction Rate: 98, Numeric Consistency: 100, Formality Sim.: 95.

Panel of Table 14 — Table 14: Example from Emergency Nursing . ROUGE-1: 0.341, ROUGE-2: 0.140, ROUGE-L: 0.182, BLEU: 0.027, Perplexity: 20.2, Semantic Sim.: 88, Writing Style Sim.: 82, Non-Contradiction Rate: 94, Numeric Consistency: 96, Formality Sim.: 90.

Panel of Table 15 — Table 15: Example from Pollution . ROUGE-1: 0.378, ROUGE-2: 0.165, ROUGE-L: 0.252, BLEU: 0.030, Perplexity: 18.8, Semantic Sim.: 62, Writing Style Sim.: 68, Non-Contradiction Rate: 78, Numeric Consistency: 100, Formality Sim.: 85.

Panel of Table 16 — Table 16: Example from Health, Toxicology and Mutagenesis . ROUGE-1: 0.381, ROUGE-2: 0.083, ROUGE-L: 0.245, BLEU: 0.027, Perplexity: 21.5, Semantic Sim.: 62, Writing Style Sim.: 78, Non-Contradiction Rate: 58, Numeric Consistency: 100, Formality Sim.: 92.

Panel of Table 17 — Table 17: Example from Computer Science Applications . ROUGE-1: 0.303, ROUGE-2: 0.082, ROUGE-L: 0.182, BLEU: 0.025, Perplexity: 96.5, Semantic Sim.: 62, Writing Style Sim.: 54, Non-Contradiction Rate: 78, Numeric Consistency: 100, Formality Sim.: 88.

Panel of Table 18 — Table 18: Example from Applied Microbiology and Biotechnology . ROUGE-1: 0.177, ROUGE-2: 0.036, ROUGE-L: 0.088, BLEU: 0.006, Perplexity: 79.3, Semantic Sim.: 62, Writing Style Sim.: 58, Non-Contradiction Rate: 78, Numeric Consistency: 12, Formality Sim.: 84.

Panel of Table 19 — Table 19: Example from Software . ROUGE-1: 0.264, ROUGE-2: 0.050, ROUGE-L: 0.198, BLEU: 0.008, Perplexity: 106.9, Semantic Sim.: 62, Writing Style Sim.: 58, Non-Contradiction Rate: 88, Numeric Consistency: 100, Formality Sim.: 84.

Category	$\Delta$ Semantic Sim.	$\Delta$ Writing Style	$\Delta$ Non-Contrad. Rate	$\Delta$ Numeric Cons.	$\Delta$ Formality Sim.
Immunology & Microbiology	$+6.0$	$+13.0$	$+4.8$	$+28.0$	$+2.3$
Pharm., Toxicology & Pharmaceutics	$+5.5$	$+13.5$	$+4.9$	$+33.0$	$+2.4$
Exp. & Cognitive Psychology	$+5.3$	$+7.9$	$+3.4$	$+27.5$	$+2.1$
Arts & Humanities	$+5.1$	$+12.3$	$+6.5$	$+32.4$	$+2.8$
Virology	$+4.8$	$+13.0$	$+7.2$	$+23.2$	$+3.1$
Nursing	$+4.4$	$+13.7$	$+3.8$	$+30.3$	$+1.8$

Appendix

Table 20: Top 10 categories ranked by

|\Delta\text{Semantic Similarity}|

(Mode {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) A;

-

{tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) B; ; GPT-5.4 ). Gap magnitudes vary substantially across dimensions within this group, indicating that the specific dimensions along which conclusion and summary modes diverge are category-dependent.

Category	$\Delta$ Semantic Sim.	$\Delta$ Writing Style	$\Delta$ Non-Contrad. Rate	$\Delta$ Numeric Cons.	$\Delta$ Formality Sim.
Cancer Research	$+0.2$	$+7.6$	$-0.5$	$+16.7$	$+0.7$
Organic Chemistry	$-0.2$	$+7.1$	$-1.2$	$+18.1$	$+1.2$
Endocrinology, Diabetes & Metab.	$+0.1$	$+7.3$	$-0.2$	$+14.3$	$+0.9$
Cellular & Mol. Neuroscience	$-0.1$	$+6.8$	$+0.4$	$+16.5$	$+1.1$
Hepatology	$-0.1$	$+4.8$	$+0.7$	$+15.1$	$+1.1$
Biochemistry	$-0.1$	$+5.4$	$+0.6$	$+16.4$	$+0.2$

Appendix

Table 21: Bottom 10 categories ranked by

|\Delta\text{Semantic Similarity}|

(Mode {tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) A;

-

{tikzpicture} [baseline=(char.base)] \node [shape=circle, draw, inner sep=0.5pt, fill=black, text=white, font= ] (char) B; ; GPT-5.4 ). Despite near-zero semantic gaps, writing style (

+4.8

+8.4

) and numeric consistency (

+14.3

+23.1

) remain substantially positive.

Figure 4: Publication-year density of abstracts in MedConclusion from 2000–2025. The median publication year is 2018.

Figure 5: Distribution statistics of MedConclusion. (a) Top 10 subject categories by proportion of abstracts in the dataset, with Medicine being the dominant category at 13.8%. (b) Distribution of SJR scores across the 3,772 journals (median

=0.77

, mean

=0.98

), showing a right-skewed distribution concentrated in the low-to-moderate prestige range.

Panel of Table 22 — Table 22: Conclusion label variants used to identify conclusion-type sections in structured abstracts.

Short Name	Full Name	Access	Capability	Scale	Max New Tokens	Temp.
General-purpose Models
GPT-5.4	gpt-5.4 ( OpenAI, 2026 )	Proprietary	General-purpose	Large	1024	0
Gemini 3.1 Pro	gemini-3.1-pro ( Google DeepMind, 2026 )	Proprietary	General-purpose	Large	1024	0
Gemini 3 Flash	gemini-3-flash ( Google DeepMind, 2025 )	Proprietary	General-purpose	Medium	1024	0
DeepSeek-V3.2	DeepSeek-V3.2 ( DeepSeek-AI, 2025b )	Proprietary	General-purpose	Large	1024	0
MiniMax-M2.1	MiniMax-M2.1 ( MiniMax, 2025 )	Proprietary	General-purpose	Large	1024	0

Appendix

Table 23: Model metadata and run-configuration summary for all evaluated models. For general-purpose models that expose a thinking mode, thinking=none was used.

Evidence-Grounded Generation

Evidence-grounded generation produces answers and documents whose claims are supported by traceable sources.

See all 148 papers

MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies
May 10, 2026Huy Hoang Ha, Benoit Favre, Francois PortetHealthcare RAG Evaluation
Divide-Prompt-Refine: a Training-Free, Structure-Aware Framework for Biomedical Abstract Generation
May 20, 2026Sylvey Lin, Joe Menke, Shufan Ming +3AI-Assisted Scientific Research Text Summarization
School of Information Sciences, University of Illinois Urbana-Champaign, Champaign, IL · Department of Psychiatry, University of Illinois College of Medicine, Chicago, IL
PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering
Mar 28, 2026Yiqing Zhang, Xiaozhong Liu, Fabricio MuraiBiomedical QA Agentic RAG
PayPal · Worcester Polytechnic Institute

May 10, 2026J/K move · Enter open · S save

MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies

Huy Hoang Ha, Benoit Favre, Francois Portet

Healthcare RAG Evaluation Clinical Language Model Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) have saturated standard medical benchmarks that test factual recall, yet their ability to perform higher-order reasoning, such as synthesizing evidence from multiple sources, remains critically under-explored. To address this gap, we introduce MedMeta, the first benchmark designed to evaluate an LLM's ability to generate conclusions from medical meta-analyses using only the abstracts of cited studies. MedMeta comprises 81 meta-analyses from PubMed (2018--2025) and evaluates models using two distinct workflows: a Retrieval-Augmented Generation (Golden-RAG) setting with ground-truth abstracts, and a Parametric-only approach relying on internal knowledge. Our evaluation framework is validated by a well-structured analysis showing our LLM-as-a-judge protocol strongly aligns with human expert ratings, as evidenced by high Pearson's r correlation (0.81) and Bland-Altman analysis revealing negligible systematic bias, establishing it as a reliable proxy for scalable evaluation. Our findings underscore the critical importance of information grounding: the Golden-RAG workflow consistently and significantly outperforms the Parametric-only approach across models. In contrast, the benefits of domain-specific fine-tuning are marginal and largely neutralized when external material is provided. Furthermore, stress tests show that all models, regardless of architecture, fail to identify and reject negated evidence, highlighting a critical vulnerability in current RAG systems. Notably, even under ideal RAG conditions, current LLMs achieve only slightly above-average performance (~2.7/5.0). MedMeta provides a challenging new benchmark for evidence synthesis and demonstrates that for clinical applications, developing robust RAG systems is a more promising direction than model specialization alone.

Abstract

Evidence-Grounded Generation

Explore similar work

MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies

Divide-Prompt-Refine: a Training-Free, Structure-Aware Framework for Biomedical Abstract Generation

PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering