Loss-Based Active Learning for Neural Abstractive Summarization

Michail Ioannou; Tatiana Passali; George Michalopoulos; Grigorios Tsoumakas

cs.CLAug 26, 2026

Loss-Based Active Learning for Neural Abstractive Summarization

Authors: Michail Ioannou, Tatiana Passali, George Michalopoulos, Grigorios Tsoumakas

Organizations: School of Informatics Aristotle University of Thessaloniki Greece

Abstract

Fine-tuning abstractive summarization models requires high-quality annotated data. However, obtaining such corpora is expensive and time-consuming, as it requires human annotators to read and comprehend long documents to create accurate summaries. Active learning mitigates this issue by selecting only the most informative instances for annotation, allowing models to achieve competitive results with significantly fewer labels. However, the application of active learning to summarization remains under-explored, and existing studies often suffer from instability and significant computational bottlenecks. To overcome these challenges, we propose LOBSTER (LOss-BaSed acTivE leaRning), a novel active learning framework designed specifically for abstractive summarization. LOBSTER improves performance by prioritizing unlabeled instances semantically similar to the model's current high-loss training examples, enabling the model to explicitly correct its specific weaknesses. Our empirical evaluation across three benchmark datasets and two summarization backbone models demonstrates that LOBSTER consistently matches or outperforms current state-of-the-art approaches while achieving a query selection speedup of up to 665x.

Figures & tables

Figure 1: The three-stage pipeline of LOBSTER. (1) Loss-based Hard Example Selection identifies “Hard Example” set $\mathcal{H}$ (red squares) via high cross-entropy loss to serve as semantic anchors. (2) Representative Candidate Filtering uses IDDS to distill a pool of representative candidates (green circles) from the unlabeled set $\mathcal{U}$ . (3) Semantic Hardness Projection computes the Max-Similarity Score $S(x_{u})$ (Eq. 3) using Sentence-BERT embeddings. This helps to identify a “semantic twin” for each hard example, forming the final Top- $B$ batch for annotation.

Panel of Figure 2 — Figure 2: Number of labeled instances vs. ROUGE-1. Comparison of active strategies using BART and PEGASUS backbones on AESLC, XSum, and CNN/DM datasets.

		Avg. Selection Time (s)
Model	Dataset	BAS	DUAL	IDDS	LOBSTER
BART-base	AESLC	46.1	38.7	0.02	0.6
	XSum	66.6	63.7	0.4	0.8
	CNN/DM	70.2	69.2	0.3	0.8
PEGASUS-large	AESLC	221.1	113.6	0.01	1.0
	XSum	390.0	267.6	0.4	1.5

Table 1: Average runtime (in seconds) for instance selection, for one active learning iteration, across different strategies. Comparison across backbones and datasets.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Subset	# Ins.	Doc. len.	Sum. len.
Gigaword	Train	200	39.6	11.2
Gigaword	Test	1K	38.3	10.4
AESLC	Train	14,436	115.3	4
AESLC	Test	1,906	114.6	4.1
XSum	Train	204,045	388.6	23.1
XSum	Test	1K	384.3	23

Appendix

Table 2: Statistics for the datasets used in this work. We report the number of instances (# Ins.) and the average length of source documents (Doc. len.) and summaries (Sum. len.) for both training and test splits.

Hyperparameter	BART-base	PEGASUS-large
Learning Rate	$2e^{-5}$	$5e^{-5}$
Batch Size (Train)	16	8
Batch Size (Eval)	32	16
Num. Epochs	6	4
Optimizer	AdamW	AdamW
Weight Decay	0.028	0.03

Appendix

Table 3: Hyperparameters used for fine-tuning the backbone summarization models.

Dataset: AESLC
Model	Method	ROUGE-1	ROUGE-2	ROUGE-L	BERTScore
BART-base	Random	26.6 $\pm$ 0.65	13.2 $\pm$ 0.65	26.0 $\pm$ 0.63	13.51 $\pm$ 0.57
	BAS	26.5 $\pm$ 0.17	13.1 $\pm$ 0.31	25.8 $\pm$ 0.24	13.63 $\pm$ 0.99
	IDDS	27.0 $\pm$ 0.32	12.5 $\pm$ 0.32	25.4 $\pm$ 0.31	13.03 $\pm$ 0.19
	DUAL	26.9 $\pm$ 0.86	13.2 $\pm$ 0.70	26.0 $\pm$ 0.85	12.95 $\pm$ 0.14
	LOBSTER	27.1 $\pm$ 0.88	13.4 $\pm$ 0.45	26.3 $\pm$ 0.80	12.96 $\pm$ 0.65

Appendix

Table 4: Summarization performance (ROUGE and rescaled BERTScore) with a 150-instance annotation budget. Results are reported as mean

\pm

standard deviation. The best performance for each metric and dataset is highlighted in bold.

ROUGE-1
		LOBSTER vs. Baseline ( $p$ -value)
Dataset	Model	DUAL	Random	IDDS	BAS
AESLC	BART-base	0.1681	0.4250	0.0083 ${}^{*}\uparrow$	0.0203 ${}^{*}\uparrow$
AESLC	PEGASUS-large	$<$ 0.001 ${}^{*}\uparrow$	$<$ 0.001 ${}^{*}\uparrow$	0.5113	$<$ 0.001 ${}^{*}\uparrow$
XSum	BART-base	0.0224 ${}^{*}\downarrow$	0.1649	0.6326	0.2942
XSum	PEGASUS-large	0.2192	0.8924	$<$ 0.001 ${}^{*}\uparrow$	0.2532

Appendix

Table 5: Paired permutation test p-values comparing LOBSTER against baseline methods at the 150-instance annotation budget for ROUGE-1 and BERTScore. An asterisk (*) indicates a statistically significant difference (

p<0.05

). For significant results, arrows indicate whether LOBSTER scored significantly higher (

\uparrow

) or lower (

\downarrow

) than the baseline.

Dataset	Setup	ROUGE (1 / 2 / L)	BERTScore
AESLC	Llama-3-8B-Instruct (Zero-shot)	27.16 / 12.06 / 25.22	11.61
	BART-base + LOBSTER	27.10 / 13.40 / 26.30	12.96
	PEGASUS-large + LOBSTER	22.10 / 10.90 / 21.50	0 7.66
XSum	Llama-3-8B-Instruct (Zero-shot)	29.41 / 0 8.40 / 21.20	29.13
	BART-base + LOBSTER	32.50 / 10.60 / 25.30	37.00
	PEGASUS-large + LOBSTER	43.50 / 20.30 / 35.00	47.72

Appendix

Table 6: Performance comparison of models fine-tuned using examples selected by LOBSTER (at a 150-instance annotation budget) against a zero-shot Llama-3-8B-Instruct baseline. Bold values indicate the best performance for each metric within each dataset.

Model	Dataset	Active Learning
		LOBSTER	DUAL	BAS	IDDS	Rand.
BART-base	AESLC	1900	1900	1300	—	2000
	XSum	2000	1700	1700	—	1600
	CNN/DM	50	60	110	100	100
PEGASUS-large	AESLC	1800	1800	2000	2000	1800
	XSum	80	90	80	100	80

Appendix

Table 7: Data Efficiency Analysis: Number of annotated samples required to reach 90% of the full-dataset performance. Bold values indicate the winning strategy for each configuration, and a dash (—) indicates that the configuration failed to reach the 90% target.

		Avg Selection time (s)
Model	Dataset	BAS	DUAL	IDDS	LOBSTER
BART-base	AESLC	61.8	366.2	0.3	3.9
	XSum	96.1	560.5	2.8	12.1
	CNN/DM	70.2	69.2	0.3	0.8
PEGASUS-large	AESLC	164.6	618.1	0.3	12.8
	XSum	390.0	267.6	0.4	1.5

Appendix

Table 8: Data Efficiency Analysis: Average runtime (in seconds) for instance selection, for one active learning iteration, across different strategies. Comparison across backbones and datasets.

Panel of Figure 3 — Figure 3: Number of labeled instances vs. rescaled BERTScore. Comparison of active strategies using BART and PEGASUS backbones on AESLC, XSum, and CNN/DM datasets.

Panel of Figure 4 — Figure 4: Number of labeled instances vs. ROUGE-1. Comparison of active learning strategies using BART and PEGASUS backbones on AESLC, XSum, and CNN/DM datasets. The dashed horizontal line in each subplot denotes the 90% performance threshold of the full-dataset baseline.

Panel of Figure 5 — Figure 5: Number of labeled instances vs. BERTScore. Comparison of LOBSTER and the LOBSTER (Random Anchors) ablation using BART and PEGASUS backbones on AESLC, XSum, and CNN/DM datasets.

Panel of Figure 6 — Figure 6: Impact of IDDS on Selection Diversity (PCA visualization of selected instances). Comparison of the proposed LOBSTER method with IDDS versus the ablation without IDDS. The IDDS module prevents semantic collapse by filtering redundant candidates, leading to higher ROUGE-1 scores.

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 290 papers

A Multi-Model Metric-based Selection Framework for Abstractive Text summarization
Jun 3, 2026Ahmed Alansary, Ali HamdiText Summarization Language Model Ensembles
Faculty of Computer Science · MSA University · Giza, Egypt
BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion
Mar 12, 2026Varun Iyer, Cornelia CarageaText Summarization Language Model Decoding
University of Illinois Chicago
Optimizing Abstractive Summarization With Fine-Tuned PEGASUS
Jun 24, 2026Sadiul Arefin Rafi, Naimur Rahman, Kazi Nazibul Islam +2Text Summarization
BRAC University

Jun 3, 2026J/K move · Enter open · S save

A Multi-Model Metric-based Selection Framework for Abstractive Text summarization

Ahmed Alansary, Ali Hamdi

Faculty of Computer Science · MSA University · Giza, Egypt

Text Summarization Language Model Ensembles

Open paper AskPDF arXiv

Automatic text summarization has become increasingly important due to the rapid growth of digital textual information. This paper presents a Multi-Model Summarization Framework designed to improve the robustness and quality of abstractive text summarization. Relying on a single model often leads to inconsistent summarization quality across articles with varying structures and topics. To address this limitation, the proposed framework integrates multiple fine-tuned transformer-based summarization models and introduces a metric-based selection mechanism. In this framework, each model independently generates a candidate summary for the same input article. The generated summaries are then evaluated using automatic evaluation metrics that capture both lexical similarity and semantic relevance. Based on these scores, the framework selects the highest-quality summary as the final output. The models are fine-tuned and evaluated on the widely used CNN/DailyMail news summarization dataset. Experimental results demonstrate that the proposed framework achieves the highest BERTScore among all compared methods with a score of 88.63%. It also outperforms several LLMs such as GPT3-D2, Falcon-7b, and Mpt-7b, highlighting its effectiveness and robustness. These findings highlight the effectiveness of leveraging multiple transformer-based models within a metric-based selection strategy to improve the quality and robustness of automatic text summarization systems.

Abstract

Active Learning

Explore similar work

A Multi-Model Metric-based Selection Framework for Abstractive Text summarization

BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion

Optimizing Abstractive Summarization With Fine-Tuned PEGASUS