LIDS: LLM Summary Inference Under the Layered Lens

Dylan Park; Yingying Fan; Jinchi Lv

cs.LGFeb 18, 2026

LIDS: LLM Summary Inference Under the Layered Lens

Authors: Dylan Park, Yingying Fan, Jinchi Lv

Organizations: Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Data Sciences and Operations Department, Marshall School of Business, University of Southern California, Los Angeles, CA 90089

Abstract

Large language models (LLMs) have gained significant attention by many researchers and practitioners in natural language processing (NLP) since the introduction of ChatGPT in 2022. One notable feature of ChatGPT is its ability to generate summaries based on prompts. Yet evaluating the quality of these summaries remains challenging due to the complexity of language. To this end, in this paper we suggest a new method of LLM summary inference with BERT-SVD-based direction metric and SOFARI (LIDS) that assesses the summary accuracy equipped with interpretable key words for layered themes. The LIDS uses a latent SVD-based direction metric to measure the similarity between the summaries and original text, leveraging the BERT embeddings and repeated prompts to quantify the statistical uncertainty. As a result, LIDS gives a natural embedding of each summary for large text reduction. We further exploit SOFARI to uncover important key words associated with each latent theme in the summary with controlled false discovery rate (FDR). Comprehensive empirical studies demonstrate the practical utility and robustness of LIDS through human verification and comparisons to other similarity metrics, including a comparison of different LLMs.

Figures & tables

Figure 1: Rescaled boxplots of the LIDS, BLEU, ROUGE-1, ROUGE-L, METEOR, and BERTScore similarity measures for GPT-5 (dark blue) over

50

repeated prompts and two benchmark summary mechanisms (light blue and gray) over

50

random repetitions on the Utah article.

Figure 2: Boxplots of the LIDS similarity measures for GPT-5 over

50

repeated prompts and two benchmark summary mechanisms over

50

random repetitions on the Utah article.

Figure 3: Scatter plot of the average human evaluated summary quality scores on the horizontal axis and the LIDS similarity measure scores of the summaries on the vertical axis, with a linear regression line showing a Pearson correlation of

0.904

between the two, for the experiment in Section 4.2 .

Similarity Metric	Seconds
LIDS	25.519812
BLEU	0.629676
ROUGE-1	8.013586
ROUGE-L	32.912094
METEOR	8.805413
BERTScore	158.460620

Table 1: Computational costs of different similarity metrics over

50

summaries on the Utah article

Similarity Metric	Memory Usage (MB)
	Incremental	Peak Usage
LIDS	19.65	30.04
BLEU	1.87	2.08
ROUGE-1	0.05	0.20
ROUGE-L	0.03	2.78
METEOR	86.37	113.99

Table 2: Memory usages of different similarity metrics over

50

summaries on the Utah article

Similarity Metric	Pearson Cor		Distance Cor		Kendall’s Tau
LIDS	$0.904$	$[0.81,0.95]$	$0.873$	$[0.77,0.95]$	$0.664$	$[0.51,0.79]$
BLEU	$0.232$	$[-0.14,0.55]$	$0.135$	$[0.00,0.37]$	$0.370$	$[0.08,0.60]$
ROUGE-1	$0.685$	$[0.43,0.84]$	$0.567$	$[0.35,0.75]$	$0.568$	$[0.40,0.70]$
ROUGE-L	$0.733$	$[0.51,0.86]$	$0.604$	$[0.38,0.77]$	$0.575$	$[0.37,0.72]$
METEOR	$0.662$	$[0.40,0.83]$	$0.575$	$[0.35,0.76]$	$0.598$	$[0.43,0.74]$
BERTScore	$0.976$	$[0.95,0.99]$	$0.955$	$[0.93,0.98]$	$0.880$	$[0.79,0.95]$

Table 3: The linear (Pearson) correlations, distance correlations, Kendall’s tau correlations, and the associated 95% confidence intervals (CIs) between different similarity metrics and human evaluations for the experiment in Section 4.2

Figure 4: LIDS visualization word cloud plots with FDR control at level $q=0.005$ for the first three latent SVD layers of a representative LLM summary of the Utah article.

Figure 5: Comparison of different LLMs with LIDS in terms of the Sharpe ratio-type measure of accuracy per unit of uncertainty, i.e., the mean similarity divided by the corresponding standard deviation over

50

repeated prompts on the Utah article. Larger values indicate better performance.

Figure 6: LIDS visualization word cloud plots with FDR control at level $q=0.005$ for the first three latent SVD layers of a representative LLM summary of the novel chapter (i.e., Chapter $3$ of “Pride and Prejudice”).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Boxplots of the LIDS similarity measures for different LLMs over

50

repeated prompts on the Utah article.

Summary Method	Mean	SD	Sharpe Ratio
GPT-5 Sum	0.961594	0.003868	248.616394
Naive Sum	0.675266	0.006546	103.157628
Rand Topic Sum	0.775645	0.033374	23.240836

Appendix

Table 4: Means, standard deviations (SDs), and Sharpe ratios of the LIDS similarity measures for GPT-5 over

50

repeated prompts and two benchmark summary mechanisms over

50

random repetitions on the Utah article

LLM	Mean	SD	Sharpe Ratio
GPT-5	0.961594	0.003868	248.616394
LLAMA 4	0.949213	0.006797	139.658456
Gemini 2.5 Flash	0.960815	0.006013	159.789622
Claude 4 Sonnet	0.963233	0.005397	178.475634
Grok 3	0.958152	0.002849	262.873284
DeepSeek-V3.1	0.952481	0.004745	200.735187

Appendix

Table 5: Means, standard deviations (SDs), and Sharpe ratios of the LIDS similarity measures for different LLMs over

50

repeated prompts on the Utah article

Figure 8: Rescaled boxplots of the LIDS, BLEU, ROUGE-1, ROUGE-L, METEOR, and BERTScore similarity measures for GPT-5 (dark blue) over

50

repeated prompts and two benchmark summary mechanisms (light blue and gray) over

50

random repetitions on the NASA article.

Figure 9: Boxplots of the LIDS similarity measures for GPT-5 over

50

repeated prompts and two benchmark summary mechanisms over

50

random repetitions on the NASA article.

Figure 10: Comparison of different LLMs with LIDS in terms of the Sharpe ratio-type measure of accuracy per unit of uncertainty, i.e., the mean similarity divided by the corresponding standard deviation over

50

repeated prompts on the NASA article. Larger values indicate better performance.

Figure 11: Boxplots of the LIDS similarity measures for different LLMs over

50

repeated prompts on the NASA article.

Figure 12: LIDS visualization word cloud plots with FDR control at level $q=0.005$ for the first three latent SVD layers of a representative LLM summary of the NASA article.

Figure 13: Boxplots of the LIDS similarity measures for GPT-5 over

50

repeated prompts and two benchmark summary mechanisms over

50

random repetitions on the legal document (i.e., a school district’s alcohol and drug free policy).

Figure 14: LIDS visualization word cloud plots with FDR control at level $q=0.005$ for the first three latent SVD layers of a representative LLM summary of the legal document (i.e., a school district’s alcohol and drug free policy).

Figure 15: Boxplots of the LIDS similarity measures for GPT-5 over

50

repeated prompts and two benchmark summary mechanisms over

50

random repetitions on the novel chapter (i.e., Chapter

3

of “Pride and Prejudice”).

Summarization Evaluation

Summarization evaluation measures whether summaries preserve important information, remain faithful to their sources, and serve readers’ needs.

See all 43 papers

LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation
Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2LLM Evaluation LLM Self-Refinement
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA
Summarization is Not Dead Yet
Jun 6, 2026Dongqi Liu, Chenxi Whitehouse, Zheng Zhao +3LLM Evaluation Text Summarization
Saarland University, Max Planck Institute for Informatics · Tencent YouTu Research · University of Cambridge +2
Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation
Sep 18, 2026Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger +2LLM Evaluation Text Summarization
ServiceNow Canada

Apr 28, 2026J/K move · Enter open · S save

LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation

Huyen Nguyen, Haoxuan Zhang, Yang Zhang, Haihua Chen+1

Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA

LLM Evaluation LLM Self-Refinement Text Summarization Summarization Evaluation Automated Evaluation

Open paper AskPDF arXiv

Reliable evaluation of large language model (LLM)-generated summaries remains an open challenge, particularly across heterogeneous domains and document lengths. We conduct a comprehensive meta-evaluation of 14 automatic summarization metrics and LLM-based evaluators across seven datasets spanning five domains, covering documents from short news articles to long scientific, governmental, and legal texts (2K-27K words) with over 1,500 human-annotated summaries. Our results show that traditional lexical overlap metrics (e.g., ROUGE, BLEU) exhibit weak or negative correlation with human judgments, while task-specific neural metrics and LLM-based evaluators achieve substantially higher alignment, especially for linguistic quality assessment. Leveraging these findings, we propose LLM-ReSum, a self-reflective summarization framework that integrates LLM-based evaluation and generation in a closed feedback loop without model finetuning. Across three domains, LLM-ReSum improves low-quality summaries by up to 33% in factual accuracy and 39% in coverage, with human evaluators preferring refined summaries in 89% of cases. We additionally introduce PatentSumEval, a new human-annotated benchmark for legal document summarization comprising 180 expert-evaluated summaries. All code and datasets will be released in GitHub.

Abstract

Summarization Evaluation

Explore similar work

LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation

Summarization is Not Dead Yet

Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation