Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams

Yukyung Lee; Yebin Lim; Woojun Jung; Wonjun Choi; Susik Yoon

cs.CLFeb 25, 2026

Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams

Authors: Yukyung Lee, Yebin Lim, Woojun Jung, Wonjun Choi, Susik Yoon

Organizations: Boston University, Boston, USA · Korea University, Seoul, Korea

Abstract

Evaluating language models in streaming environments is critical, yet underexplored. Existing benchmarks either focus on single complex events or provide curated inputs for each query, and do not evaluate models under the conflicts that arise when multiple concurrent events are mixed within the same document stream. We introduce StreamBench, a benchmark built from major news stories in 2016 and 2025, comprising 605 events and 15,354 documents across three tasks: Topic Clustering, Temporal Question Answering, and Summarization. To diagnose how models fail, we compare performance with and without structural cues, which organize key facts by event. We find that structural cues improve performance on clustering (up to +4.37%) and temporal QA (up to +9.63%), helping models locate relevant information and separate distinct events. While temporal reasoning remains an open challenge inherent to current LLMs, consistent gains across tasks show that structural cues are a promising direction for future work in massive document streams.

Figures & tables

Figure 1. Two challenges in streaming environments. Intra-topic conflict: within Dixie Fire, the most recent event (Event 3) has fewer documents than the outdated event (Event 1), making it harder to find the latest information. Inter-topic conflict: when asked about Dixie Fire (Topic 1), the LLM may confuse “8 firefighters” (Event 2) from Bootleg Fire (Topic 2).

Story	Contents	Duration	Topics	Events	Docs	Avg. Tok
A	California Fire	Jan’24–Nov’25	32	113	768	1,555
B	South Korea Martial Law	Jun’24–Nov’25	15	108	1,135	905
C	60th US Presidential Election	Jan’24–Nov’25	32	111	724	2,061
D	Summer Olympics	Apr’16–Dec’16	20	35	977	1,296
E	Israel-Palestine Conflict	Jan’16–Dec’16	13	45	387	732
F	58th US Presidential Election	Jan’16–Dec’16	88	193	11,363	1,916

Table 1. Dataset statistics. StreamBench comprises six news stories spanning two time periods: 2024–25 (Stories A–C) and 2016 (Stories D–F). Token counts are measured using the Llama-3 tokenizer.

Figure 2. Document stream volume over time. The x-axis shows the normalized story timeline, and the y-axis indicates the number of documents per 7-day window. We vary

k\in{1,3,5,10}

, the number of documents sampled per event. The 2016 stories (D–F) contain more documents per window than the 2025 stories (A–C).

Figure 3. Performance bottleneck analysis across tasks and model scales. Stacked bar charts show base performance and structural cue effects for Small (1–4B) and Large (70B+) models. Green bar indicates positive effect from structural cues; red indicates negative effect. Hatched bar represents headroom to the ceiling.

k

indicate number of documents sampled per event.

Panel of Figure 4 — Figure 4. $\Delta_{\text{org}}$ across model scales and document sizes per event ( $k$ ) for each task. * indicates statistical significance ( $p<0.05$ ).

Task	Scale	$k$ =1	$k$ =3	$k$ =5	$k$ =10
Topic Clustering	Small	.988	$<$ .001*	.003*	.042*
Topic Clustering	Large	$<$ .001*	$<$ .001*	.001*	.016*
Temporal QA	Small	$<$ .001*	$<$ .001*	$<$ .001*	$<$ .001*
Temporal QA	Large	$<$ .001*	$<$ .001*	$<$ .001*	$<$ .001*
Summarization	Small	.747	.456	.345	.623
Summarization	Large	.008*	.018*	.090	.265

Table 2. Statistical significance of structural cue effect (Wilcoxon signed-rank test). * indicates

p<0.05

Model	Metric	w/o cue	w/ cue	$\Delta_{\text{org}}$
Small	ROUGE-1	26.9	26.7	+ 0.2
	ROUGE-2	0 8.0	0 9.0	+ 1.0
	ROUGE-L	15.6	15.9	+ 0.3
	BLEU	0 3.6	0 4.2	+ 0.7
	METEOR	25.3	26.6	+ 1.4
Large	ROUGE-1	26.9	27.0	+ 0.1

Table 3. Summarization performance across multiple metrics. w/ cue indicates whether structural cues are provided.

Setting	Clustering	Temporal QA	Summarization
	(B³-F1)	(Accuracy)	(ROUGE-L / METEOR)
Gemma-2-2B
w/o Cue	71.5	80.3	15.9 / 26.3
w/ Cue	78.0	88.3	16.3 / 27.7
– Location	76.4	87.4	16.1 / 26.3
– Event Attrs	73.6	88.1	16.3 / 27.6

Table 4. Ablation on structural cue components. Each row under w/ cue removes one element to measure its contribution. Bold indicates the best score per setting.

Model	w/o cue	RAG	Serialized Facts	w/ cue
Gemma-2-2B	80.3	82.4	85.3	88.3
Qwen2.5-72B	89.7	90.2	92.4	95.3

Table 5. Effect of cue structure on temporal QA (Acc %). Bold indicates the best score.

	Small (1–4B)			Large (70B+)
Metric	w/o cue	w/ cue	$\Delta_{\text{org}}$	w/o cue	w/ cue	$\Delta_{\text{org}}$
ROUGE-L	15.6	15.9	+0.3	15.4	16.1	+0.7
METEOR	25.7	27.2	+1.5	29.5	31.8	+2.3
CheckEval	72.4	75.4	+3.0	84.8	89.7	+4.9

Table 6. Structural cue effect across evaluation metrics for summarization.

Figure 5. CheckEval results for summarization, with

\Delta_{\text{org}}

broken down by evaluation dimension.

Figure 6. Temporal Document distribution of StreamBench. Each row represents a story (A–F), with bubbles indicating active 7-day windows. Bubble size reflects the number of events per window.

Size	Model	Params	Cutoff
Small	Llama-3.2-1B-Instruct	1B	Dec 2023
	Llama-3.2-3B-Instruct	3B	Dec 2023
	Gemma-2-2B-it	2B	Jun 2024
	Gemma-3-4B-it	4B	Aug 2024
Medium	Llama-3.1-8B-Instruct	8B	Dec 2023
	Qwen2.5-7B-Instruct	7B	Oct 2023 ∗

Table 7. Model specifications and knowledge cutoff dates. ∗ denotes an approximate cutoff, as it is not officially documented by the provider.

Size	Model	$k$ =1				$k$ =3				$k$ =5				$k$ =10
		2016		2025		2016		2025		2016		2025		2016		2025
		wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue
Small	Llama-3.2-1B	88.37	86.19	88.98	88.93	63.65	66.85	70.88	71.25	63.45	68.38	72.78	72.79	66.43	71.28	76.05	75.60
	Gemma-2-2B	92.92	91.10	91.92	85.90	58.28	64.00	70.28	74.54	51.49	64.71	63.33	74.58	49.06	68.71	60.55	76.89
	Llama-3.2-3B	86.35	87.69	84.91	88.73	67.19	71.02	72.98	72.50	64.01	65.41	70.03	67.80	61.02	59.13	68.44	63.21
	Gemma-3-4B	77.37	78.10	77.40	79.54	77.99	78.63	79.08	80.33	78.87	79.11	80.25	81.40	80.19	80.53	81.91	83.25

Table 8. Model and year specific topic clustering performance (B 3 F1) across temporal cue conditions and documents per event (

k

). Bold indicates the best score in each column.

Size	Model	$k$ =1				$k$ =3				$k$ =5				$k$ =10
		2016		2025		2016		2025		2016		2025		2016		2025
		wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue
Small	Llama-3.2-1B	42.92	50.74	53.88	66.25	44.45	52.55	55.89	64.72	49.27	53.28	55.27	62.47	45.96	53.07	53.79	60.76
	Gemma-2-2B	76.72	87.05	78.45	90.52	78.12	87.86	79.78	86.65	78.55	88.04	81.99	87.89	74.23	87.08	80.85	87.95
	Llama-3.2-3B	80.61	90.42	81.32	92.41	86.82	90.54	87.50	93.53	88.01	92.30	87.02	93.13	86.58	90.31	86.36	92.81
	Gemma-3-4B	80.95	88.47	82.10	88.11	85.27	87.42	90.51	93.19	86.22	90.28	91.17	92.95	84.72	88.15	90.64	95.39

Table 9. Model and year specific temporal question answering accuracy across temporal cue conditions and documents per event (

k

). Bold indicates the best score in each column.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ROUGE-L
Size	Model	$k$ =1				$k$ =3				$k$ =5				$k$ =10
		2016		2025		2016		2025		2016		2025		2016		2025
		wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue	wo/ cue	w/ cue
Small	Llama-3.2-1B	16.69	17.42	13.51	12.30	17.30	18.68	13.21	12.52	17.59	18.43	13.09	12.52	17.94	18.09	13.22	12.33
	Gemma-2-2B	12.81	10.31	15.80	15.31	8.63	7.84	15.89	15.33	7.42	7.31	15.97	15.15	6.90	6.79	15.90	15.38
	Llama-3.2-3B	17.92	18.97	14.96	14.92	18.37	19.35	15.26	15.19	18.32	19.18	15.55	15.39	18.43	19.32	15.52	15.61

Appendix

Table 10. Model and year specific multi-document summarization performance (ROUGE-L and METEOR) across temporal cue conditions and documents per event (

k

). Bold indicates the best score in each column within each metric block.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

LLMStructBench: Benchmarking Large Language Model Structured Data Extraction
Feb 16, 2026Sönke Tenckhoff, Mario Koddenbrock, Erik RodnerLLM Information Extraction Document Parsing Benchmarks
KI-Werkstatt / FB2, University of Applied Sciences Berlin
Less Back-and-Forth: A Comparative Study of Structured Prompting
May 19, 2026Saurav Ghosh, Gabriella Polach, Abdou SowLLM Evaluation Structured Prompting
Washington University in St. Louis Saint Louis, Missouri, USA
Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs
Mar 31, 2026Zhuowen Liang, Xiaotian Lin, Zhengxuan Zhang +3Question Answering Document QA
The Hong Kong University of Science and Technology (Guangzhou) · EvenUp, USA

Feb 16, 2026J/K move · Enter open · S save

LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

Sönke Tenckhoff, Mario Koddenbrock, Erik Rodner

KI-Werkstatt / FB2, University of Applied Sciences Berlin

LLM Information Extraction Document Parsing Benchmarks Structured Output Generation LLM Evaluation LLM Prompting

Open paper AskPDF arXiv

We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies. We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability. In particular, we show that choosing the right prompting strategy is more important than standard attributes such as model size. This especially ensures structural validity for smaller or less reliable models but increase the number of semantic errors. Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.

Abstract

LLM Evaluation

Explore similar work

LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

Less Back-and-Forth: A Comparative Study of Structured Prompting

Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs