Can LLMs Access their World Knowledge for Event Prediction?

Zijun Wu; Zhitian Zhang; Hossein Hajimirsadeghi; Nikita Dvornik; Alexander Pashevich

cs.CLOct 13, 2026

Can LLMs Access their World Knowledge for Event Prediction?

Authors: Zijun Wu, Zhitian Zhang, Hossein Hajimirsadeghi, Nikita Dvornik, Alexander Pashevich

Organizations: RBC Borealis · University of Alberta

Abstract

Recent work has questioned whether large language models (LLMs) can be useful for temporal event prediction. Despite vast pretrained world knowledge, LLMs often underperform simpler specialized models. We ask whether this knowledge is truly inaccessible, or whether current methods fail to present event prediction in a form LLMs can use. We argue for the latter and show that LLMs become effective event forecasters when the interface is designed around two choices. The first grounds each event in a natural-language description, and the second separates event semantics from the prediction space to prevent semantic bias from label associations. To test this, we introduce DescribeEvent, a benchmark of ~150K events across seven datasets and six domains with event-level descriptions. We find that capable LLMs substantially benefit from descriptions and symbolic labels, outperforming description-free variants and non-LLM baselines, while weaker models gain little. These results suggest that accessing LLM world knowledge for event prediction is an interface problem, and that event descriptions offer a practical way to elicit the knowledge LLMs already encode.

Figures & tables

Figure 1: LLM Average performance across all datasets of DescribeEvent . Adding natural-language descriptions of events (as indicated by ‘+ Desc’) consistently improves LLM performance, and pairing descriptions with using symbolic labels instead of semantic event names as prediction targets yields the largest gains. Dashed lines indicate the best non-LLM baseline for each metric.

Figure 2: Accuracy under random label permutations. Semantic labels are permutation-sensitive, while symbolic labels are not.

Figure 3: DescribeEvent data curation pipeline.

Figure 4: Dataset distribution of DescribeEvent by total events. The benchmark spans seven datasets across six diverse domains.

Figure 5: Mean inter-event time across datasets. DescribeEvent covers temporal granularities from minutes to weeks.

	Recoverability			Similarity
Dataset	Random	$R$	$\Delta_{\text{gain}}$	$\cos_{\text{other}}$	$\cos_{\text{true}}$	$\Delta_{\text{leak}}$
GitHub Repo	0.125	0.905	0.780	0.114	0.187	0.073
GitHub User	0.125	0.897	0.772	0.122	0.228	0.106
Amazon	0.100	0.904	0.804	0.155	0.358	0.203
Earthquake	0.125	0.884	0.759	0.257	0.388	0.131
GDELT	0.143	0.611	0.468	0.112	0.227	0.115

Table 1: Description validation diagnostics across the seven datasets in DescribeEvent .

\Delta_{\text{gain}}

measures recovery above chance, where higher is better.

\Delta_{\text{leak}}

measures label leakage, where lower is better and indicates that descriptions do not paraphrase labels.

	GitHub Repo		GitHub User		Amazon		Earthquake		GDELT		NBA		Wikipedia
Model	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$
Semantic Event Type
Qwen3.5-4B	0.226	3.002	0.206	2.327	0.140	6.728	0.150	0.406	0.141	10.303	0.191	0.128	0.190	18.171
Qwen3.5-9B	0.290	2.832	0.245	1.830	0.148	6.622	0.167	0.370	0.133	10.409	0.262	0.092	0.304	17.634
Qwen3.5-27B	0.349	2.334	0.305	1.563	0.159	5.959	0.225	0.365	0.137	10.188	0.278	0.082	0.280	17.696
GPT-5 Mini	0.332	2.252	0.317	1.473	0.131	6.012	0.220	0.344	0.146	10.139	0.281	0.075	0.311	17.268

Table 2: LLM performance on DescribeEvent . Best results per dataset are in bold .

Figure 6: Effect of adding event descriptions on Avg F1 under Symbolic and Semantic label settings. Models with strong reasoning capability benefit from descriptions, while less capable models see degraded performance.

Figure 7: Precision gain from using Symbolic over Semantic labels plotted against event type frequency.

non-LLM Baselines
	GitHub Repo		GitHub User		Amazon		Earthquake		GDELT		NBA		Wikipedia
Model	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$
ARIMA	0.166	3.075	0.193	2.152	0.130	7.010	0.149	0.374	0.151	10.363	0.059	0.105	0.175	24.561
KNN	0.332	2.906	0.327	2.244	0.117	6.875	0.175	0.397	0.153	11.139	0.260	0.089	0.256	21.239
Hawkes	0.067	4.215	0.109	3.805	0.056	6.427	0.067	0.561	0.055	18.596	0.046	0.154	0.084	102.286
NHP	0.159	2.462	0.131	1.665	0.044	5.934	0.148	0.343	0.095	10.772	0.148	0.105	0.084	18.227

Table 3: Per-dataset performance of non-LLM baselines. Best result per dataset is in bold .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Source	Prediction Task
GitHub Repo	GH Archive	Repository event
GitHub User	GH Archive	User action
Amazon	Amazon Reviews’23	Product category
Earthquake	USGS API	Tectonic zone
GDELT	GDELT 2.0 Database	Diplomatic action
NBA	NBA Stats API	Play type

Appendix

Table 4: Dataset sources and prediction tasks in DescribeEvent .

Dataset Statistics	GitHub User	GitHub Repo	Wikipedia	NBA	Earthquake	GDELT	Amazon	Total
Number of Sequences	382	373	276	286	215	179	229	1,940
Number of Events	33,856	25,576	22,063	29,102	10,510	13,779	13,299	148,185
Number of Event Types	8	8	6	9	8	7	10	-
Time Unit	Hours	Hours	Hours	Minutes	Days	Days	Weeks	-
Avg Sequence Length	88.6	68.6	79.9	101.8	48.9	77.0	58.1	-
Avg Description Words	62.3	59.9	20.8	23.8	42.9	8.6	39.4	-

Appendix

Table 5: Dataset statistics for DescribeEvent .

Dataset	$\|\mathcal{Y}\|$	Semantic Label Space
Amazon	10	Beauty & Personal Care covers foot/hand/nail care, fragrance, hair care, makeup, men’s grooming, personal care, salon & spa, shave & hair removal, skin care, and tools & accessories. Books covers arts & photography, biographies & memoirs, children’s books, Christian books & bibles, crafts/hobbies/home, history, literature & fiction, mystery/thriller/suspense, politics & social sciences, and teen & young adult. Electronics covers accessories & supplies, camera & photo, car & vehicle electronics, computers & accessories, GPS finders & accessories, headphones/earbuds/accessories, home audio, portable audio & video, security & surveillance, and television & video. Home & Kitchen covers bath, bedding, event & party supplies, furniture, heating/cooling/air quality, home décor, kitchen & dining, seasonal décor, storage & organization, and wall art.
Earthquake	8	North America; Central and South America; Northwest Pacific; Southeast Asia and Oceania; Southwest Pacific; Mediterranean and Middle East; Central and South Asia; Atlantic and Africa.
GDELT	7	public statement; consultation; diplomatic engagement; material cooperation; disapproval; coercion; armed conflict.
NBA	9	made 2-point; made 3-point; missed field goal; free throw; turnover; foul; rebound; steal or block; timeout.
Wikipedia	6	revert; reference; content addition; content removal; copy edit; minor edit.
GitHub	8	issue opened; issue closed; PR opened; PR merged; push; release; PR reviewed; comment.

Appendix

Table 6: Label space type_event for each dataset.

|\mathcal{Y}|

denotes the number of semantic event labels.

Dataset	Dominant %	Repeat %	Min types	Length	Domain-specific filters
Amazon	30%	35%	3	50–80	Verified-purchase reviews only; user time span 3–60 months; other -typed events stripped.
Earthquake	50%	–	4	30–50	Magnitude at least 5.0.
GDELT	35%	35%	3	40–100	Root events only; at least 2 mentions, with at least 3 required for armed conflict; business, media, and educational actors excluded from armed conflict; same-country pairs excluded; span 30–1200 days; Goldstein swing of at least 12 on at most 15% of adjacent events.
NBA	40%	30%	4	90–110	All four regulation quarters present; overtime excluded.
Wikipedia	40%	40%	4	60–80	Bot accounts excluded; chunk duration at most 4380 hours which is about six months.
GitHub Repo	50%	–	5	60–80	Repositories with more than 20% bot events dropped; at least 150 characters per event on average; 2–5 distinct actors per repository.

Appendix

Table 7: Filtering thresholds for each dataset. Dominant % denotes the maximum share of events with the same type_event within a sequence, and Repeat % denotes the maximum share of adjacent event pairs with the same type. Min types specifies the minimum number of distinct event types per sequence, and Length gives the allowed sequence-length range.

Model	GPQA Diamond	HLE	Desc. helps?
Qwen3.5-27B	85.8%	22.2%	✓
Qwen3.5-9B	80.6%	13.3%	✓
Qwen3.5-4B	77.1%	7.8%	✓
GPT-5 Mini	80.3%	14.6%	✓
Grok-4.1	85.3%	17.6%	✓
Gemini-2.5	79.0%	11.1%	✓

Appendix

Table 8: Reasoning benchmark scores on GPQA and HLE, and whether adding descriptions improves event prediction.

Semantic Event Type
Model	F1 $\uparrow$	MAE $\downarrow$
Qwen3.5-4B	0.178	5.866
Qwen3.5-9B	0.221	5.684
Qwen3.5-27B	0.248	5.455
GPT-5 Mini	0.248	5.366
Grok-4.1	0.253	5.692

Appendix

Table 9: Dataset-averaged macro F1 and MAE for every row of Tables 2 and 3 , organized into Semantic, Symbolic, and non-LLM groups with their description-aware variants. The best Avg F1 and Avg MAE within each group are in bold .

	GitHub Repo		GitHub User		Amazon		Earthquake		GDELT		NBA		Wikipedia
Setting	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$	F1 $\uparrow$	MAE $\downarrow$
GPT-4o Mini
Symbolic	0.216	2.152	0.243	1.421	0.163	5.711	0.219	0.348	0.175	10.222	0.177	0.115	0.280	16.603
+ Description	0.199	2.276	0.183	1.518	0.187	5.780	0.202	0.391	0.162	10.139	0.088	0.107	0.255	16.701
Semantic	0.206	2.177	0.197	1.452	0.121	5.781	0.206	0.355	0.135	10.065	0.202	0.089	0.232	16.621
+ Description	0.218	2.325	0.171	1.563	0.131	5.849	0.199	0.371	0.155	10.179	0.163	0.104	0.249	16.639

Appendix

Table 10: Performance of smaller models across datasets. Adding descriptions does not consistently help, suggesting that sufficient model capability is needed to leverage descriptive information.

Figure 8: Per-type gain from Symbolic over Semantic labels, both with descriptions, versus event type frequency, for precision, recall, and F1.

Panel of Figure 9 — Figure 9: Prompt templates for the four evaluation settings, illustrated on a GitHub User event sequence.

Event Time Prediction

Event time prediction estimates when future events will occur from event histories, time series, and contextual information.

See all 44 papers

SocietyBench: Forecasting Counterfactual Social-World Evolution
Aug 4, 2026Zhenran Wang, Zhonghan Bian, Jinsong Li +1Counterfactual Evaluation Forecasting Benchmarks
LEAF: A Living Benchmark for Event-Augmented Forecasting
May 9, 2026Mingtian Tan, Mihir Parmar, Palash Goyal +5Financial Forecasting Time Series Forecasting
Google · University of Virginia
TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction
Sep 21, 2026Jie Gong, Maowei Jiang, Zhiwei Liu +6LLM Evaluation LLM-Based Time Series Forecasting
School of Artificial Intelligence, Wuhan University, China · Nanjing Audit University, China · The University of Manchester, United Kingdom +3

Aug 4, 2026J/K move · Enter open · S save

SocietyBench: Forecasting Counterfactual Social-World Evolution

Zhenran Wang, Zhonghan Bian, Jinsong Li, Zhangyang Qi

Counterfactual Evaluation Forecasting Benchmarks Probabilistic Forecasting Language Model Calibration

Open paper AskPDF arXiv

Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a browser, operate a GUI. A complementary social ability, namely how well a model understands and forecasts the way real social events unfold, has barely been measured. We introduce SocietyBench, an end-to-end benchmark that takes a one-line event topic, collects Web news and social-media posts across five platforms, distills them into a date-indexed timeline that keeps factual events and a public-opinion layer separate, and then turns every cutoff date on that timeline into an audited bank of forecasting questions. Questions are scored on two orthogonal 100-point axes: probability calibration and temporal accuracy. Before any model sees a timeline, a three-phase procedure replaces every named entity and shifts every date by a per-event constant, turning a real arc into a counterfactual social world -- structurally identical to what happened, but stripped of the surface labels a model could match against pre-training memory. On five heterogeneous events and 125 prediction points in Chinese and English editions, the strongest of six frontier LLMs reaches only 75.0 out of 100, against a trivial anchor of 50. The two axes come apart: a model can be calibration-strong but time-weak, or the reverse. Three agent frameworks built on a shared base model fail to improve on that base, and two model-free heuristics trail every LLM. Per-event gaps reach 21.4 points on a single axis, which is our main argument for evaluating on several events rather than one. All anonymized timelines, question banks, ground truth, and scoring code are released.

Abstract

Event Time Prediction

Explore similar work

SocietyBench: Forecasting Counterfactual Social-World Evolution

LEAF: A Living Benchmark for Event-Augmented Forecasting

TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction