Narrative Lookahead in LLMs: Event-Level Decodability, Predictive Advantage, and Steering

Donghoon Jung; Min Kim; Seojin Choi; Songeun Chae; Seohyon Jung

cs.CLOct 13, 2026

Narrative Lookahead in LLMs: Event-Level Decodability, Predictive Advantage, and Steering

Authors: Donghoon Jung, Min Kim, Seojin Choi, Songeun Chae, Seohyon Jung

Organizations: School of Digital Humanities and Computational Social Sciences, KAIST, South Korea

Abstract

Do language models carry information about narrative events before writing them, and how far ahead? We examine narrative lookahead through salient, state-changing events rather than tokens or response-level attributes. We extract 23,355 LLM-annotated events from 2,000 stories generated by four open-weight LLMs, labeling them along three nonexclusive dimensions of change: within agents, between agents, and in the story world, with human validation on a subset. Event segmentation and plot-independent labels enable comparisons across stories at horizons measured in events rather than in tokens. At event ends, we probe hidden states for upcoming event types and compare predictions with text-only predictors using matched prefixes. Across four models, future event types are decodable above prevalence baselines for at least four subsequent events, but probes significantly outperform both primary text-only predictors only for the next event. In exploratory steering with OLMo-3.1-32B, a probe-derived direction shifts the proportion of the target event type in both intended directions at higher strengths. Our event-level approach links the prediction of future narrative developments to interventions that reshape subsequent event composition.

Figures & tables

Figure 1: Overview of the research workflow. Generated stories are annotated with event spans and nonexclusive state-change labels. Hidden-state and matched text-only predictors target not-yet-started events across horizons. A next-event SOCIAL probe guides sustained steering of OLMo-3.1-32B-Instruct from the original prompt. New stories are re-annotated to evaluate event composition and generation-quality diagnostics.

Figure 2: Future-event decodability. Each model has a Macro-AP panel above smaller INTERNAL (I), SOCIAL (S), and WORLD-STATE (W) panels. Points show activation AP minus constant-prevalence AP, with pointwise 95% paired story-bootstrap intervals. Filled Macro markers indicate positive, within-model Holm-significant gains; attribute panels are exploratory. Scales are shared across models within each panel type. L identifies the validation-selected layer;

n

gives story counts at the first and last horizons. Test anchors are event ends; horizons count not-yet-started events.

Figure 3: Predictive advantage over text-only predictors, using the same model and attribute layout as Figure 2 . Points show activation AP minus the larger primary text AP for each metric; maxima are recomputed in every paired bootstrap replicate. Whiskers are pointwise 95% intervals. Filled Macro markers require positive, Holm-significant gains against both predictors in the shared 120-test family. Attribute panels are exploratory: their separate text maxima need not average to the Macro maximum. L and

n

follow Figure 2 .

Figure 4: OLMo sustained SOCIAL steering (

100

matched seeds per condition and strength). (a) Mean zero-extended per-story SOCIAL scores. (b,c) Paired differences against no steering and the fixed random direction. Whiskers are pointwise 95% bootstrap intervals. Filled directional markers in (b,c) indicate within-strength Holm-adjusted

p<.05

in the respective two-comparison family; random versus no steering is diagnostic. All outputs are retained, including zero-event and flagged outputs. Correction does not extend across strengths.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

	OLMo	Qwen	Gemma	Mistral
Events	4,541	6,741	6,630	5,443
Events/story (mean)	9.08	13.48	13.26	10.89
Events/story (SD)	1.96	2.72	2.21	2.71
Events/story (range)	4–17	6–22	7–21	5–20
INTERNAL (%)	52.6	51.3	59.7	53.7
SOCIAL (%)	37.5	19.2	26.4	23.4

Appendix

Table 1: Event annotation statistics across four models (500 stories per model). Domain labels are not mutually exclusive. Multi-domain denotes events assigned to two or more domains.

Candidate group	$N$	Majority (%)	Pooled (%)
LLM events	270	91.9	85.2
Easy controls	61	4.9	7.1
Hard controls	77	28.6	31.2

Appendix

Table 2: Human validation of main-event extraction across 408 candidates from 30 OLMo-generated stories. Majority denotes acceptance by at least two of three annotators; pooled denotes the proportion of positive judgments across all annotators. Human inter-annotator agreement was Fleiss’

\kappa=0.543

Domain	H–H	L–Ind.	L–Maj.	Pos. F1
	$\kappa$
INTERNAL	0.579	0.585	0.628	0.809
SOCIAL	0.552	0.689	0.764	0.839
WORLD-STATE	0.590	0.706	0.849	0.934

Appendix

Table 3: Agreement on narrative domain labels for 175 LLM-extracted events unanimously accepted as main events by three human annotators. H–H: human Fleiss’

\kappa

; L–Ind.: mean LLM–individual Cohen’s

\kappa

; L–Maj.: LLM–human-majority Cohen’s

\kappa

; Pos. F1: positive-class F1 against human-majority labels. Each domain is evaluated independently.

Model	Layers	Selected	Horizons
OLMo	64	22	1–11
Qwen	64	20	1–18
Gemma	60	36	1–17
Mistral	40	21	1–14

Appendix

Table 4: Validation-selected prediction layers (one-based) and eligible horizons. The same selected layers are used in the observable-text comparisons.

Model	$k$	$\Delta_{+}$ [95% CI]	$\Delta_{-}$ [95% CI]	R ( $+/-$ )	D ( $+/-$ )	Z ( $+/-$ )
Gemma	1	$-1.14$ [-7.49, 5.36]	$-1.52$ [-6.90, 3.70]	0/0	0/0	0/0
	2	$+2.83$ [-2.67, 8.14]	$-0.39$ [-6.64, 6.09]	0/0	0/0	0/0
	4	$+0.47$ [-4.60, 5.51]	$+1.60$ [-4.92, 7.90]	1/0	0/0	0/0
	8	$-11.68^{*}$ [-17.38, -6.01]	$-13.88^{*}$ [-21.74, -5.10]	23/29	22/29	0/7
	16	$-23.67^{*}$ [-27.89, -19.44]	$-23.67^{*}$ [-27.89, -19.44]	25/30	30/30	30/30
	32	$-26.46^{*}$ [-30.44, -22.10]	$-26.46^{*}$ [-30.44, -22.10]	30/29	30/30	30/30

Appendix

Table 5: Model-selection pilot (

n=30

per condition and strength).

\Delta_{+}

and

\Delta_{-}

are paired differences in mean per-story SOCIAL score versus no steering, in percentage points. CIs are pointwise paired-bootstrap intervals; ∗ denotes a two-sided sign-flip test with Holm-adjusted

p<.05

across the two directional comparisons within a strength, without correction across strengths. R, D, and Z count outputs with annotator-flagged disruptive repetition, narrative degeneration, and zero events, respectively, separately for positive/negative steering. All outputs are retained; zero-event outputs receive score zero. No-steering and random-direction controls had no degenerate or zero-event outputs.

Model	Activation	Full-prefix	Chunk-avg.	Recent	TF–IDF	Length	Prevalence
OLMo	.701	.572	.611	.622	.539	.541	.472
Qwen	.643	.575	.602	.609	.542	.520	.430
Gemma	.754	.673	.660	.653	.608	.590	.473
Mistral	.656	.575	.567	.590	.528	.513	.466

Appendix

Table 6: Next-event (

h=1

) story-balanced Macro-AP. Full-prefix and chunk-averaged embeddings are the two primary text predictors; Recent, TF–IDF, and Length are supporting predictors. Bold identifies activation scores for readability, not additional significance tests. All rows use 100 test stories.

Model	Attribute	Act. AP	Prev. AP	$\Delta_{\mathrm{prev}}$ [95% CI]	$\Delta_{\mathrm{text}}$ [95% CI]
OLMo	INTERNAL	.722	.516	.205 [.171, .241]	.063 [.028, .100]
	SOCIAL	.679	.370	.309 [.259, .361]	.121 [.059, .174]
	WORLD-STATE	.702	.529	.173 [.134, .215]	.086 [.041, .123]
Qwen	INTERNAL	.764	.536	.228 [.197, .259]	.036 [.010, .062]
	SOCIAL	.400	.186	.214 [.164, .272]	.036 [-.015, .073]
	WORLD-STATE	.765	.567	.198 [.167, .229]	.050 [.018, .079]

Appendix

Table 7: Next-event attribute-specific scores and gains over prevalence and the larger primary text AP for each attribute. The text maximum is recomputed in each bootstrap replicate. All intervals are pointwise; these attribute-specific comparisons are exploratory and do not use the primary Macro-AP significance criterion.

Figure 5: Absolute story-balanced Macro-AP for all predictors across future-event horizons at the fixed validation-selected prediction layers. All predictors use matched test samples. Curves are descriptive point estimates. Only training/validation-eligible horizons are shown; single-class targets follow the descriptive conventions in Appendix D.4 .

$h$	Stories/anchors	$\Delta_{\mathrm{prev}}$ [95% CI]	$p_{\mathrm{prev}}$	$\Delta_{\mathrm{text}}$ [95% CI]	$p_{\mathrm{full}}$	$p_{\mathrm{chunk}}$
1	100/825	.229 [.203, .256]	.0011	.090 [.061, .115]	.012	.012
2	100/725	.141 [.114, .174]	.0011	.027 [-.002, .053]	.952	1.000
3	100/625	.120 [.089, .158]	.0011	.027 [-.000, .052]	1.000	1.000
4	100/525	.094 [.064, .129]	.0011	.026 [-.013, .052]	1.000	1.000
5	98/425	.074 [.045, .112]	.0011	.006 [-.031, .041]	.864	1.000
6	93/327	.052 [.017, .107]	.199	.035 [-.022, .080]	1.000	1.000

Appendix

Table 8: OLMo horizon-level Macro-AP contrasts.

\Delta_{\mathrm{prev}}

is activation minus prevalence;

\Delta_{\mathrm{text}}

is activation minus the larger of the two primary text scores, with the maximum recomputed in every bootstrap replicate. All CIs are pointwise.

p_{\mathrm{prev}}

is Holm-adjusted within the model across eligible horizons;

p_{\mathrm{full}}

and

p_{\mathrm{chunk}}

use the shared 120-test primary text family. Text superiority requires both positive pairwise differences and both adjusted

p<.05

$h$	Stories/anchors	$\Delta_{\mathrm{prev}}$ [95% CI]	$p_{\mathrm{prev}}$	$\Delta_{\mathrm{text}}$ [95% CI]	$p_{\mathrm{full}}$	$p_{\mathrm{chunk}}$
1	100/1254	.213 [.189, .240]	.0018	.041 [.020, .060]	.012	.012
2	100/1154	.158 [.134, .187]	.0018	.016 [-.009, .035]	1.000	1.000
3	100/1054	.139 [.114, .171]	.0018	.008 [-.019, .032]	1.000	1.000
4	100/954	.102 [.077, .137]	.0018	-.017 [-.043, .007]	1.000	1.000
5	100/854	.114 [.083, .152]	.0018	-.002 [-.028, .021]	1.000	1.000
6	98/754	.075 [.051, .110]	.0018	-.018 [-.045, .009]	1.000	1.000

Appendix

Table 9: Qwen horizon-level Macro-AP contrasts.

\Delta_{\mathrm{prev}}

is activation minus prevalence;

\Delta_{\mathrm{text}}

is activation minus the larger of the two primary text scores, with the maximum recomputed in every bootstrap replicate. All CIs are pointwise.

p_{\mathrm{prev}}

is Holm-adjusted within the model across eligible horizons;

p_{\mathrm{full}}

and

p_{\mathrm{chunk}}

use the shared 120-test primary text family. Text superiority requires both positive pairwise differences and both adjusted

p<.05

$h$	Stories/anchors	$\Delta_{\mathrm{prev}}$ [95% CI]	$p_{\mathrm{prev}}$	$\Delta_{\mathrm{text}}$ [95% CI]	$p_{\mathrm{full}}$	$p_{\mathrm{chunk}}$
1	100/1237	.282 [.259, .305]	.0017	.081 [.055, .102]	.012	.012
2	100/1137	.215 [.191, .239]	.0017	.012 [-.006, .028]	1.000	.327
3	100/1037	.181 [.156, .209]	.0017	-.014 [-.035, .006]	1.000	1.000
4	100/937	.160 [.134, .190]	.0017	-.022 [-.044, -.001]	1.000	1.000
5	100/837	.155 [.129, .188]	.0017	-.020 [-.043, .005]	1.000	1.000
6	100/737	.104 [.075, .137]	.0017	-.043 [-.067, -.019]	1.000	.089

Appendix

Table 10: Gemma horizon-level Macro-AP contrasts.

\Delta_{\mathrm{prev}}

is activation minus prevalence;

\Delta_{\mathrm{text}}

is activation minus the larger of the two primary text scores, with the maximum recomputed in every bootstrap replicate. All CIs are pointwise.

p_{\mathrm{prev}}

is Holm-adjusted within the model across eligible horizons;

p_{\mathrm{full}}

and

p_{\mathrm{chunk}}

use the shared 120-test primary text family. Text superiority requires both positive pairwise differences and both adjusted

p<.05

$h$	Stories/anchors	$\Delta_{\mathrm{prev}}$ [95% CI]	$p_{\mathrm{prev}}$	$\Delta_{\mathrm{text}}$ [95% CI]	$p_{\mathrm{full}}$	$p_{\mathrm{chunk}}$
1	100/963	.190 [.168, .216]	.0014	.081 [.054, .105]	.012	.012
2	100/863	.112 [.091, .138]	.0014	.008 [-.020, .029]	1.000	1.000
3	100/763	.091 [.069, .119]	.0014	-.014 [-.050, .015]	1.000	1.000
4	100/663	.075 [.048, .107]	.0014	-.019 [-.048, .009]	1.000	1.000
5	98/563	.044 [.018, .078]	.053	-.057 [-.088, -.026]	.165	.045
6	98/465	.065 [.037, .104]	.0040	-.015 [-.048, .016]	1.000	1.000

Appendix

Table 11: Mistral horizon-level Macro-AP contrasts.

\Delta_{\mathrm{prev}}

is activation minus prevalence;

\Delta_{\mathrm{text}}

is activation minus the larger of the two primary text scores, with the maximum recomputed in every bootstrap replicate. All CIs are pointwise.

p_{\mathrm{prev}}

is Holm-adjusted within the model across eligible horizons;

p_{\mathrm{full}}

and

p_{\mathrm{chunk}}

use the shared 120-test primary text family. Text superiority requires both positive pairwise differences and both adjusted

p<.05

Figure 6: Layer selection by mean event-end validation Macro-AP across eligible horizons. Dots and vertical lines identify the selected layers. The prompt-end anchor is excluded from this criterion, and test results do not alter these choices.

Figure 7: Exploratory test Macro-AP gain over prevalence by layer and horizon. Dotted lines mark the validation-selected layers. Gray cells have no displayed estimate. No additional layer selection or significance testing uses this heatmap.

Figure 8: Time-resolved activation Macro-AP gain over prevalence, indexed by event-end anchor and future-event horizon. Dots identify positive differences with within-model secondary-family Holm-adjusted $p<.05$ . Gray cells have no displayed estimate. Missing significance markers do not establish absence of predictive information.

Figure 9: Eligible test-story counts for the time-resolved analyses. Later anchors and longer horizons have fewer contributing stories; these counts should accompany interpretation of the effect heatmaps.

Figure 10: Descriptive time-resolved activation Macro-AP minus the larger primary text Macro-AP. Gray cells have no displayed estimate. The recorded bootstrap resolution cannot reach the first Holm threshold of the 1,300-test secondary family (Appendix F.4 ); lack of significance is not evidence of no advantage.

$k$	$+\mathrm{SOCIAL}$ : $\Delta$ [95% CI]	$p_{+}$	$-\mathrm{SOCIAL}$ : $\Delta$ [95% CI]	$p_{-}$
1	+2.36 [-2.09, 7.00]	.645	+1.13 [-3.02, 5.18]	.645
2	+1.58 [-2.93, 6.06]	.498	-2.36 [-6.15, 1.37]	.465
4	+5.85 [.84, 10.95]	.055	-5.62 [-10.47, -.68]	.055
8	+3.60 [-1.28, 8.52]	.153	-13.69 [-18.53, -8.74]	$<.001$
16	+17.13 [12.45, 21.88]	$<.001$	-24.36 [-28.89, -19.84]	$<.001$
32	+26.07 [18.76, 33.41]	$<.001$	-32.70 [-37.18, -28.03]	$<.001$

Appendix

Table 12: OLMo SOCIAL contrasts against no steering (

n=100

matched seeds at each strength). Differences and pointwise bootstrap intervals are in percentage points. Two-sided sign-flip

p

-values are Holm-adjusted across the two directional comparisons within each strength; no correction is applied across strengths.

$k$	$+\mathrm{SOCIAL}$ : $\Delta$ [95% CI]	$p_{+}$	$-\mathrm{SOCIAL}$ : $\Delta$ [95% CI]	$p_{-}$
1	+0.81 [-3.81, 5.44]	1.000	-0.43 [-5.02, 4.24]	1.000
2	+3.53 [-.58, 7.62]	.205	-0.42 [-4.56, 3.73]	.845
4	+8.60 [3.77, 13.45]	.0016	-2.86 [-7.81, 2.14]	.266
8	+6.65 [1.91, 11.14]	.0063	-10.64 [-15.14, -6.05]	$<.001$
16	+23.70 [18.73, 28.56]	$<.001$	-17.78 [-21.85, -13.63]	$<.001$
32	+33.88 [27.43, 40.17]	$<.001$	-24.88 [-29.01, -20.68]	$<.001$

Appendix

Table 13: OLMo SOCIAL contrasts against the fixed orthogonal random direction (

n=100

matched seeds at each strength). Differences and pointwise bootstrap intervals are in percentage points. Two-sided sign-flip

p

-values are Holm-adjusted across the two directional comparisons within each strength; no correction is applied across strengths.

$k$	Condition	Tokens	Events	Rep-4 (%)
1	No steering	1218.1	9.06	0.92
	$+\mathrm{SOCIAL}$	1224.7	8.98	1.05
	$-\mathrm{SOCIAL}$	1197.5	8.91	0.94
	Random	1157.8	8.40	0.82
2	No steering	1205.8	9.11	0.91
	$+\mathrm{SOCIAL}$	1149.1	8.61	0.91

Appendix

Table 14: OLMo generation diagnostics (

100

outputs per row). D, Z, and R count annotator-flagged degeneration, zero-event outputs, and annotator-flagged disruptive repetition. Limit counts outputs reaching the 4,000-token limit. Tokens and Events are per-story means. Rep-4 is the mean repeated-token-4-gram fraction, in percent. Flags may overlap. No output ends by EOS before 32 content tokens. All outputs enter the primary score.

$k$	Condition	Score (%)	Common (%)	$N_{\mathrm{common}}$	S count	S/1k tokens	Events
16	No steering	39.46	39.16	96	3.61	3.03	9.14
	$+\mathrm{SOCIAL}$	56.59	56.71	96	4.61	4.10	8.19
	$-\mathrm{SOCIAL}$	15.10	15.73	96	1.44	1.21	9.11
	Random	32.89	33.02	96	2.92	2.47	9.00
32	No steering	41.03	40.51	92	3.67	3.12	9.12
	$+\mathrm{SOCIAL}$	67.10	72.25	92	4.83	4.17	6.64

Appendix

Table 15: Descriptive SOCIAL sensitivity summaries at the two strengths with bidirectional support against no steering. Score includes all 100 outputs with zero-event scores set to zero. Common is the mean ratio restricted to seeds whose outputs have events under all four conditions. S count, S/1k tokens, and Events are per-story means over all outputs. The common-eventful subset is post-intervention; it does not define an additional causal test.

Figure 11: Supplementary OLMo steering diagnostics ( $100$ matched seeds per condition and strength). (a) Mean SOCIAL events per 1,000 tokens. (b) INTERNAL (I) and WORLD-STATE (W) differences against no steering, rounded to percentage points. (c) Degeneration (D), zero-event (Z), and disruptive-repetition (R) counts for positive/negative steering. No steering has zero flags; random has one R flag at $k=32$ and no D/Z flags. Flags can overlap. All outputs are retained. These panels are descriptive summaries without new significance tests.

$k$	Outcome	$+\mathrm{SOCIAL}$ : $\Delta$ [95% CI]	$p_{+}$	$-\mathrm{SOCIAL}$ : $\Delta$ [95% CI]	$p_{-}$
1	INTERNAL	-2.34 [-6.66, 1.97]	.881	-0.46 [-4.93, 4.12]	.881
	WORLD-STATE	-2.71 [-7.16, 1.65]	.881	-2.64 [-6.82, 1.49]	.881
2	INTERNAL	-4.23 [-8.86, .22]	.277	+2.30 [-2.43, 6.88]	.431
	WORLD-STATE	-2.68 [-6.88, 1.49]	.431	-3.56 [-7.82, .88]	.336
4	INTERNAL	-2.26 [-6.87, 2.51]	1.000	-0.26 [-4.61, 4.11]	1.000
	WORLD-STATE	-2.31 [-6.54, 1.93]	1.000	+0.03 [-4.08, 4.08]	1.000

Appendix

Table 16: Exploratory INTERNAL and WORLD-STATE contrasts against no steering, in percentage points, retaining all 100 outputs per condition. CIs are pointwise. Two-sided sign-flip

p

-values use one four-comparison Holm family within each strength (two attributes by two directions), without correction across strengths.

Figure 12: Absolute mean per-story scores for OLMo under all four steering conditions (

n=100

per condition and strength). Whiskers are pointwise 95% seed-bootstrap intervals. Scores include zero-event outputs with value zero; multi-label attribute scores need not sum to one. Curves summarize the strength sweep descriptively and do not imply monotonic effects.

Next-Event Prediction

See all 16 papers

Can LLMs Access their World Knowledge for Event Prediction?
Oct 13, 2026Zijun Wu, Zhitian Zhang, Hossein Hajimirsadeghi +2Event Time Prediction LLM Evaluation
RBC Borealis · University of Alberta
Language Models Have Interpretable Narrative Mechanisms
Oct 9, 2026David Yiyang Liu, Aditya Joshi, Paul Dawson +1LLM Interpretability Circuits in Language Models
UNSW Sydney · McGill University
Characterizing Narrative Content in Web-scale LLM Pretraining Data
Jun 17, 2026Teagan Johnson, Elliott Ash, Andrew Piper +1Document Understanding Language Model Pretraining
University of Colorado Boulder · ETH Zürich · McGill University

Oct 13, 2026J/K move · Enter open · S save

Can LLMs Access their World Knowledge for Event Prediction?

Zijun Wu, Zhitian Zhang, Hossein Hajimirsadeghi, Nikita Dvornik+1

RBC Borealis · University of Alberta

Event Time Prediction LLM Evaluation LLM Grounding

Open paper AskPDF arXiv

Recent work has questioned whether large language models (LLMs) can be useful for temporal event prediction. Despite vast pretrained world knowledge, LLMs often underperform simpler specialized models. We ask whether this knowledge is truly inaccessible, or whether current methods fail to present event prediction in a form LLMs can use. We argue for the latter and show that LLMs become effective event forecasters when the interface is designed around two choices. The first grounds each event in a natural-language description, and the second separates event semantics from the prediction space to prevent semantic bias from label associations. To test this, we introduce DescribeEvent, a benchmark of ~150K events across seven datasets and six domains with event-level descriptions. We find that capable LLMs substantially benefit from descriptions and symbolic labels, outperforming description-free variants and non-LLM baselines, while weaker models gain little. These results suggest that accessing LLM world knowledge for event prediction is an interface problem, and that event descriptions offer a practical way to elicit the knowledge LLMs already encode.

Abstract

Next-Event Prediction

Explore similar work

Can LLMs Access their World Knowledge for Event Prediction?

Language Models Have Interpretable Narrative Mechanisms

Characterizing Narrative Content in Web-scale LLM Pretraining Data