Humans vs Vision-Language Models: A Unified Measure of Narrative Coherence

Nikolai Ilinykh; Hyewon Jang; Shalom Lappin; Asad Sayeed; Sharid Loáiciga

cs.CLMar 26, 2026

Humans vs Vision-Language Models: A Unified Measure of Narrative Coherence

Authors: Nikolai Ilinykh, Hyewon Jang, Shalom Lappin, Asad Sayeed, Sharid Loáiciga

Organizations: Dept. of Philosophy, Linguistics, and Theory of Science, University of Gothenburg · School of Electronic Engineering and Computer Science, Queen Mary University of London · Dept. of Informatics, King’s College London

Abstract

We study narrative coherence in visually grounded stories by comparing human-written narratives with those generated by vision-language models (VLMs) on the Visual Writing Prompts corpus. Using a set of metrics that capture different aspects of narrative coherence, including coreference, discourse relation types, topic continuity, character persistence, and multimodal character grounding, we compute a narrative coherence score. We find that VLMs show broadly similar coherence profiles that differ systematically from those of humans. In addition, differences for individual measures are often subtle, but they become clearer when considered jointly. Overall, our results indicate that, despite human-like surface fluency, model narratives exhibit systematic differences from those of humans in how they organise discourse across a visually grounded story. Our code is available at https://github.com/GU-CLASP/coherence-driven-humans.

Figures & tables

Prompt	System	Seqs	Seg/Seq	Sent/Seg	Sent/Seq	Words/Seq	Words/Sent	Words/Seg
Short	Human	$60$	$5.80$	$1.29$	$7.48$	$76.48$	$10.22$	$13.19$
Short	LLMs	$300$	$5.71$	$1.98$	$11.32$	$176.77$	$15.62$	$30.98$
Long	Human	$180$	$5.70$	$2.12$	$12.07$	$164.91$	$13.66$	$28.93$
Long	LLMs	$300$	$5.93$	$1.74$	$10.34$	$203.02$	$19.63$	$34.24$

Table 1: General descriptive statistics of the data. Seqs = number of story sequences, Seg = segments, Sent = sentences, and Words = word tokens.

Panel of Figure 1 — Figure 1: Example visual story sequence with numbered story and character images. Four short excerpts are shown from human and GPT-4o outputs under the short and long prompt conditions. Both humans and models have access to the full sequence of images at once. The [SEP] marker indicates segment boundaries. Each segment is a chunk of text about a single image, in the same left-to-right order as the numbered images. Segments are used as the unit of analysis in several of our metrics.

System	$C$	$S$	$R$	$t$
Claude 4.5	6.90	4.61	0.58 (0.18)	6.90 ∗∗
GPT-4o	5.88	4.73	0.65 (0.21)	4.60 ∗∗
InternVL3	7.05	5.26	0.63 (0.18)	4.60 ∗∗
Llama 4 Scout	4.87	5.43	0.73 (0.23)	1.12
Qwen3-VL	4.58	4.33	0.70 (0.20)	2.22 ∗
Humans	3.80	4.67	0.77 (0.19)	-

Table 2: Coreference metric, short prompt.

System	$U$	$T$	$D$	$t$
Claude 4.5	2.13	4.67	0.43 (0.13)	1.19
GPT-4o	2.02	4.60	0.41 (0.15)	1.95
InternVL3	1.83	4.95	0.35 (0.15)	4.33 ∗∗
Llama 4 Scout	1.92	4.65	0.39 (0.15)	2.63 ∗
Qwen3-VL	1.70	4.67	0.35 (0.15)	4.50 ∗∗
Humans	2.40	4.78	0.46 (0.15)	-

Table 3: Implicit discourse relation typology, short prompt.

Figure 2: Implicit discourse relation type composition, short prompt. Each bar shows the mean within-story proportion of predicted implicit relation types, averaged across stories (and displayed as 100% stacked bars).

System	$N$	$T$	$t$
Claude 4.5	5.67	0.27 (0.20)	5.54 ∗∗
GPT-4o	5.60	0.28 (0.18)	5.66 ∗∗
InternVL3	5.95	0.27 (0.18)	5.76 ∗∗
Llama 4 Scout	5.65	0.26 (0.17)	5.76 ∗∗
Qwen3-VL	5.67	0.38 (0.19)	1.65
Humans	5.80	0.43 (0.14)	-

Table 4: Topic switch, short prompt.

Figure 3: Topic switch under topic space compression. The highlighted region (

\mathbf{nr\_topics}=15

5

) marks a pronounced drop in human topic switch.

System	ChC	ChS	ChP	$t$
Claude 4.5	0.27	0.47	0.32 (0.29)	1.93
GPT-4o	0.35	0.51	0.36 (0.29)	1.02
InternVL3	0.43	0.60	0.38 (0.31)	0.35
Llama 4 Scout	0.07	0.10	0.08 (0.22)	7.18 ∗∗
Qwen3-VL	0.28	0.52	0.30 (0.26)	2.50 ∗
Humans	0.36	0.52	0.39 (0.29)	-

Table 5: Character metrics, short prompt.

System	GV	MCC	MCG	$t$
Claude 4.5	0.75	0.41	0.46 (0.28)	-0.20
GPT-4o	0.56	0.45	0.58 (0.34)	-2.13 ∗
InternVL3	0.60	0.41	0.53 (0.34)	-1.35
Llama 4 Scout	0.35	0.10	0.15 (0.32)	4.46 ∗∗
Qwen3-VL	0.72	0.45	0.50 (0.30)	-0.78
Humans	0.45	0.38	0.45 (0.49)	-

Table 6: Multimodal character grounding, short prompt.

System	$\mathrm{NCS}^{\text{arith}}$	$t_{\text{arith}}$	$\mathrm{NCS}^{\text{geom}}$	$t_{\text{geom}}$
Claude 4.5	0.41 (0.12)	4.68 ∗∗	0.27 (0.21)	2.84 ∗∗
GPT-4o	0.46 (0.12)	2.58 ∗	0.32 (0.21)	1.77
InternVL3	0.43 (0.13)	3.72 ∗∗	0.30 (0.20)	2.01 ∗
Llama 4 Scout	0.32 (0.10)	8.14 ∗∗	0.06 (0.15)	8.21 ∗∗
Qwen3-VL	0.45 (0.11)	2.77 ∗∗	0.32 (0.22)	1.59
Humans	0.50 (0.14)	-	0.36 (0.27)	-

Table 7: Narrative coherence scores, short prompt.

t_{\text{arith}}

and

t_{\text{geom}}

are human-vs-model paired tests.

System	Short	Long	$\Delta_{\text{short}}$	$\Delta_{\text{long}}$	$t$
Claude 4.5	0.42 (0.12)	0.41 (0.12)	0.083	0.068	0.86
GPT-4o	0.45 (0.11)	0.46 (0.11)	0.040	0.012	1.73
InternVL3	0.43 (0.12)	0.40 (0.13)	0.068	0.074	-0.31
Llama 4 Scout	0.32 (0.10)	0.38 (0.14)	0.182	0.095	2.77 ∗∗
Qwen3-VL	0.45 (0.11)	0.41 (0.11)	0.052	0.065	-0.63
Humans	0.50 (0.14)	0.48 (0.11)	–	–	–

Table 8: Narrative coherence scores on the same 54-story subset under short and long prompts. Table (a) reports results for

\mathrm{NCS}^{\text{arith}}

, and Table (b) reports results for

\mathrm{NCS}^{\text{geom}}

\Delta_{\text{short}}

and

\Delta_{\text{long}}

are the human-model gaps under short and long prompts, respectively, and

t

is the paired test statistic comparing the two gaps.

{}^{*}p<.05

{}^{**}p<.01

	Scraped data	Short prompt		Long prompt
Evaluator	Human	Human	Models	Human	Models
qwen3vl	14.21 (8.32)	13.58 (5.21)	3.12–4.31	11.54 (2.43)	2.67–4.11
llama4scout	37.00 (36.44)	25.98 (11.69)	1.83–18.04	20.85 (5.34)	1.67–14.42
internvl3	9.39 (3.19)	11.44 (4.48)	2.41–5.93	10.30 (2.06)	3.39–5.05

Table 9: Perplexity by evaluator. Columns are grouped by data condition. Human cells report mean perplexity for human-authored texts, with standard deviations in parentheses. Model cells report the range of mean perplexities across model-generated sources under the same evaluator. Darker shading indicates higher perplexity.

Visual Storytelling

See all 23 papers

When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs
May 27, 2026Marcell Fekete, Johannes Bjerva, Tamás KáldiVLM Evaluation Visual Question Answering
Department of Computer Science, Aalborg University, Copenhagen, Denmark · Department of Psycholinguistics and Neurolinguistics, ELTE Research Centre for Linguistics, Budapest, Hungary · ELTE Bárczi Gusztáv Faculty of Special Needs Education, Budapest, Hungary
Evaluating VQA in Vision Language Models using Cooperative Principles
Oct 2, 2026Monika Shah, Sudarshan Balaji, Somdeb Sarkhel +2VLM Evaluation Visual Question Answering
University of Memphis, TN, USA · Adobe Research, San Jose, CA, USA · University of Maryland Baltimore County, USA
Do Large Language Models Always Tell The Same Stories?
Jun 15, 2026Thennal DK, Hans Ole HatzelLLM Evaluation Diverse Text Generation
Language Technology Group University of Hamburg

May 27, 2026J/K move · Enter open · S save

When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs

Marcell Fekete, Johannes Bjerva, Tamás Káldi

Department of Computer Science, Aalborg University, Copenhagen, Denmark · Department of Psycholinguistics and Neurolinguistics, ELTE Research Centre for Linguistics, Budapest, Hungary · ELTE Bárczi Gusztáv Faculty of Special Needs Education, Budapest, Hungary

VLM Evaluation Visual Question Answering Vision-Language Models Language Modeling

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly evaluated for whether they identify the right visual content, but little is known about whether they express such content in a discourse-appropriate form. We address this research gap using information structure (IS), testing whether VLMs distinguish discourse-old Topics from discourse-new Foci in visually grounded question answering. We exploit Hungarian, a language in which Topic and Focus map onto dedicated syntactic positions, making IS choices observable in text. Comparing six VLMs with human participants, we find that models produce IS-relevant constructions, but over-regularise this sensitivity. Under the interacting pressures of discourse status, grammatical role (preference for subject Topics) and definiteness (preference for indefinite Foci), humans choose variable strategies for IS realisation. VLMs, by contrast, collapse onto narrow response templates, resembling mode collapse (Kirk et al., 2024). Our findings suggest that VLM evaluation should look beyond content accuracy to how content is packaged for the discourse.

Abstract

Visual Storytelling

Explore similar work

When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs

Evaluating VQA in Vision Language Models using Cooperative Principles

Do Large Language Models Always Tell The Same Stories?