CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles

Mir Tafseer Nayeem; Susmoy Chakraborty; Davood Rafiei

cs.CLSep 28, 2026

CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles

Authors: Mir Tafseer Nayeem, Susmoy Chakraborty, Davood Rafiei

Organizations: University of Alberta · Independent Researcher

Abstract

Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering, and cybersecurity, yet film remains comparatively underexplored despite requiring long-form narrative integration, multilingual interpretation, and culturally situated audience judgments. We introduce CineSubBench, a benchmark for evaluating long-context film understanding from multilingual movie subtitles. A subtitle track represents a film as thousands of short, temporally ordered utterances from which models must reconstruct characters, relationships, events, causal progression, and themes without explicit scene or event structure. CineSubBench contains 1,012 films with complete subtitle coverage in six languages, yielding 6,072 tracks and 8.13M timestamped subtitle entries. It provides a matched multi-task, multilingual, and multicultural (MultiX) evaluation setting: seven tasks span narrative reconstruction and abstraction, genre prediction, age suitability, country-specific motion-picture ratings across ten national classification systems, and subtitle-grounded language safety. Across nine LLMs, plot premises are recovered more reliably than event-complete synopses; cross-lingual consistency varies substantially across models and languages; national rating systems expose distinct calibration patterns; and strong profanity is far easier to ground than mild obscenity. CineSubBench establishes film as a long-context LLM evaluation domain and provides a unified benchmark for measuring narrative, multilingual, cultural, and evidence-grounding capabilities.

Figures & tables

Figure 1: Overview of Cine Sub Bench , a long-form MultiX benchmark for film understanding from subtitles. Each of 1,012 films has subtitle tracks in English, Arabic, Indonesian, Persian, Romanian, and Vietnamese, yielding 6,072 tracks and 8.13M timestamped subtitle entries. Seven tasks span Narrative Understanding and Generation and Cultural Prediction and Assessment, with ratings across ten national classification systems.

Figure 2: Construction and quality-assurance pipeline for Cine Sub Bench . Cross-source entity resolution and filtering reduce the initial 6,186-film inventory to a metadata-complete cohort. Complete-coverage optimization then selects six subtitle languages and ten national rating systems. Subtitle candidates undergo structural, temporal, encoding, language-consistency, and runtime checks with manual inspection and cross-provider recovery where needed. Task-oriented curation yields the final matched MultiX benchmark of 1,012 films.

Rank	Model	Family	Composite	Narrative Understanding and Generation					Cultural Prediction and Assessment					LC Assessment
			Overall score (0–100) $\uparrow$	Plot $\uparrow$	Synopsis $\uparrow$	Key message $\uparrow$	Overall $\uparrow$	Genre micro-F1 (%) $\uparrow$	Age exact (%) $\uparrow$	Age $\pm$ 1 (%) $\uparrow$	Age MAE $\downarrow$	Country EM (%) $\uparrow$	Country MAE $\downarrow$	LC agnostic F1 (%) $\uparrow$	LC strict F1 (%) $\uparrow$
	Gemini 3.8 Flash	Closed frontier	78.04	4.16	3.34	3.11	3.54	84.93	34.25	81.42	0.93	77.87	0.31	86.8	77.8
	GPT-5.6 Sol	Closed frontier	74.7	4.1	3.4	3.08	3.52	81.55	31.08	78.75	0.98	65.42	0.44	81.7	74.5
	DeepSeek V4 Pro 1.6T	Open-weight	65.52	3.91	2.93	2.9	3.25	81.63	30.67	79.33	0.98	58.03	0.52	57.5	48.3
4	Gemini 3.5 Flash Lite	Closed baseline	64.7	3.83	2.91	2.99	3.25	80.92	31.83	72.17	1.09	64.65	0.46	65.3	42.8
5	Claude Haiku 4.5	Closed frontier	57.5	3.45	2.44	2.84	2.91	71.45	27.42	72.75	1.16	54.32	0.57	58.1	37.2

Table 1: All-language comparison across the nine models with complete benchmark coverage. Narrative, genre, age, and country metrics are averaged equally over English, Arabic, Indonesian, Persian, Romanian, and Vietnamese subtitle inputs; language-content (LC) evidence metrics are English-only. Models are ordered by the composite score in Equation 1 . LC agnostic F1 matches safety-relevant subtitle evidence regardless of category, while strict F1 also requires the correct lexical category. Best values are bold and second-best distinct values are underlined; background shading is normalized within each metric column.

Figure 3: Narrative performance across subtitle languages. Plot remains more reliably recovered than synopsis, while Persian is the lowest observed narrative setting across the evaluated models.

Figure 5: Evidence-grounded language-safety performance. (A) Category-agnostic F1 measures recovery of relevant subtitle evidence; strict F1 also requires the correct lexical category. (B) Mean precision and recall by category across models. Strong profanity is substantially easier to ground than mild obscenity.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Primary role	Quality and provenance treatment
Kids-in-Mind	Initial 6,186-film inventory; review-derived language metadata.	The native identifier supports cross-source traceability; records without a usable IMDb link for subtitle retrieval are excluded.
Common Sense Media	General age-suitability label, storyline, and review page.	Cross-linked through IMDb ID; anomalous embedded IMDb links receive case-by-case manual adjudication before retention.
IMDb	Stable title identifier; plot synopsis, runtime, film metadata, and regional motion-picture ratings.	Provides the entity key used to link sources and subtitle assets; populated plot-synopsis and rating fields are mandatory for the metadata-complete cohort.
Subtitle providers	Timestamped SRT files in six languages, with provider metadata.	OpenSubtitles is the primary source and SubDL supports recovery. The highest-download candidate is selected per film and language when alternatives exist; its count is retained in the JSON. Structural and temporal verification remains independent of this popularity proxy.

Appendix

Table 2: Source roles in CineSubBench construction. Each source contributes a distinct component of the benchmark, while the final record preserves provenance links and exposes only subtitle text to models during evaluation.

Step	Decision	Retention rule	Films
1	Source crawl	Kids-in-Mind films collected	6,186
2	IMDb link screen	4,307 films with links; exclude 33 invalid/search-unusable entries	4,274
3	Entity linking	IMDb-ID agreement across Kids-in-Mind and Common Sense Media; anomalous links manually reviewed	2,831
4	Metadata completeness	Populated IMDb plot synopsis and motion-picture-rating fields	2,322
5	Language coverage	Subtitles in all six selected languages	1,231
6	Country coverage	Ratings from all ten selected countries in the final benchmark	1,012

Appendix

Table 3: Progressive construction gates in CineSubBench. Counts report the number of films retained after each filtering or coverage decision. The detailed subtitle-verification audit is applied to the resulting 1,231-film multilingual cohort.

Country	Ordered rating labels
Australia	G, PG, M, MA15+, R18+
Brazil	Livre, 10, 12, 14, 16, 18
France	Tous publics, 12, 16, 18
Germany	0, 6, 12, 16, 18
Netherlands	AL, 6, 9, 12, 14, 16, 18
Singapore	G, PG, PG13, NC16, M18, R21

Appendix

Table 4: Country-specific ordered motion-picture rating label spaces used in CineSubBench. Labels are normalized within each national classification system and retained as distinct ordinal scales rather than treated as interchangeable across countries.

Figure 6: Availability before complete-coverage optimization. Algorithm A.4 is applied to (a) the top 29 subtitle languages over the 2,322 metadata-complete films and (b) the top 21 national rating systems over the 1,231-film multilingual cohort.

Verification layer	Automated rule	Audit purpose and response
Provider download count	Among alternative SRT files for the same film and language, select the file with the highest source-reported count; retain the selected count in the JSON.	Use popularity only as an initial quality proxy. Subject the selected file to the remaining checks and recover it if verification finds a genuine problem.
Provider-body integrity	Scan subtitle bodies for server-error signatures, including Cloudflare and Guru Meditation messages.	Detect error pages returned in place of dialogue; retrieve or inspect a replacement asset.
Timestamp syntax	Require valid SRT time ranges with hours, minutes, seconds, millisecond separator, and arrow structure (e.g., 00:09:09,949 --> 00:09:12,179 ).	Catch malformed headers before JSON creation; repair or re-download the source asset.
Chronological consistency	Compare the start and end time of every subtitle block.	Flag negative-duration temporal inversions.
Single-block lifespan	Flag any subtitle block lasting 15 minutes or more.	Detect frozen or malformed subtitle segments.
Delayed onset	Flag a first operational subtitle beginning after five minutes.	Detect missing opening dialogue or late-start tracks.

Appendix

Table 5: Subtitle-level verification layers applied to the optimized six-language cohort. Each automated trigger initiates targeted manual inspection, source recovery, or an ingestion stop, rather than silently removing the affected subtitle track.

Tier	Encoding	Purpose and representative case
1	utf-8-sig	Default modern-web decoding while removing a byte-order mark when present.
2	utf-16	Handles UTF-16 subtitle assets, including files exported by automated transcription or speech-to-text systems.
3	cp1256	Handles legacy Windows Arabic and Persian encodings, preserving right-to-left dialogue instead of dropping unsupported characters.
4	cp1252	Handles legacy Western-European encodings and extended diacritics, such as é , ü , and ~n .
Failure	none succeeds	Reject the asset and require manual normalization or a clean replacement before ingestion resumes.

Appendix

Table 6: Deterministic subtitle-decoding cascade. Decoding proceeds to the next encoding tier only when the preceding one fails to read the source asset; lossy error suppression is never used.

Source website	Public dataset fields
Common Sense Media	ageSuitabilityRating ; storyline .
Kids-in-Mind	keyMessage ; languageContentReference .
IMDb	title ; releaseYear ; duration ; imdbRating ; countriesOfOrigin ; originalLanguages ; genres ; topics ; plot ; synopsis ; motionPictureRatings ; directors ; writers ; topCast ; fullCast .
OpenSubtitles and SubDL	subtitles , including each track’s language , downloadCount , and timestamped entries ( index , timeframe , and content ).
Cine Sub Bench annotation pipeline	languageContentAssessment , containing manually audited, subtitle-localized category counts and evidence indices derived from the English subtitle track.

Appendix

Table 7: Mapping from public CineSubBench fields to their source or construction process. Website names link to the corresponding source. The subtitle-derived languageContentAssessment follows Kids-in-Mind definitions and uses its reference counts for auditing, while the final labels and evidence are constructed directly from the English subtitle track.

Statistic	Value
Films	1,012
Subtitle tracks	6,072
Subtitle entries	8.13M
Release years	1977–2026
Runtime, mean / median	114.6 / 113.0 min
Runtime, 95th pct. / max	149.4 / 242.0 min

Appendix

Table 8: Core Cine Sub Bench dataset statistics.

Language	Mean tok.	Median tok.	95th pct.	Max tok.	Median entries	Median ratio
English (en)	43,123	41,759	68,833	144,268	1,418	1.00
Arabic (ar)	39,867	38,536	65,023	98,701	1,214	0.95
Indonesian (id)	40,641	39,739	64,760	120,034	1,308	0.97
Persian (fa)	45,162	43,916	73,731	112,920	1,325	1.08
Romanian (ro)	41,010	39,676	65,542	129,799	1,244	0.99
Vietnamese (vi)	43,613	42,374	70,575	108,761	1,322	1.05

Appendix

Table 9: Subtitle input size by language. Token counts are computed from prompt-formatted subtitle inputs using the tokenizer adopted for prompt-length accounting; they reflect tokenization-dependent input length rather than language-neutral semantic length.

Figure 7: Task-relevant statistics for all 1,012 Cine Sub Bench films. Panels show (A) subtitle input length by language, (B) age-suitability label distributions, (C) normalized genre frequencies, (D) subtitle-grounded language-safety counts and corresponding evidence-line counts, and (E) national motion-picture rating distributions represented by minimum-age values. Token counts are computed from prompt-formatted subtitle inputs using the o200k_base tokenizer.

Reference field	Mean	Median	P95	Max
Plot	33	33	50	66
Synopsis	1,448	1,262	2,802	7,151
Storyline	156	154	242	437
Key message	14	12	26	55

Appendix

Table 10: Reference output length statistics for narrative tasks.

Reference field	Mean	Median	P95	Max
Plot	33	33	50	66
Synopsis	1,448	1,262	2,802	7,151
Storyline	156	154	242	437
Key message	14	12	26	55

Appendix

Table 10: Reference output length statistics for narrative tasks.

Category	Gold count	Evidence lines
Strong profanity	21,892	20,602
Crude bodily language	20,131	19,149
Religious profanity/exclamation	10,861	10,447
Mild obscenity	10,360	10,088

Appendix

Table 11: Subtitle-grounded language-content statistics. Evidence is defined over the English subtitle track.

Language	Flagged	Span/run.	Span p05	Tok. ratio	Entry ratio	Low tok.	Low entry
English (en)	9	0.95	0.89	1.00	1.00	0	0
Arabic (ar)	142	0.93	0.87	0.95	0.88	27	120
Indonesian (id)	112	0.95	0.89	0.97	0.93	28	84
Persian (fa)	123	0.96	0.88	1.08	0.95	23	77
Romanian (ro)	155	0.95	0.89	0.98	0.89	27	125
Vietnamese (vi)	93	0.96	0.90	1.04	0.95	20	57

Appendix

Table 12: Subtitle-duration coverage audit by language. Span/runtime measures the fraction of the film timeline covered by each subtitle track, while token and entry ratios compare each track with the English subtitle track for the same film. Low-token and low-entry counts use thresholds of

<0.60

and

<0.70

relative to English, respectively.

Genre	Films
Drama	531
Comedy	380
Thriller	350
Adventure	340
Action	336
Fantasy	215

Appendix

Table 13: Most frequent normalized core genres.

Genre	Films
Drama	531
Comedy	380
Thriller	350
Adventure	340
Action	336
Fantasy	215

Appendix

Table 13: Most frequent normalized core genres.

Age label	Films
16+	157
17+	153
14+	130
13+	129
15+	109
12+	66

Appendix

Table 14: Common Sense age-suitability labels.

Figure 8: Additional corpus context. (a) Release-year and runtime coverage. (b) Topic frequency after splitting comma-separated tags and removing generic core-genre labels; font size indicates frequency.

Evaluation role	Models	Model access	Subtitle inputs	Tasks
Closed frontier	GPT-5.6 Sol ; Gemini 3.8 Flash ; Claude Haiku 4.5	OpenAI, Gemini, and Anthropic APIs	English + Arabic, Indonesian, Persian, Romanian, Vietnamese	Narrative and cultural; English language safety where available
Closed baseline	GPT-5 Nano ; Gemini 3.5 Flash Lite	OpenAI and Gemini APIs	English + Arabic, Indonesian, Persian, Romanian, Vietnamese	Narrative and cultural; English language safety where available
Open-weight	DeepSeek V4 Pro 1.6T ; Qwen3 235B ; Mistral 4 119B ; Llama 4 Scout 17B	OpenRouter API	English + Arabic, Indonesian, Persian, Romanian, Vietnamese	Narrative and cultural; language safety for completed English lanes
Local scaling	Ministral 3B , 8B , and 14B	Local GPU inference	English	Controlled English scaling analysis
Narrative evaluator	GPT-5.6 Luna	OpenAI API	Narrative outputs from all compared models	Reference-based narrative scoring and structured error annotations

Appendix

Table 15: Models and task coverage in Cine Sub Bench . All models receive subtitle text as their only film evidence and generate English outputs for narrative generation tasks. Six-language comparisons use the same matched film set, while evidence-grounded language-safety assessment is evaluated only on the English subtitle track.

Task	Paired scores	Pearson $r$	Spearman $\rho$	Weighted $\kappa$	Exact (%)	Within 1 (%)	Mean $\Delta$
All three tasks	1350	0.808	0.815	0.778	56.8	98.4	$-0.192$
Plot	450	0.761	0.710	0.675	43.4	96.9	$-0.438$
Synopsis	450	0.750	0.751	0.729	61.5	98.2	$-0.084$
Key message	450	0.766	0.776	0.763	65.5	100.0	$-0.053$

Appendix

Table 17: Agreement between Claude Sonnet 5 and GPT-5.6 Luna on narrative-task scores in the 50-film audit. The analysis covers all 450 model–film cases from nine evaluated models, yielding 1,350 paired task ratings across plot, synopsis, and key message.

Figure 9: Response-format recovery in affected runs. Repairs modify serialization only; they do not change task content or scores.

Model	Plot (S/T)	Synopsis (S/T)	Key message (S/T)	Mean change [95% CI]
GPT-5.6 Sol	4.58 / 4.68	3.38 / 3.54	3.00 / 2.96	$+0.07$ [ $-0.05$ , $+0.19$ ]
Gemini 3.8 Flash	4.58 / 4.68	3.58 / 3.36	3.14 / 3.20	$-0.02$ [ $-0.13$ , $+0.09$ ]
Claude Haiku 4.5	4.12 / 3.94	2.48 / 2.24	2.94 / 2.70	$-0.22$ [ $-0.40$ , $-0.05$ ]
DeepSeek V4 Pro	4.38 / 4.52	3.30 / 2.96	2.98 / 3.06	$-0.04$ [ $-0.20$ , $+0.13$ ]
Llama 4 Scout	2.82 / 2.60	1.88 / 1.60	2.50 / 2.30	$-0.23$ [ $-0.47$ , $-0.01$ ]

Appendix

Table 19: Narrative performance with title and year only versus subtitles. Task entries are mean rubric scores on a 1–5 scale, shown as subtitle-only/title-year-only for paired films. The final column is the paired change in the per-film mean across the three tasks, with a 95% film-level bootstrap interval.

Model	Plot (S/C)	Synopsis (S/C)	Key message (S/C)	Mean change [95% CI]
GPT-5.6 Sol	4.70 / 3.80	3.30 / 2.70	3.00 / 3.05	$-0.48$ [ $-0.82$ , $-0.17$ ]
Gemini 3.8 Flash	4.65 / 4.50	3.55 / 2.80	3.15 / 3.00	$-0.35$ [ $-0.57$ , $-0.15$ ]
Claude Haiku 4.5	4.10 / 3.95	2.40 / 2.35	2.90 / 2.75	$-0.12$ [ $-0.30$ , $+0.07$ ]
DeepSeek V4 Pro	4.45 / 4.25	3.20 / 2.60	2.95 / 3.15	$-0.20$ [ $-0.45$ , $+0.03$ ]
Llama 4 Scout	2.80 / 2.65	1.90 / 1.70	2.55 / 2.25	$-0.22$ [ $-0.53$ , $+0.10$ ]

Appendix

Table 20: Narrative performance on the character-name counterfactual subset. Task entries are mean rubric scores on a 1–5 scale, shown as subtitle-only/counterfactual-subtitle for paired films. The final column is the paired change in the per-film mean across the three tasks, with a 95% film-level bootstrap interval.

Model	Micro-F1 (S)	Micro-F1 (T)	Change (pp)	Exact (S, %)	Exact (T, %)
GPT-5.6 Sol	75.7	82.3	$+6.6$	20.0	30.0
Gemini 3.8 Flash	79.8	84.7	$+4.9$	24.0	40.0
Claude Haiku 4.5	67.3	72.4	$+5.1$	14.0	8.0
DeepSeek V4 Pro	79.3	81.8	$+2.5$	30.0	34.0
Llama 4 Scout	61.2	72.2	$+11.0$	2.0	18.0

Appendix

Table 21: Genre prediction from subtitles versus film title and year on the paired 50-film sample. Micro-F1 summarizes multi-label decisions across all genre labels; exact match requires the complete predicted genre set to match the reference.

Model	Composite score	Difference from Gemini (95% CI)	Holm-adjusted $p$
Gemini 3.8 Flash	78.04	Reference	—
GPT-5.6 Sol	74.70	$-3.34\;[-3.86,-2.84]$	$<0.001$
DeepSeek V4 Pro 1.6T	65.52	$-12.52\;[-13.61,-11.44]$	$<0.001$

Appendix

Table 25: Paired film-level comparisons of the leading composite scores. Differences are computed as comparator minus Gemini 3.8 Flash, so negative values indicate a higher Gemini score.

Figure 10: Narrative error profiles by model and subtask. Each cell reports the share of that subtask’s structured error annotations assigned to the indicated type. Synopsis errors concentrate in missing and invented events; key-message errors concentrate in semantic mismatch and over-generic themes.

Model	Family	Narrative Understanding and Generation						Cultural Prediction and Assessment
		Overall (1–5)			Genre micro-F1 (%)			Age exact (%)			Country EM (%)
		EN	CL	$\Delta$	EN	CL	$\Delta$	EN	CL	$\Delta$	EN	CL	$\Delta$
GPT-5.6 Sol	Closed frontier	3.56	3.52	$\downarrow$ -0.05	82.2	81.4	$\downarrow$ -0.8	33.5	30.6	$\downarrow$ -2.9	67.2	65.1	$\downarrow$ -2.1
Gemini 3.8 Flash	Closed frontier	3.61	3.52	$\downarrow$ -0.09	84.7	85.0	$\uparrow$ +0.3	36.5	33.8	$\downarrow$ -2.7	78.8	77.7	$\downarrow$ -1.1
Claude Haiku 4.5	Closed frontier	3.02	2.89	$\downarrow$ -0.14	72.2	71.3	$\downarrow$ -0.9	29.5	27.0	$\downarrow$ -2.5	58.0	53.6	$\downarrow$ -4.4
GPT-5 Nano	Closed baseline	2.66	2.47	$\downarrow$ -0.19	74.8	72.0	$\downarrow$ -2.8	19.5	20.9	$\uparrow$ +1.4	43.8	43.9	$\uparrow$ +0.1

Appendix

Table 27: English-to-cross-lingual comparison across headline tasks. EN denotes English, while CL is the equal-weight mean over Arabic, Indonesian, Persian, Romanian, and Vietnamese;

\Delta=\mathrm{CL}-\mathrm{EN}

. Positive and negative markers indicate the direction of the observed change. Paired film-level uncertainty intervals are reported in Table 37 .

Figure 11: Six-language-average scores for plot, synopsis, and key-message generation. Each row represents one model, and the connecting segment highlights the separation between premise-level plot recovery and fuller synopsis reconstruction.

Figure 12: Change in cultural prediction relative to English subtitle input. The upper panel reports age accuracy within one year and the lower panel reports country-rating exact match, in percentage points, for each model under Arabic, Indonesian, Persian, Romanian, and Vietnamese subtitles. Absolute language-specific values are reported in the accompanying tables.

Figure 13: Strict-category evidence F1 by model and lexical category. Darker teal indicates higher F1. Strong profanity is easiest overall, while mild obscenity combines low recall with frequent false positives.

Figure 14: English-only Ministral scaling in original metric units. The panels report narrative overall, genre micro-F1, age MAE (lower is better), and national-rating exact match.

Explore similar work

Jan 13, 2026cs.CL

STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Stories

Movie screenplays are rich long-form narratives that interleave complex character relationships, temporally ordered events, and dialogue-driven interactions. While prior benchmarks target individual subtasks such as question answering or dialogue generation, they rarely evaluate whether models can construct a coherent story world and use it consistently across multiple forms of reasoning and generation. We introduce STAGE (Screenplay Text, Agents, Graphs and Evaluation), a unified benchmark for narrative understanding over full-length movie screenplays. STAGE defines four tasks: knowledge graph construction, scene-level event summarization, long-context screenplay question answering, and in-script character role-playing, all grounded in a shared narrative world representation. The benchmark provides cleaned scripts, curated knowledge graphs, and event- and character-centric annotations for 150 films across English and Chinese, enabling holistic evaluation of models' abilities to build world representations, abstract and verify narrative events, reason over long narratives, and generate character-consistent responses.

Qiuyu Tian, Zequn Liu, Yiding Li +9

Southeast University, Nanjing, China · Beijing Zhongguancun Academy, Beijing, China · ZhuiWen Technology Co., Ltd., Beijing, China +1

Narratives Summarization Knowledge Graphs

Ask PDF

Apr 21, 2026cs.CL

Lost in Translation: Do LVLM Judges Generalize Across Languages?

Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their growing importance, these evaluators are almost exclusively assessed on English-centric benchmarks, leaving open the question of how well these evaluators generalize across languages. To answer this question, we introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation, which includes over 60K pairwise preference instances spanning 25 typologically diverse languages. MM-JudgeBench integrates two complementary subsets: a general vision-language preference evaluation subset extending VL-RewardBench, and a chart-centric visual-text reasoning subset derived from OpenCQA, enabling systematic analysis of reward models (i.e., LVLM judges) across diverse settings. We additionally release a multilingual training set derived from MM-RewardBench, disjoint from our evaluation data, to support domain adaptation. By evaluating 22 LVLMs (15 open-source, 7 proprietary), we uncover substantial cross-lingual performance variance in our proposed benchmark. Our analysis further shows that model size and architecture are poor predictors of multilingual robustness, and that even state-of-the-art LVLM judges exhibit inconsistent behavior across languages. Together, these findings expose fundamental limitations of current reward modeling and underscore the necessity of multilingual, multimodal benchmarks for developing reliable automated evaluators.

Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5

York University · University of Alberta · Nanyang Technological University +1

Multilingual Benchmark Large Language Model Judges Multimodal Benchmarks Evaluation Benchmarks Cross-Lingual Consistency

Ask PDF

Jun 16, 2026cs.CL

NarrativeWorldBench: A Frontier-Saturated Benchmark and a Latent World Model for Long-Horizon Co-Creative Audio Drama

Long-form serialized audio drama, with arcs that run for 200 to 800 episodes, is a major creative medium and a setting where frontier large language models (LLMs) fail. We benchmark 21 models, spanning classical, fine-tuned, open-frontier, closed-frontier, and reasoning tiers, on a uniform set of structural narrative metrics. All closed-frontier systems saturate at a plot-beat F1 in the band [0.78, 0.81] and collapse by about -0.20 F1 at horizon h=200. We introduce NarrativeWorldBench, an open benchmark of nine narrative-structure metrics evaluated across horizons h in {10, 20, 50, 100, 200}, with cross-lingual evaluation across four Indic languages (Hindi, Tamil, Telugu, Marathi). We introduce N-VSSM, a Narrative Variational State-Space Model that maintains a structured 256-dimensional latent world state over more than 200 episodes via a Mamba-2 backbone with an event-conditioned posterior and an 8B decoder. N-VSSM holds plot-beat F1 >= 0.84 across all horizons at 4x lower compute than the closed-frontier band. A learned Cultural Transfer Function lifts cross-language fidelity by +0.20 to +0.23 Likert points. In a within-subjects writer study (n = 12 professional authors, 240 trials), N-VSSM is preferred over Claude Opus 4.5 on long-arc consistency 71% of the time and rated +1.3 Likert points higher on controllability.

Logan Mann, Abdur Rahman, Mohammad Saifullah +2

University of California, Santa Barbara · Pocket FM

Narratives Large Audio Language Models Frontier Models Frontiers Audio Understanding Latent World Models

Ask PDF