Cultural Moment Benchmark: Evaluating Video Cultural Reasoning and Grounding in Southeast Asia

Burak Satar; Zhixin Ma; Cheng Yu-Tong; Huy Hoang Tran; Phuong Anh Nguyen; Chong-Wah Ngo

cs.CVAug 24, 2026

Cultural Moment Benchmark: Evaluating Video Cultural Reasoning and Grounding in Southeast Asia

Authors: Burak Satar, Zhixin Ma, Cheng Yu-Tong, Huy Hoang Tran, Phuong Anh Nguyen, Chong-Wah Ngo

Organizations: Singapore Management University · UIT, VNU-HCM

Abstract

Cultural understanding in video means more than recognizing what is visible; it requires grasping the symbolic and temporal significance of cultural concepts. We decompose this into three abilities: naming what a concept symbolizes, visually recognizing it on video, and locating its sub-events in time. Existing video-cultural benchmarks tend to test what is seen, collapsing these three abilities into a single score that hides the bottleneck. We introduce the Cultural Moment Benchmark (CMB): 306 expert-curated concepts from seven countries in Southeast Asia across five categories. We evaluate each concept through three stages, one per ability. Given a description, Stage 1 (S1) selects from four candidate concept names, Stage 2 (S2) selects from four candidate video moments, and Stage 3 (S3) predicts the start and end times of the moment in a video. To keep each stage focused on a distinct ability, we use three design choices: semantic-similarity distractors (S1, S2), unlabeled video moments (S2), and free-form localization on a different example video (S3). Across six vision-language models, failure modes vary by ability and modality. i) Even the strongest closed-source models score below 30% when all three stages must be correct; ii) The three abilities do not fully cascade: naming a concept correctly helps half the models recognize it on video, but recognizing it has little effect on locating the sub-event in time; iii) Audio is complementary, redundant, or distracting depending on the concept, more often distracting in non-Latin-script countries; removing both audio and subtitles hurts Games and Music the most. Our 14-rater human study shows that even Expert raters score below chance on concepts from a neighboring country, indicating that CMB requires country-specific cultural knowledge. CMB acts as a diagnostic harness, attributing failures to a specific ability or modality.

Figures & tables

Figure 1: CMB three-stage probe. Stage 1 picks among four concept names. Stage 2 keeps the same stem but replaces the options with four video moments from example video $A$ ; it uses an independent random seed, so the option positions differ from S1. Stage 3 predicts a free-form $[t_{\mathrm{start}},t_{\mathrm{end}}]$ span for a sub-event of the concept on a different example video $B$ . The shift from $A$ to $B$ prevents within-clip memorization in Stage 3.

Benchmark	SEA	Multi-stage	Cond.	Free-form	Avg. dur.	Human	Cultures
	focus	cascade	metrics	temp. loc.	(min)	eval	covered
SCB ${}^{\text{image}}$ ( 37 )	✓	✓	✓	✗	n/a	✗	7
AVMeme Exam ( 17 )	✗	✗	✗	✗	$\leq$ 0.5	✓ (closed)	5+
ChineseVideoBench ( 31 )	✗	✗	✗	✗	$\approx$ 1.0	✗	1
GIMMICK ( 40 )	partial	✗	✗	✗	0.17	✗	139
MINERVA-Cultural ( 43 )	partial	✗	✓	✗	12.62	✓	18

Table 1: Comparison of CMB to recent multicultural video benchmarks. SCB

{}^{\text{image}}

is the image-based precursor; all other rows are video benchmarks. Multi-stage cascade : distinct ability stages scored sequentially with conditional dependency. Cond. metrics : conditional metrics that report cross-stage success-given-success (e.g.,

P(\text{S2}\mid\text{S1 correct})

). Free-form temp. loc. : model predicts an unconstrained

[\hat{t}_{\text{start}},\hat{t}_{\text{end}}]

span rather than selecting from candidate timestamps or orderings. Human eval : ✓indicates humans recruited separately from annotators were evaluated on the same task; closed adds an explicit prohibition on web search or external aids. *: S3 videos.

Figure 2: CMB construction pipeline. A) Preprocessing yields 306 concepts (proposed by GPT-5, validated by Local Annotators) and 624 source videos (Set

A

for S2, Set

B

for S3); descriptions are drafted by DeepSeek-V3.1 and Qwen-SEA-LION-v4 and rewritten by a Cultural Annotator. B) Each concept is materialized into the S1 and S2 questions (four-way MCQ with semantic-similarity distractors) and the S3 question (free-form span on a different video); flagged S3 sub-events are also audited by a Reviewer. C) A Reviewer runs an Outsider filter check on S1/S2 descriptions (does the Reviewer answer correctly from regional cues alone?) and a triage check on flagged S3 spans. Concepts that pass both enter the Validated CMB; failures loop back to Stage Construction.

Model	Stage 1 (%)	Stage 2 (%)			Stage 3 (%)			Joint (%)
Model	Stage 1 (%)	Reset	Carry	Feedback	Reset	Carry	Feedback	Reset	Carry	Feedback
Gemini 3.1 Pro	77.8	62.7	72.9	85.9	31.9	33.4	33.3	20.6	27.0	27.8
GPT-5.4	70.9	52.6	68.3	81.7	29.5	31.0	30.3	14.5	21.7	21.5
Qwen3-VL-32B	51.0	39.9	31.4	28.4	22.0	22.3	30.0	6.4	4.3	4.6
Qwen3.5-27B	53.6	39.9	40.5	39.9	23.2	24.1	30.2	6.7	8.1	7.7
InternVL3.5-14B (n)	46.7	26.5	24.5	9.8	2.1	2.5	12.5	0.4	0.4	0.7

Table 2: Overall results from the 3-stage

\times

3-mode analysis. S1 and S2 scores refer to accuracy, and S3 score refers to mIoU. Midrules separate closed-source models (top), 27–32B open-source models (middle), and the 14B InternVL configurations (bottom). S1 has a single column because there is no prior context to carry.

Model	S2	c-S2	mIoU	c-mIoU
	S1 $\rightarrow$ S2 (%)		S2 $\rightarrow$ S3 (%)
Gemini 3.1 Pro	72.9	89.5 ∗∗∗	33.4	33.0
GPT-5.4	68.3	87.6 ∗∗∗	31.0	29.7
Qwen3.5-27B	40.5	46.3 ∗	24.1	23.4
Qwen3-VL-32B	31.4	30.8	22.3	23.1
InternVL-14B (t)	26.8	22.2	3.0	5.0 ∗

Table 3: Cascade analysis. The S2 and mIoU columns report unconditional performance; c-S2 and c-mIoU are the conditional metrics. Bold marks conditional metrics that differ significantly from their unconditional counterparts.

{}^{*}\,p{<}.05

{}^{**}\,p{<}.01

{}^{***}\,p{<}.001

Figure 3: Share of S3 video pairs (

n{=}631

) in each modality role (Complementary, Redundant, Distracting; defined in § 4.4 ) for the three ablations. The mIoU change

\Delta

is averaged across three models before binning. Complementary and Distracting shares are roughly balanced and cancel in the mean mIoU.

Figure 4: Modality roles across all S3 video pairs, segmented by category (top 5 rows) and country writing-system clusters (bottom 2 rows).

	S1 (%)		S2 (%)
Rater	Native	Non-native	Native	Non-native
Expert	58.4	21.3	51.9	28.7
Non-expert	39.2	27.9	45.2	31.5

Table 4: Human evaluation accuracy (%) on S1 and S2, by rater group (Expert vs Non-expert) and role (Native vs Non-native). Each cell averages over the seven country-level mean scores. More in Appendix B .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Country	Category	Concepts
Cambodia	Celebration (5)	Bon Chrat Preah Neangkoal, Choul Chnam Thmey, Meak Bochea, Pchum Ben, Visak Bochea
	Dance (7)	Robam Bes Kravanh, Robam Chuon Por, Robam Kngaok, Robam Phlet, Robam Phloy Suoy, Robam Tep Apsara, Robam Yike
	Game (6)	Bos Angkunh, Chol Chhoung, Krob Moin, Leak Kanseng, Ok Chaktrong, Sey
	Music (9)	Chhing, Khaen, Khloy, Kong Thom, Kong Touch, Roneat Ek, Skor Thom, Sneng, Sralai
	Wedding (1)	Kar Khmer
Indonesia	Celebration (14)	Bulan Puasa, Festival Gamelan Yogyakarta, Festival Rawa Pening, Festival Tabot, Galungan, Idul Adha, Kuningan, Lebaran, Nyepi, Pasola, Perang Topat, Pesta Kesenian Bali, Sekaten, Waisak

Appendix

Table 5: The 306 CMB cultural concepts (Common Latin script), grouped by country and category. Counts in parentheses. Official Latin and native-script (Local) variants are released in the JSON dataset; see Appendix A.11.

Figure 5: CMB concept and S3 question distribution. (a, b) Concepts per country and per category (

n{=}306

). Each concept corresponds to one S1 and one S2 question on Set

A

. (c, d) S3 questions per country and per category on Set

B

(

n{=}631

across 318 source videos).

Figure 6: Question length and video duration distributions. (a) Average S1, S2, and S3 question length (words) per country. (b) Video-duration histogram: Set

A

S2 moments are short (median 20 s), Set

B

S3 source videos are long (median 316 s). (c) S3 sub-event span lengths per category, median 13 s overall.

Figure 7: Sentence-BERT cosine similarity between the correct concept and (red) the three chosen S1/S2 distractors versus (gray) the remaining non-chosen candidates in the same per-category pool. Chosen distractors are systematically harder, with a median similarity of

0.70

0.60

for non-chosen candidates.

Statistic	Value
Concepts	306
S1 questions ( $=$ concepts)	306
S2 questions ( $=$ concepts)	306
S3 questions (Set $B$ , with repeats)	631
Unique Set $A$ S2-moment videos	306
Unique Set $B$ S3-source videos	318

Appendix

Table 6: CMB dataset summary statistics. Set A counts are 1-to-1 with concepts (one S2-moment video per concept); Set B counts are larger because multi-event traditions contribute more than one S3 sub-event span.

Country (target)	Cluster	Local raters	Non-local raters	Non-local clusters
Cambodia	Local	KH-a, KH-b	VN-a, TH-b	Latin + Local
Indonesia	Latin	ID-a, ID-b	KH-a, VN-b	Local + Latin
Malaysia	Latin	MY-a, MY-b	ID-a, KH-b	Latin + Local
Myanmar	Local	MM-a, MM-b	MY-a, ID-b	Latin + Latin
Philippines	Latin	PH-a, PH-b	MM-a, MY-b	Local + Latin
Thailand	Local	TH-a, TH-b	PH-a, MM-b	Latin + Local

Appendix

Table 7: Annotator assignment under the

+1/+2

rotation. Each of the seven countries is evaluated by two Local raters (own-country) and two Non-local raters drawn from two different other countries. Six of seven countries receive Non-local ratings from both script clusters (Latin-script: Indonesia(ID)/Malaysia(MY)/Philippines(PH)/Vietnam(VN); non-Latin-script: Cambodia(KH)/Myanmar(MM)/Thailand(TH)); Myanmar is the structural exception discussed in the text.

#	Annotator	Local (60% of own)	Non-local country	Non-local (60%)	Total
1	KH-a	17 (of 28)	Indonesia	29	46
2	KH-b	17	Malaysia	26	43
3	ID-a	29 (of 48)	Malaysia	26	55
4	ID-b	29	Myanmar	22	51
5	MY-a	26 (of 44)	Myanmar	22	48
6	MY-b	26	Philippines	31	57

Appendix

Table 8: Per-annotator S1 workload under the split-and-overlap scheme. Each annotator rates 60% of their own country’s concepts (40% solo + 20% overlap) plus 60% of one Non-local country under the same split. Per-annotator range: 42–64 MCQs (

\sim

32–48 min); total ratings collected:

\sim

732.

Country	$S_{1}$ $\kappa$	$S_{1}$ n	$S_{2}$ $\kappa$	$S_{2}$ n
Cambodia	$+0.005$	6	$+0.120$	6
Indonesia	$+0.079$	11	$+0.175$	11
Malaysia	$+0.056$	9	$+0.193$	9
Myanmar	$-0.172$	8	$+0.099$	8
Philippines	$-0.202$	10	$+0.225$	10
Thailand	$-0.052$	9	$+0.152$	9

Appendix

Table 9: Per-country 4-way Fleiss’

\kappa

on S1 and S2 across the 20% overlap items. Pooled row uses all 65 overlap items together. S2

\kappa

is positive across all seven countries; S1

\kappa

fluctuates around zero, consistent with S2’s visual grounding reducing rater disagreement relative to S1’s abstract symbolic prompts.

Figure 8: Human evaluation accuracy on a stratified 20% CMB subset. (a) Expert raters = 7 kits (ID-a, KH-b, MM-a, MY-b, PH-a, TH-b, VN-a) with prior cultural-research experience. (b) Non-expert raters = 7 kits (ID-b, KH-a, MM-b, MY-a, PH-b, TH-a, VN-b) from the general population. “Native” = answering one’s own country; “Non-native” = answering one other country via a fixed

+1/+2

rotation. The Model bar averages over the six Carry-mode models on the same concept set the humans saw (N = unique concepts

\times

6 models).

Cell	$n$	Gemini	GPT-5.4	Qwen3-VL	Qwen3.5	IVL (n)	IVL (t)
Indonesia	48	85.4 [74.5, 94.6]	70.8 [57.1, 83.3]	56.2 [42.0, 70.3]	58.3 [44.0, 72.4]	47.9 [33.3, 62.0]	54.2 [40.0, 68.1]
Malaysia	44	86.4 [75.6, 95.7]	86.4 [75.6, 95.8]	52.3 [37.5, 66.7]	61.4 [46.3, 75.7]	54.5 [39.4, 69.6]	43.2 [28.6, 58.3]
Philippines	52	84.6 [74.4, 93.8]	76.9 [64.6, 87.7]	55.8 [42.5, 69.0]	57.7 [44.2, 71.1]	48.1 [34.5, 62.2]	42.3 [28.8, 55.9]
Vietnam	55	89.1 [80.0, 96.5]	74.5 [62.7, 85.7]	52.7 [39.3, 66.1]	58.2 [44.8, 71.2]	45.5 [32.6, 58.8]	32.7 [20.8, 45.5]
Cambodia	28	67.9 [50.0, 84.6]	57.1 [38.5, 75.7]	50.0 [31.0, 69.2]	32.1 [15.4, 50.0]	53.6 [34.5, 71.4]	39.3 [21.4, 58.3]
Myanmar	37	56.8 [40.0, 72.7]	62.2 [46.2, 77.5]	56.8 [40.0, 72.7]	45.9 [29.6, 62.9]	45.9 [30.0, 62.5]	45.9 [30.0, 62.5]

Appendix

Table 10: Cluster-bootstrap 95% confidence intervals for S1 (%) under Carry with the base prompt, resampling the

306

concepts with replacement (

10{,}000

resamples, seed

42

, percentile method);

n

is concepts per cell. The statistic is concept-level: each resampled concept contributes one binary outcome. All six models are evaluated on the same

306

concepts and

631

Stage-3 pairs, so cells are directly comparable. Model abbreviations: Gemini = Gemini 3.1 Pro, GPT-5.4, Qwen3-VL = Qwen3-VL-32B, Qwen3.5 = Qwen3.5-27B, IVL = InternVL3.5-14B with (t) and without (n) inference-time reasoning.

Cell	$n$	Gemini	GPT-5.4	Qwen3-VL	Qwen3.5	IVL (n)	IVL (t)
Indonesia	48	81.2 [69.4, 91.8]	68.8 [55.3, 81.5]	29.2 [16.7, 42.6]	50.0 [35.7, 64.3]	22.9 [11.5, 35.6]	25.0 [13.0, 37.8]
Malaysia	44	86.4 [75.0, 95.6]	77.3 [63.8, 89.2]	31.8 [18.2, 46.2]	40.9 [26.2, 56.0]	22.7 [11.1, 36.0]	27.3 [14.3, 40.8]
Philippines	52	82.7 [71.8, 92.3]	75.0 [62.7, 86.3]	36.5 [23.7, 50.0]	40.4 [27.3, 54.2]	25.0 [13.6, 37.0]	19.2 [8.9, 30.8]
Vietnam	55	81.8 [70.9, 91.5]	80.0 [69.0, 90.0]	30.9 [18.9, 43.5]	43.6 [30.3, 57.1]	25.5 [14.1, 37.1]	36.4 [24.0, 50.0]
Cambodia	28	53.6 [34.5, 72.0]	67.9 [50.0, 85.2]	25.0 [9.5, 42.3]	28.6 [12.1, 46.7]	35.7 [18.2, 54.2]	25.0 [10.0, 42.3]
Myanmar	37	56.8 [40.0, 72.4]	59.5 [43.2, 75.0]	40.5 [24.3, 56.8]	40.5 [25.0, 57.1]	21.6 [9.1, 35.5]	32.4 [17.6, 48.3]

Appendix

Table 11: Cluster-bootstrap 95% confidence intervals for S2 (%) under Carry with the base prompt, resampling the

306

concepts with replacement (

10{,}000

resamples, seed

42

, percentile method);

n

is concepts per cell. The statistic is concept-level: each resampled concept contributes one binary outcome. All six models are evaluated on the same

306

concepts and

631

Cell	$n$	Gemini	GPT-5.4	Qwen3-VL	Qwen3.5	IVL (n)	IVL (t)
Indonesia	48	33.4 [24.8, 44.0]	27.2 [19.4, 38.2]	23.3 [16.9, 30.8]	25.1 [18.5, 33.3]	2.5 [1.2, 4.4]	4.7 [2.1, 8.4]
Malaysia	44	33.4 [23.0, 47.1]	37.0 [26.8, 45.6]	23.9 [13.2, 34.5]	26.8 [16.6, 39.6]	3.2 [1.2, 5.9]	2.2 [1.0, 3.7]
Philippines	52	25.3 [14.3, 47.1]	24.8 [11.9, 46.3]	15.1 [9.5, 28.9]	17.8 [11.1, 31.2]	2.7 [1.1, 5.8]	2.0 [0.6, 5.3]
Vietnam	55	30.3 [23.5, 38.1]	27.8 [20.3, 37.1]	20.4 [13.2, 29.7]	20.5 [14.1, 29.1]	2.2 [0.6, 4.0]	2.3 [0.4, 4.7]
Cambodia	28	53.3 [38.1, 71.0]	47.4 [31.1, 67.0]	36.8 [22.6, 53.9]	28.9 [16.7, 44.1]	3.4 [0.0, 8.7]	4.0 [0.3, 9.0]
Myanmar	37	43.6 [31.4, 56.7]	35.2 [23.8, 46.7]	32.6 [20.0, 45.9]	32.3 [20.0, 45.1]	1.9 [0.4, 3.7]	2.2 [0.0, 6.3]

Appendix

Table 12: Cluster-bootstrap 95% confidence intervals for mIoU (%) under Carry with the base prompt, resampling the

306

concepts with replacement (

10{,}000

resamples, seed

42

, percentile method);

n

is concepts per cell. The statistic is pair-weighted: each resampled concept contributes all of its Stage-3 pairs, so concepts drawn twice count twice. All six models are evaluated on the same

306

concepts and

631

Model	Scope	S1	S2	mIoU
all values in %, matching Table 2
Gemini	micro	77.8 [73.2, 82.4]	72.9 [67.6, 77.8]	33.4 [28.3, 39.1]
	macro-ctry	76.0 [71.1, 80.8]	70.7 [65.5, 75.7]	37.2 [33.1, 43.5]
	macro-cat	78.5 [72.1, 84.2]	74.2 [67.5, 80.1]	36.2 [32.5, 40.7]
GPT-5.4	micro	70.9 [65.7, 75.8]	68.3 [63.1, 73.5]	31.0 [25.8, 36.7]
	macro-ctry	69.6 [64.2, 74.7]	67.3 [61.9, 72.5]	34.6 [30.4, 40.7]

Appendix

Table 13: Micro and macro averages with cluster-bootstrap 95% confidence intervals (Carry, base prompt). micro pools all

306

concepts and, for mIoU, all

631

Stage-3 pairs. macro-ctry and macro-cat give each of the

7

countries and

5

categories equal weight. Macro intervals are bootstrapped through the same concept resamples as the cells, not assembled from their bounds. S1 and S2 agree within

2.5

points across every model and scope. All values are percentages, matching Table 2 . mIoU runs

1

4

points higher under macro because Stage-3 pair mass is concentrated in Wedding, which supplies

3.6\%

of concepts but

23.5\%

of pairs and is the hardest cell for every model. Model abbreviations: Gemini = Gemini 3.1 Pro, GPT-5.4, Qwen3-VL = Qwen3-VL-32B, Qwen3.5 = Qwen3.5-27B, IVL = InternVL3.5-14B with (t) and without (n) inference-time reasoning.

Comparison	$\Delta$ S1	$\Delta$ S2	$\Delta$ mIoU
Gemini vs. GPT-5.4	+6.9 [+2.3, +11.4]	+4.6 [-0.7, +9.8]	+2.5 [-1.6, +6.4]
Gemini vs. Qwen3-VL	+26.8 [+20.6, +33.0]	+41.5 [+34.6, +48.4]	+11.2 [+7.8, +15.2]
Gemini vs. Qwen3.5	+24.2 [+17.6, +30.7]	+32.4 [+25.5, +39.2]	+9.4 [+5.7, +13.5]
Gemini vs. IVL (n)	+31.0 [+24.5, +37.6]	+48.4 [+41.2, +55.2]	+31.0 [+26.0, +36.4]
Gemini vs. IVL (t)	+36.6 [+29.7, +43.5]	+46.1 [+38.9, +52.9]	+30.4 [+25.5, +35.9]
GPT-5.4 vs. Qwen3-VL	+19.9 [+13.4, +26.1]	+36.9 [+30.1, +43.5]	+8.7 [+4.7, +13.2]

Appendix

Table 14: Paired cluster bootstrap on pairwise model differences (Carry, base prompt, micro). Because all six models are evaluated on the same

306

concepts, the difference is taken within each of the

10{,}000

concept resamples; this is materially more powerful than checking whether two marginal intervals overlap.

\Delta

mIoU is in mIoU points

\times 100

. Bold marks differences whose

95\%

interval excludes zero. Every closed-weight against open-weight comparison is significant on all three metrics; the Gemini lead over GPT-5.4 is significant on S1 only; and inference-time reasoning yields no measurable difference for InternVL3.5-14B. Model abbreviations: Gemini = Gemini 3.1 Pro, GPT-5.4, Qwen3-VL = Qwen3-VL-32B, Qwen3.5 = Qwen3.5-27B, IVL = InternVL3.5-14B with (t) and without (n) inference-time reasoning. Significance markers are per-comparison; we do not correct for multiple comparisons.

Ablation	$n$ pairs	$\overline{\Delta}$ mIoU points	$p$
No-audio	631	$-0.02$	.55
No-subtitles	631	$-0.65$	.10
Visual-only	631	$-0.81$	.045

Appendix

Table 15: Aggregate mIoU shift per modality ablation, pooled across the three ablation-instrumented models (Gemini 3.1 Pro, Qwen3-VL-32B, Qwen3.5-27B).

p

-values from paired tests against the baseline. Per-pair role classifications derived from these shifts appear in Figure 3 .

Cluster	Distr. audio (%)	Distr. subs (%)
Latin-script (ID/MY/PH/VN)	14.0	11.9
non-Latin-script (KH/MM/TH)	29.1	20.4

Appendix

Table 16: Distracting role share (%) by writing-system cluster, pooled across Gemini 3.1 Pro, Qwen3-VL-32B, and Qwen3.5-27B over the same 631 S3 pairs as Figure 3 . Non-Latin-script cluster shows a

+15

pp audio gap (

p<.001

) and a

+8.5

pp subtitle gap.

Cut at $\pm 3$ points
Ablation	Comp.	Redun.	Distr.
No-audio	20.6	59.6	19.8
No-subtitles	20.3	63.5	16.2
Visual-only	24.2	56.3	19.5
Cut at $\pm 5$ points, used in the paper
No-audio	17.0	66.4	16.6

Appendix

Table 17: Modality-role shares as a percentage of the

631

Stage-3 pairs, at three values of the classification cut.

\Delta

is the mean change in mIoU across the three ablation-instrumented models. Redundant stays the majority role in every row.

Figure 9: Per-(model, mode) cascade deltas. Top:

\Delta\text{S2}=\text{c-S2}-\text{S2}

(percentage points), measuring how much correct naming lifts visual recognition. Bottom:

\Delta\text{mIoU}=\text{c-mIoU}-\text{mIoU}

, measuring how much correct visual recognition lifts temporal localization. Bars are sorted by effect size within each panel. Stars:

*\,p<0.05

**\,p<0.01

***\,p<0.001

against the unconditional baseline.

\Delta\text{mIoU}

is in IoU units on [0,1]; multiplied by 100, it gives the points used in the text.

Figure 10: S3 mIoU under Carry mode, bucketed by Set B source-video duration. Below each bucket:

n=

number of S3 pairs in the bucket, and

\overline{\ell}=

mean source-video duration. Markers annotate the mean ground-truth sub-event span in the bucket (right axis). Every model falls sharply once videos exceed about ten minutes, while mIoU rises across the shorter buckets as the mean sub-event span grows. Closed-source models degrade more gracefully. mIoU on [0,1].

Figure 11: Per-clip S3 IoU distribution under Carry mode, across all 306 concepts. (a) ECDF of per-clip IoU; the dashed vertical line marks IoU

=

0.3; one minus the ECDF value at the line gives the [email protected] hit rate per model. (b) Violin plot of the same distributions, with a strip of mean (filled dot) and median (horizontal bar) markers. A single mIoU number hides a long-tailed distribution with a heavy mass near zero for every model.

Figure 12: S3 IoU@0.3 (percentage of pairs with IoU $\geq 0.3$ ) for all six models under all three modes, split by category (left) and by country writing-system cluster (right). Cells report the percentage; row $n$ is the number of pairs aggregated. — Figure 12: S3 [email protected] (percentage of pairs with IoU $\geq 0.3$ ) for all six models under all three modes, split by category (left) and by country writing-system cluster (right). Cells report the percentage; row $n$ is the number of pairs aggregated.

Figure 13: S3 mIoU (mean Intersection-over-Union) for all six models under all three modes, split by category (left) and by country writing-system cluster (right). Cells report mIoU in percent; row $n$ is the number of pairs aggregated.

	TimeSuite	TRACE	MUSEG
	mIoU (%)	mIoU (%)	mIoU (%)
Celebration	9.5	8.8	22.4
Dance	3.9	2.0	15.5
Game	11.2	6.0	21.6
Music	10.6	18.6	26.4
Wedding	10.5	13.5	13.8

Appendix

Table 18: S3 temporal-localization mIoU of specialized temporal-grounding models on CMB, broken down by category. All three models receive the concept name and sub-event cue as text input and produce a free-form

[\hat{t}_{\text{start}},\hat{t}_{\text{end}}]

span on video

B

. They all evaluate at 1 fps with 224

\times

224 resolution.

Figure 14: S1 reference variance across model configurations. For each model, S1 is rerun four times using the four name-reference variants per concept (common vs. official

\times

Latin transliteration vs. local script); the bar reports the maximum minus the minimum S1 accuracy across the four runs (in percentage points), and the value in parentheses is the mean S1 accuracy (

\mu

) across them. Two configurations are reference-sensitive at the

\geq 5

pp threshold (GPT-5.4 low-1024, Qwen3.5-27B nothink); the remaining five stay within 3 pp, indicating that S1 scores are robust to the choice of name reference.

Figure 15: S1 (Naming) accuracy for all six models, split by category (left) and by country writing-system cluster (right). Cells report accuracy (%); row $n$ is the number of concepts aggregated.

Video Understanding

Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.

See all 117 papers

Sep 1, 2026

MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models

Meme understanding goes beyond recognizing visual content or literal text; it requires implicit cultural knowledge and pragmatic inference that most vision-language models still lack. We introduce MemeCULT-1K, a multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi, where each meme is paired with a cultural context note and three human-written explanations, along with a supplementary set of 54 Bengali regional dialect memes. We evaluate thirteen popular Vision Language Models (VLMs) under two settings: meme-only and context-aware. Providing minimal cultural context yields consistent gains across all models and languages: mean SBERT similarity improves from 44.6 to 56.4 (+11.8), BLEURT from 37.3 to 42.3 (+5.0), and LLM-as-a-Judge scores from 2.57 to 3.43 out of 5 (+0.86). Fine-grained error analysis reveals that closed-source models fail mainly on entity and reference misidentification, while open-source models are bottlenecked by broader cultural knowledge gaps, with linguistic and phonological failures proving the most context-resistant across both. These results highlight the difficulty of culturally grounded meme understanding and motivate future work on explicit cultural knowledge integration. Our dataset and code are publicly available at TawsifDipto17/MemeCULT-1K.

Tawsif Tashwar Dipto, Mehedi Ahamed, Radib Bin Kabir +5

1Islamic University of Technology · 2South East University · 3Vector Institute +3

VLM Evaluation Multilingual VLM Evaluation Cultural Knowledge in Language Models

Ask PDF

Aug 3, 2026

CultureVidBench: Benchmarking Cultural Understanding in Text-to-Video Generation

Text-to-video (T2V) generation models have advanced rapidly, yet their ability to represent diverse cultural contexts remains underexplored. Existing benchmarks mainly focus on perceptual quality, physical plausibility, and text-video alignment, but do not directly assess whether generated videos capture culturally specific objects, actions, rituals, visible text, or audio cues. We introduce CultureVidBench, a comprehensive benchmark for evaluating cultural understanding in T2V generation. CultureVidBench contains 1,000 curated prompts covering 12 countries, 6 continents, 8 cultural regions, and 14 cultural aspects organized into three categories: material culture, social practice & performance, and ritual & ceremony. Designed specifically for video generation, CultureVidBench emphasizes dynamic and multimodal cultural representation, including social interactions, ritual procedure, and culturally appropriate visible text and audio. We evaluate seven representative T2V models through human user studies and MLLM-based automatic assessment across cultural faithfulness, multimodal cultural rendering, semantic adherence, and perceptual quality. Results show that although current models achieve strong semantic adherence and visual quality, they often fail to faithfully capture fine-grained cultural details, particularly for underrepresented regions, rituals, and multimodal cultural cues.

Xianjing Han, Yuhan Su, Yang Deng +3

Nanyang Technological University · Centrale Supélec · Singapore Management University +1

T2V Generation Video Generation Video Generation Evaluation

Ask PDF

Apr 8, 2026

Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic alignment with reference annotations. To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.

Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir +4

University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA

Cultural Knowledge in Language Models VLM Evaluation Cross-Cultural Language Model Evaluation

Ask PDF

Abstract

Video Understanding

Explore similar work

MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models

CultureVidBench: Benchmarking Cultural Understanding in Text-to-Video Generation

Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images