Order Matters: A Chinese Multi-Panel Meme Benchmark for Vision-Language Reasoning

Haihan Li; Haihao Li; Zhenfei Xu; Jize Qian; Yubo Xie

cs.CVAug 27, 2026

Order Matters: A Chinese Multi-Panel Meme Benchmark for Vision-Language Reasoning

Authors: Haihan Li, Haihao Li, Zhenfei Xu, Jize Qian, Yubo Xie

Organizations: Shanghai Maritime University, Shanghai, China · Dalian Maritime University, Dalian, China · Oxford Suzhou Centre for Advanced Research

Abstract

Many multimodal tasks depend on how visual elements are ordered and composed, not only on recognizing them in isolation. Internet memes are a compact case of this problem: their punchline often depends on a constrained reading order and cross-panel visual--textual cues. While large vision-language models (LVLMs) show strong performance on single-image understanding, it remains unclear whether they can perform sequence-aware reasoning over structured meme layouts, especially in Chinese social media. We introduce CMPM, a Chinese Multi-Panel Meme benchmark with 1,214 annotated samples covering five structural types, ordering dependency, panel-order constraints, and optional comment context. We formulate a two-layer evaluation: Task1 probes structure typing and order-sensitive panel sequencing (with a context ablation setting), and Task2 evaluates Chinese meme explanation generation with human ratings on five 1-3 Likert dimensions (visual, panel, humor, context, and faithfulness). We benchmark five representative LVLMs under a unified protocol. Results indicate that canonical-display accuracy is not by itself evidence of order understanding: the primary shuffled condition produces a sharp accuracy drop, revealing a persistent gap in order-sensitive multimodal reasoning. Task2 preferences place Gemini 3.1 Pro and GPT-5.5 above the open models, while comment context yields only a small and mixed Core4 gain. Code and data will be released upon acceptance.

Figures & tables

Figure 1: Order matters in multi-panel memes. In the canonical order, P1 establishes a truth-versus-lie premise, P2–P3 form an interrupted conversational transition, and P4 delivers the service-style punchline “Customer first.” The shuffled presentation preserves the local visual content but breaks this setup-to-punchline progression, illustrating why local cue matching is insufficient for global meme understanding. The Chinese dialogue reads: P1, “One of us can only lie forever, and the other can only tell the truth forever”; P2, “This…”; P3, “Oops!”; P4, “Customer first.”

Resource	Lang.	Multi-panel	OS	Context	HE	Scale
Hateful Memes [ 7 ]	EN					$\sim$ 10k
MemeReaCon [ 20 ]	EN			$\checkmark^{*}$		1,565
CHIME [ 18 ]	ZH				$\checkmark^{\dagger}$	1.5k
CMPM (ours)	ZH	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	1,214

Table 1: Comparison with representative meme / multimodal resources. Context: use of post text or comments. HE: human-evaluated explanations. OS: order-sensitive panel sequencing.

Task	Sub-task	Setting / $n$	Metrics
Task 1	Structure typing	Canonical, no comments; $N{=}1214$	Acc., macro-/w- $F_{1}$
	Order recovery	Shuffled, no comments; $n{=}868$	Group-aware order Acc.
	CAS	$2{\times}2$ display $\times$ context; ${\approx}252$ /cell	Order Acc.
Task 2	Explanation + human ratings	Main 64 full-context memes; paired ablation on 31 matched pairs; 3 raters	Mean Likert (5 dims, 1–3)

Table 2: CMPM task taxonomy. Primary order pool: gold- strong non-parallel samples with order groups.

Figure 2: Five primary structure types in CMPM with representative multi-panel examples (narrative, progressive, conversational, comparison, and parallel).

Type	$n$	%	Ordering	$n$	%
comparison	373	30.7	strong	868	71.5
narrative	319	26.3	weak	346	28.5
conversational	285	23.5
parallel	137	11.3
progressive	100	8.2

Table 3: CMPM corpus statistics under final gold labels (

N=1214

Model	$n$ scored	Acc. (%)
InternVL3.5-8B	863	6.1
Qwen3.5-9B	865	27.4
GLM-4.1V-9B-Thinking	865	15.6
GPT-5.5	868	57.8
Gemini 3.1 Pro	868	75.2
Display-order	868	0.0

Table 4: Task 1 primary order recovery (gold- strong , shuffled, no context). Acc. denotes group-aware logical-order accuracy (%).

Model	Acc.	Macro- $F_{1}$	W.- $F_{1}$	Parse
InternVL3.5-8B	53.6	41.8	48.7	99.7
Qwen3.5-9B	56.1	47.4	51.2	100.0
GLM-4.1V-9B-Thinking	55.8	50.7	52.5	99.8
GPT-5.5	76.3	74.9	76.4	100.0
Gemini 3.1 Pro	69.7	67.9	67.0	100.0

Table 5: Task 1 structure typing (

N=1214

, canonical order, no context). Metrics in %.

	Correct display		Shuffled display
Model	no ctx	+cmts	no ctx	+cmts
InternVL3.5-8B	92.8	85.6	4.8	6.8
Qwen3.5-9B	85.7	88.9	23.5	23.1
GLM-4.1V-9B-Thinking	97.2	96.0	10.4	13.5
GPT-5.5	92.9	91.7	51.2	51.2
Gemini 3.1 Pro	90.5	94.8	73.0	71.3

Table 6: CAS logical-order accuracy (%). Each cell uses comment-available order-pool memes (

n\approx 250

–

252

scored).

Model	ann_A	ann_B	ann_C
InternVL3.5-8B	1.91	2.03	2.16
Qwen3.5-9B	2.00	2.08	2.33
GLM-4.1V-9B-Thinking	2.03	2.36	2.47
GPT-5.5	2.43	2.49	2.65
Gemini 3.1 Pro	2.78	2.71	2.79

Table 7: Task 2 main scalar evaluation (64 unique context-present memes). Mean of five Likert dimensions (1–3) per annotator. Higher is better; relative preferences are derived separately from holistic rankings.

Model	full_context	image_only	$\Delta$
InternVL3.5-8B	2.03	1.87	+0.16
Qwen3.5-9B	2.09	2.09	+0.00
GLM-4.1V-9B-Thinking	2.38	2.37	+0.01
GPT-5.5	2.52	2.49	+0.03
Gemini 3.1 Pro	2.77	2.69	+0.08
Overall	2.36	2.30	+0.06

Table 8: Task 2 context ablation (ABC combined; 31 matched pairs from the curated Task 2 evaluation pool;

n{=}93

ratings per model–condition cell and

n{=}465

for the overall aggregate). Core4

=

mean of Visual, Panel, Humor, and Faithfulness (excludes Context).

Figure 3: Illustrative Task 1 order-recovery errors under shuffled display. Numbers are shuffled display indices; each subfigure contrasts gold and observed orders.

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

Jul 30, 2026

MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memes

Large vision-language models have improved at describing visual content, but accurate descriptions do not ensure interpretation when meaning depends on knowledge beyond the pixels. Memes expose this gap because they rely on cultural entities, background knowledge, and community conventions. Most meme benchmarks reduce interpretation to labels or holistic scores, obscuring where an explanation breaks down. We introduce MemeBench, a diagnostic benchmark of 1,253 Chinese and English memes with human-written references and quality-controlled VIKR annotations, centered on anime, comics, games, and adjacent online subcultures. Its VIKR schema decomposes explanations into Visual clues, Identity links, Knowledge units, and Reasoning mechanisms. Across 26 LVLMs, every model covers visible content more reliably than the knowledge needed to interpret it, and even the strongest retains a 22.6% Visual-Knowledge gap. To test whether this diagnosis can guide improvement, we introduce KAR, an entity-guided retrieval baseline built on CultureBase. Across four controlled models, KAR raises VIKR Success by 3.6-7.4% and, compared with generic retrieval, repairs more answers and breaks fewer. Yet both retrieval conditions improve Identity and Knowledge while reducing Visual coverage in every comparison. MemeBench reveals whether an interpretation succeeds, what is missing, and whether targeted evidence fills the diagnosed gap.

Weihang Wang, Kainan Tu, Jielei Zhang +9

1Bilibili · 2Fudan University · 3Peking University

VLM Evaluation Vision-Language Models VLM Reasoning Cultural Knowledge in Language Models

Ask PDF

Jun 21, 2026

MMGist: A Comprehensive Multimodal Benchmark for 2027

We conduct a systematic study of 18 widely used vision-language benchmarks and identify three major issues: 1) many items do not rely on visual cues and therefore fail to effectively measure multimodal understanding; 2) many items are already close to performance saturation for current LVLMs, which limits their discriminative power; 3) a small number of anomalous items affect the reliability of evaluation results. To this end, we propose MMGist, a curated benchmark that covers seven capability dimensions and contains 7,262 items. MMGist is constructed through a three-stage pipeline, which sequentially combines text-ablation filtering, cross-model saturation filtering, and anomaly detection filtering. We conduct extensive experiments on 27 leading LVLMs and compare MMGist with the raw pool of 23,250 items. The results show that MMGist preserves model rankings with high fidelity, with Spearman

ρ= 0.98

, while reducing evaluation items by 69% and improving cross-model discrimination by 78%. Further results indicate that Visual Logic remains a systematic weakness of current LVLMs, while knowledge-intensive dimensions such as Expert Knowledge dimensions remain important factors for distinguishing closed-source models from open-source models. These findings suggest that high-quality evaluation should prioritize visual dependency, discriminative power, and reliability, rather than simply pursuing benchmark scale.

Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong +3

1Shanghai Jiao Tong University · 2Sichuan University

VLM Evaluation Large Vision-Language Models Multimodal Foundation Models Multimodal Learning Multimodal Understanding MLLM Evaluation

Ask PDF

Sep 1, 2026

MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models

Meme understanding goes beyond recognizing visual content or literal text; it requires implicit cultural knowledge and pragmatic inference that most vision-language models still lack. We introduce MemeCULT-1K, a multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi, where each meme is paired with a cultural context note and three human-written explanations, along with a supplementary set of 54 Bengali regional dialect memes. We evaluate thirteen popular Vision Language Models (VLMs) under two settings: meme-only and context-aware. Providing minimal cultural context yields consistent gains across all models and languages: mean SBERT similarity improves from 44.6 to 56.4 (+11.8), BLEURT from 37.3 to 42.3 (+5.0), and LLM-as-a-Judge scores from 2.57 to 3.43 out of 5 (+0.86). Fine-grained error analysis reveals that closed-source models fail mainly on entity and reference misidentification, while open-source models are bottlenecked by broader cultural knowledge gaps, with linguistic and phonological failures proving the most context-resistant across both. These results highlight the difficulty of culturally grounded meme understanding and motivate future work on explicit cultural knowledge integration. Our dataset and code are publicly available at TawsifDipto17/MemeCULT-1K.

Tawsif Tashwar Dipto, Mehedi Ahamed, Radib Bin Kabir +5

1Islamic University of Technology · 2South East University · 3Vector Institute +3

VLM Evaluation Multilingual VLM Evaluation Cultural Knowledge in Language Models

Ask PDF

Abstract

Multimodal Reasoning

Explore similar work

MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memes

MMGist: A Comprehensive Multimodal Benchmark for 2027

MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models