Order Matters: A Chinese Multi-Panel Meme Benchmark for Vision-Language Reasoning

Haihan Li; Haihao Li; Zhenfei Xu; Jize Qian; Yubo Xie

cs.CVAug 27, 2026

Order Matters: A Chinese Multi-Panel Meme Benchmark for Vision-Language Reasoning

Authors: Haihan Li, Haihao Li, Zhenfei Xu, Jize Qian, Yubo Xie

Organizations: Shanghai Maritime University, Shanghai, China · Dalian Maritime University, Dalian, China · Oxford Suzhou Centre for Advanced Research

Abstract

Many multimodal tasks depend on how visual elements are ordered and composed, not only on recognizing them in isolation. Internet memes are a compact case of this problem: their punchline often depends on a constrained reading order and cross-panel visual--textual cues. While large vision-language models (LVLMs) show strong performance on single-image understanding, it remains unclear whether they can perform sequence-aware reasoning over structured meme layouts, especially in Chinese social media. We introduce CMPM, a Chinese Multi-Panel Meme benchmark with 1,214 annotated samples covering five structural types, ordering dependency, panel-order constraints, and optional comment context. We formulate a two-layer evaluation: Task1 probes structure typing and order-sensitive panel sequencing (with a context ablation setting), and Task2 evaluates Chinese meme explanation generation with human ratings on five 1-3 Likert dimensions (visual, panel, humor, context, and faithfulness). We benchmark five representative LVLMs under a unified protocol. Results indicate that canonical-display accuracy is not by itself evidence of order understanding: the primary shuffled condition produces a sharp accuracy drop, revealing a persistent gap in order-sensitive multimodal reasoning. Task2 preferences place Gemini 3.1 Pro and GPT-5.5 above the open models, while comment context yields only a small and mixed Core4 gain. Code and data will be released upon acceptance.

Figures & tables

Figure 1: Order matters in multi-panel memes. In the canonical order, P1 establishes a truth-versus-lie premise, P2–P3 form an interrupted conversational transition, and P4 delivers the service-style punchline “Customer first.” The shuffled presentation preserves the local visual content but breaks this setup-to-punchline progression, illustrating why local cue matching is insufficient for global meme understanding. The Chinese dialogue reads: P1, “One of us can only lie forever, and the other can only tell the truth forever”; P2, “This…”; P3, “Oops!”; P4, “Customer first.”

Resource	Lang.	Multi-panel	OS	Context	HE	Scale
Hateful Memes [ 7 ]	EN					$\sim$ 10k
MemeReaCon [ 20 ]	EN			$\checkmark^{*}$		1,565
CHIME [ 18 ]	ZH				$\checkmark^{\dagger}$	1.5k
CMPM (ours)	ZH	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	1,214

Table 1: Comparison with representative meme / multimodal resources. Context: use of post text or comments. HE: human-evaluated explanations. OS: order-sensitive panel sequencing.

Task	Sub-task	Setting / $n$	Metrics
Task 1	Structure typing	Canonical, no comments; $N{=}1214$	Acc., macro-/w- $F_{1}$
	Order recovery	Shuffled, no comments; $n{=}868$	Group-aware order Acc.
	CAS	$2{\times}2$ display $\times$ context; ${\approx}252$ /cell	Order Acc.
Task 2	Explanation + human ratings	Main 64 full-context memes; paired ablation on 31 matched pairs; 3 raters	Mean Likert (5 dims, 1–3)

Table 2: CMPM task taxonomy. Primary order pool: gold- strong non-parallel samples with order groups.

Figure 2: Five primary structure types in CMPM with representative multi-panel examples (narrative, progressive, conversational, comparison, and parallel).

Type	$n$	%	Ordering	$n$	%
comparison	373	30.7	strong	868	71.5
narrative	319	26.3	weak	346	28.5
conversational	285	23.5
parallel	137	11.3
progressive	100	8.2

Table 3: CMPM corpus statistics under final gold labels (

N=1214

Model	$n$ scored	Acc. (%)
InternVL3.5-8B	863	6.1
Qwen3.5-9B	865	27.4
GLM-4.1V-9B-Thinking	865	15.6
GPT-5.5	868	57.8
Gemini 3.1 Pro	868	75.2
Display-order	868	0.0

Table 4: Task 1 primary order recovery (gold- strong , shuffled, no context). Acc. denotes group-aware logical-order accuracy (%).

Model	Acc.	Macro- $F_{1}$	W.- $F_{1}$	Parse
InternVL3.5-8B	53.6	41.8	48.7	99.7
Qwen3.5-9B	56.1	47.4	51.2	100.0
GLM-4.1V-9B-Thinking	55.8	50.7	52.5	99.8
GPT-5.5	76.3	74.9	76.4	100.0
Gemini 3.1 Pro	69.7	67.9	67.0	100.0

Table 5: Task 1 structure typing (

N=1214

, canonical order, no context). Metrics in %.

	Correct display		Shuffled display
Model	no ctx	+cmts	no ctx	+cmts
InternVL3.5-8B	92.8	85.6	4.8	6.8
Qwen3.5-9B	85.7	88.9	23.5	23.1
GLM-4.1V-9B-Thinking	97.2	96.0	10.4	13.5
GPT-5.5	92.9	91.7	51.2	51.2
Gemini 3.1 Pro	90.5	94.8	73.0	71.3

Table 6: CAS logical-order accuracy (%). Each cell uses comment-available order-pool memes (

n\approx 250

–

252

scored).

Model	ann_A	ann_B	ann_C
InternVL3.5-8B	1.91	2.03	2.16
Qwen3.5-9B	2.00	2.08	2.33
GLM-4.1V-9B-Thinking	2.03	2.36	2.47
GPT-5.5	2.43	2.49	2.65
Gemini 3.1 Pro	2.78	2.71	2.79

Table 7: Task 2 main scalar evaluation (64 unique context-present memes). Mean of five Likert dimensions (1–3) per annotator. Higher is better; relative preferences are derived separately from holistic rankings.

Model	full_context	image_only	$\Delta$
InternVL3.5-8B	2.03	1.87	+0.16
Qwen3.5-9B	2.09	2.09	+0.00
GLM-4.1V-9B-Thinking	2.38	2.37	+0.01
GPT-5.5	2.52	2.49	+0.03
Gemini 3.1 Pro	2.77	2.69	+0.08
Overall	2.36	2.30	+0.06

Table 8: Task 2 context ablation (ABC combined; 31 matched pairs from the curated Task 2 evaluation pool;

n{=}93

ratings per model–condition cell and

n{=}465

for the overall aggregate). Core4

=

mean of Visual, Panel, Humor, and Faithfulness (excludes Context).

Figure 3: Illustrative Task 1 order-recovery errors under shuffled display. Numbers are shuffled display indices; each subfigure contrasts gold and observed orders.

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memes
Jul 30, 2026Weihang Wang, Kainan Tu, Jielei Zhang +9VLM Evaluation Vision-Language Models
1Bilibili · 2Fudan University · 3Peking University
MMGist: A Comprehensive Multimodal Benchmark for 2027
Jun 21, 2026Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong +3VLM Evaluation Large Vision-Language Models
1Shanghai Jiao Tong University · 2Sichuan University
MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
Sep 1, 2026Tawsif Tashwar Dipto, Mehedi Ahamed, Radib Bin Kabir +5VLM Evaluation Multilingual VLM Evaluation
1Islamic University of Technology · 2South East University · 3Vector Institute +3

Jul 30, 2026J/K move · Enter open · S save

MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memes

Weihang Wang, Kainan Tu, Jielei Zhang, Run Yang+8

1Bilibili · 2Fudan University · 3Peking University

VLM Evaluation Vision-Language Models VLM Reasoning Cultural Knowledge in Language Models

Open paper AskPDF arXiv

Large vision-language models have improved at describing visual content, but accurate descriptions do not ensure interpretation when meaning depends on knowledge beyond the pixels. Memes expose this gap because they rely on cultural entities, background knowledge, and community conventions. Most meme benchmarks reduce interpretation to labels or holistic scores, obscuring where an explanation breaks down. We introduce MemeBench, a diagnostic benchmark of 1,253 Chinese and English memes with human-written references and quality-controlled VIKR annotations, centered on anime, comics, games, and adjacent online subcultures. Its VIKR schema decomposes explanations into Visual clues, Identity links, Knowledge units, and Reasoning mechanisms. Across 26 LVLMs, every model covers visible content more reliably than the knowledge needed to interpret it, and even the strongest retains a 22.6% Visual-Knowledge gap. To test whether this diagnosis can guide improvement, we introduce KAR, an entity-guided retrieval baseline built on CultureBase. Across four controlled models, KAR raises VIKR Success by 3.6-7.4% and, compared with generic retrieval, repairs more answers and breaks fewer. Yet both retrieval conditions improve Identity and Knowledge while reducing Visual coverage in every comparison. MemeBench reveals whether an interpretation succeeds, what is missing, and whether targeted evidence fills the diagnosed gap.

Abstract

Multimodal Reasoning

Explore similar work

MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memes

MMGist: A Comprehensive Multimodal Benchmark for 2027

MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models