Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions

Aditya K Surikuchi; Raquel Fernández; Sandro Pezzelle

cs.CLFeb 18, 2025

Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions

Authors: Aditya K Surikuchi, Raquel Fernández, Sandro Pezzelle

Organizations: Institute for Logic, Language and Computation (ILLC), University of Amsterdam, Netherlands

Abstract

In recent years, a substantial body of work in visually grounded natural language processing has focused on real-life multimodal scenarios such as describing content depicted in images or videos. However, comparatively less attention has been devoted to studying the nature and degree of interaction between the different modalities in these scenarios. In this paper, we argue that any task dealing with natural language generation from sequences of images or frames is an instance of the broader, more general problem of modeling the intricate relationships between visual events unfolding over time and the features of the language used to interpret, describe, or narrate them. Therefore, solving these tasks requires models to be capable of identifying and managing such intricacies. We consider five seemingly different tasks, which we argue are compelling instances of this broader multimodal problem. Subsequently, we survey representative modeling and evaluation approaches adopted for these tasks in recent years and examine the common set of challenges these tasks pose. Building on this perspective, we identify key open questions and propose several research directions for future investigation.

Figures & tables

Panel of Figure 1 — Figure 1. Five tasks exemplifying the problem of text generation from visual events unfolding over time. For each task, we denote the visual input data type—sequences of images or video frames—using and respectively. In all cases, the output is natural language text. On the left, this figure shows a sequence of images or video frames positioned side-by-side. On the right, text corresponding to the visual inputs is provided.

Dataset	Visual Input	Size	Domain	Output Text
Change Captioning
Spot-the-Diff ( Jhamtani and Berg-Kirkpatrick, 2018 )	Image pairs (video frames)	13,192 pairs	Surveillance	Difference descriptions
CLEVR-Change ( Park et al., 2019 )	Image pairs	79,606 pairs; 493,735 captions	Synthetic 3D scenes	Generated change captions
LEVIR-CC ( Liu et al., 2022 )	Image pairs	10,077 pairs; 50,385 sentences	Remote sensing	Change descriptions
Video QA
MSVD-QA ( Xu et al., 2017 )	Videos (avg. 10 s)	1,970 videos; 50,505 QA pairs	Open domain	Generated one-word answers

Table 1. A selection of datasets commonly used for the tasks considered in this paper, in chronological order within each task. For each dataset, we report the type of visual input, its size, the domain of the visual data, and the type of output text (AD: audio description).

Panel of Figure 2 — Figure 2. Similarity scores obtained for the tasks along the visual and textual dimensions. We excluded Video QA from the textual analysis due to the lack of multi-sentence datasets.

Figure 3. Relative positioning of the tasks in the visual-textual shared space, based on the similarity scores obtained for the corresponding datasets. To illustrate, a task in the top right corner of the plot is both highly consistent at the textual level and highly similar at the visual level. Since no Video QA dataset contains multi-sentence answers, we arbitrarily set its average textual consistency value to

1

to allow visualization.

Model	Vision Encoder	Projector	Language Decoder
Change Captioning
CARD ( Tu et al., 2024 )	ResNet, Transformer †	NA	Transformer †
SCORER+CBR ( Tu et al., 2023b )	ResNet, MH(S/X)A †	NA	Transformer †
VARD-Trans ( Tu et al., 2023a )	ResNet, Linear(s) †	NA	Transformer †
DUDA ( Park et al., 2019 )	ResNet, RNN †	NA	RNN †
Video QA

Table 2. A selection of recent models proposed for the tasks involving visual events considered in this paper. We only report models that are end-to-end and task-specific, i.e., trained or fine-tuned for the task at hand. For each model, we report the underlying vision encoder and language decoder, as well as the projector module, when applicable ( NA : Not Applicable). Components with † have been trained from scratch using only the datasets available for the corresponding task.

Figure 4. Outline of the architecture common across modeling approaches for tasks involving language generation from visual events.

Model	Vision Encoder	Projector	Language Decoder
Qwen2.5-VL ( Bai et al., 2025 )	ViT	Linear	Qwen2.5
DeepSeek-VL ( Lu et al., 2024 )	SigLIP, SAM-B	Linear (#2)	DeepSeek
LLaVA-NeXT ( Liu et al., 2023 )	CLIP-ViT-L	Linear	Mistral
Video-LLaMA ( Zhang et al., 2023 )	ViT-G/14	Q-Former	Llama
mPLUG-Owl3 ( Ye et al., 2025 )	SigLIP SO (400M)	Linear	Qwen2
Video-LLaVA ( Lin et al., 2024 )	OpenCLIP ViT-L	Linear (#2)	Vicuna v1.5

Table 3. A selection of pre-trained multi-purpose VLMs that are used off-the-shelf for various NLG tasks involving visual events.

Visual Storytelling

See all 23 papers

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation
Dec 16, 2025Zefan Cai, Haoyi Qiu, Tianyi Ma +13Embodied Navigation Multimodal Reasoning
University of Wisconsin–Madison · University of California, Los Angeles · Michigan State University +8
Lance: Unified Multimodal Modeling by Multi-Task Synergy
May 18, 2026Fengyi Fu, Mengqi Huang, Shaojin Wu +10Unified Multimodal Models Multi-Task Learning
Intelligent Creation Lab, ByteDance
Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm
May 12, 2026Yaofang Liu, Kangning Cui, Meng Chu +7Vision-Language Models Conditional Image Generation
City University of Hong Kong · City University of Hong Kong (Dongguan) · The Hong Kong University of Science and Technology +4

Dec 16, 2025J/K move · Enter open · S save

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

Zefan Cai, Haoyi Qiu, Tianyi Ma, Haozhe Zhao+12

University of Wisconsin–Madison · University of California, Los Angeles · Michigan State University+8

Embodied Navigation Multimodal Reasoning Multimodal Model Evaluation Video Generation Evaluation

Open paper AskPDF arXiv

Modern multimodal generative models can synthesize visually compelling images and videos, but it remains unclear whether this visual fluency reflects genuine reasoning: when prompted to generate a solution, can a model preserve the physical, logical, spatial, and temporal constraints a task requires, or does it merely produce plausible-looking media? To answer this question, we introduce MMGR (Multi-Modal Generative Reasoning Benchmark and Evaluation), a benchmark for evaluating generative reasoning across video, image, and language-based systems. MMGR covers 10 tasks from three domains (Abstract Reasoning, Embodied Navigation, and Physical Commonsense) and probes five reasoning abilities: Physical, Logical, 2D Spatial, 3D Spatial, and Temporal. Its evaluation emphasizes answer-verifiable tasks and, for video generation, process-aware chain-of-frame reasoning, where intermediate frames must form valid steps toward the target outcome rather than visually smooth but incorrect transitions. Evaluating state-of-the-art video generators, image generators, and LLM/VLM baselines reveals a sharp gap between visual quality and reasoning correctness: video models perform best on Physical Commonsense, but remain weak on symbolic tasks such as Sudoku, ARC, and Math, and brittle in cross-view embodied navigation. Image generators often outperform video generators on embodied navigation despite lacking temporal outputs, showing that longer visual generation does not automatically yield stronger reasoning. MMGR reframes evaluation of multimodal generation from whether outputs look realistic to whether they solve the underlying reasoning problem.

Abstract

Visual Storytelling

Explore similar work

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

Lance: Unified Multimodal Modeling by Multi-Task Synergy

Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm