MATEO: A Multimodal Benchmark for Temporal Reasoning and Planning in LVLMs

Gabriel Roccabruna; Olha Khomyn; Giuseppe Riccardi

cs.AIFeb 16, 2026

MATEO: A Multimodal Benchmark for Temporal Reasoning and Planning in LVLMs

Authors: Gabriel Roccabruna, Olha Khomyn, Giuseppe Riccardi

Organizations: Signals and Interactive Systems Lab, University of Trento, Italy

Abstract

AI agents need to plan to achieve complex goals that involve orchestrating perception, sub-goal decomposition, and execution. These plans consist of ordered steps structured according to a Temporal Execution Order (TEO, a directed acyclic graph that ensures each step executes only after its preconditions are satisfied. Existing research on foundational models' understanding of temporal execution is limited to automatically derived annotations, approximations of the TEO as a linear chain, or text-only inputs. To address this gap, we introduce MATEO (MultimodAl Temporal Execution Order), a benchmark designed to assess and improve the temporal reasoning abilities of Large Vision Language Models (LVLMs) required for real-world planning. We acquire a high-quality professional multimodal recipe corpus, authored through a standardized editorial process that decomposes instructions into discrete steps, each paired with corresponding images. We collect TEO annotations as graphs by designing and using a scalable crowdsourcing pipeline. Using MATEO, we evaluate six state-of-the-art LVLMs across model scales, varying language context, multimodal input structure, and fine-tuning strategies.

Figures & tables

Figure 1: Example of an AI agent’s planning process and inherent uncertainties for a natural-language goal. Questions A–D decompose the high-level goal into executable actions, while E infers their Temporal Execution Order (TEO) as a directed acyclic graph. Each step introduces uncertainty, producing multiple possible paths, some correct, others wrong.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt Setting	Modality	Prompt
Baseline	Image-only	Using ONLY the information in the Context, answer the following three questions in EXACTLY this format:
		Q1: The answer is: <Yes/No/I don’t know>.
		Q2: The answer is: <Yes/No/I don’t know>.
		Q3: The answer is: <Yes/No/I don’t know>.
		Do not add anything else. Do not explain. Do not change the format.
		Context:

Appendix

Table 5: Prompt schema used for models evaluation under image-only, text-only, and image–text modalities. Repeated or previously described components are omitted for brevity.

Modality	Prompt
Image-only	<User> Your task is to determine the dependency order between two steps in a recipe. Follow these rules:
Image-only	- Before : Step A must be executed before Step B if the outcome of Step A is required to complete Step B (i.e., Step B depends on Step A).
	- After : Step A must be executed after Step B if the outcome of Step B is required to complete Step A (i.e., Step A depends on Step B).
	- Parallel : Step A and Step B can be executed in parallel if neither step depends on the outcome of the other; therefore, their order of execution can be arbitrary.
	Step A picture: [image_A]
	Step B picture: [image_B]

Appendix

Table 6: Example of prompt–response pairs used for fine-tuning across image-only, text-only, and image–text modalities. Each example includes instructions, input structure, and expected binary outputs. The <User> and <Assistant> tokens are model-specific and are included here solely for illustrative purposes.

Figure 2: This figure shows the first two points of the annotation guidelines, that present the purpose and the general description of the task.

Figure 3: This section of the annotation guidelines offers annotators the option to watch a demonstration video that walks through the task and explains how to identify dependencies between steps. Following the video, annotators are provided with a textual description that formally defines each dependency, accompanied by illustrative examples.

Figure 4: This figure continues from the previous one, illustrating a case in which a step depends on another (Example 1.1), and presenting the formal definition of independent steps

Figure 5: This figure shows an example given to illustrate the case in which steps are independent.

Figure 6: Example illustrating a fully annotated DAG for a recipe, with all relations clearly explained.

Figure 7: Illustration of the annotation tool as presented in the annotation guidelines. Highlighted regions (marked in red letters) indicate key functionalities, including dragging steps, adding or deleting connections, and saving the annotations.

Figure 8: Example of a DAG annotated by a crowd-worker. Steps 1, 2, 3, 4, and 5 can be executed in any order (or simultaneously), thus there is no directed edge between them.

Temporal Reasoning in VLMs

See all 23 papers

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
Oct 13, 2026Juntong Feng, Qingan Liu, Jingru Dong +8VLM Reasoning VLMs for Robotics
Dalian University of Technology · Nanyang Technological University
Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Jun 3, 2026Haoyu Sun, Wenxuan Wang, Mingyang Song +5Tool-Augmented Language Model Agents LLM Agent Evaluation
Tongji University · Shanghai AI Laboratory · Harbin Institute of Technology +5
ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models
Mar 19, 2026Tianlong Wang, Pinqiao Wang, Weili Shi +1LLM Evaluation LLM Planning
University of Virginia

Oct 13, 2026J/K move · Enter open · S save

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

Juntong Feng, Qingan Liu, Jingru Dong, Yue Zhu+7

Dalian University of Technology · Nanyang Technological University

VLM Reasoning VLMs for Robotics Embodied Reasoning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have achieved strong performance in visual perception and spatial reasoning, leading to growing interest in using them as high-level planners for embodied AI. However, physical tasks require models to integrate spatial information, reason over temporal states, and generate multi-step plans under various constraints, raising the question of whether current VLMs can reliably support such reasoning and planning independently of execution. We introduce Plan-Bench, a benchmark for evaluating the high-level reasoning and multi-step planning capabilities of VLMs relevant to embodied worlds, while isolating the influence of action execution. It covers three complementary dimensions, spanning 16 subclasses and 699 curated samples. Extensive evaluation of 28 VLMs reveals room for improvement, with even the strongest model achieving only 75.0% accuracy. We further find that model scaling generally improves these capabilities. Ablation studies also indicate larger models show increasing potential as high-level planners. Overall, Plan-Bench provides a lightweight probe for evaluating VLMs' reasoning and planning capabilities relevant to embodied environments and assessing their potential as decision makers for physical tasks. Our benchmark and code are available at https://github.com/MinneFeng/Plan-Bench.

Abstract

Temporal Reasoning in VLMs

Explore similar work

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models