MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

Ziqiao Shang; Ling-Yue Ge; Zian Xu; Zi-Jian Cheng; Shi-Yu Tian; Zhenyu Huang; Wenbo Fu; Weiming Wu

cs.LGFeb 20, 2026

MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

Authors: Ziqiao Shang, Ling-Yue Ge, Zian Xu, Zi-Jian Cheng, Shi-Yu Tian, Zhenyu Huang, Wenbo Fu, Weiming Wu,

Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · AMAP, Alibaba Group · School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China

Abstract

Systematically evaluating Multimodal Large Language Models (MLLMs) is essential for advancing Artificial General Intelligence (AGI). Yet existing benchmarks remain inadequate for rigorously measuring their reasoning capabilities under multi-criteria constraints. To address this gap, we introduce MapTab, a multimodal benchmark designed to assess holistic multi-criteria reasoning in MLLMs through route-planning tasks. MapTab requires models to perceive and ground visual information from map images while integrating route attributes, such as Time and Price, from structured tables. It covers two scenarios: Metromap, spanning metro networks in 160 cities across 52 countries, and Travelmap, featuring 168 representative tourist attractions from 19 countries. Overall, MapTab includes 328 images, 196,800 route-planning queries, and 3,936 QA queries, incorporating four key criteria: Time, Price, Comfort, and Reliability. Extensive evaluations of 21 representative MLLMs show that current models still struggle with multicriteria multimodal reasoning. Notably, when visual perception is unreliable, multimodal reasoning can even underperform unimodal approaches. MapTab therefore offers a challenging and realistic testbed for systematically evaluating and advancing MLLMs across core perception, integration, numerical comparison, and route planning capabilities.

Figures & tables

Figure 2: Schematic overview of the MapTab construction pipeline, comprising 5 main steps: Image Collection & Preprocessing, Tabular Construction, Quality Control, Query Generation, and Label Annotation.

Capability	Definition	Metromap Problems	Travelmap Problems
Visual Perception	Identifying lines, stations, locations, and topological structures from map images.	M-GP, M-LP, M-SR	M-GP, M-LP, M-SR
Table Understanding	Retrieving and interpreting information and attributes from Vertex and Edge tables.	E-GP, E-LP, E-SR, V-GP, V-LP, V-SR	E-GP, E-LP, E-SR, V-GP, V-LP, V-SR
Cross-modal Alignment	Matching information across maps and tables and checking whether they are consistent.	MM-GP, MM-LP, MM-SR	MM-GP, MM-LP, MM-SR
Graph Topology Reasoning	Reasoning about graph structures, adjacency, connectivity, routes, and transfer relationships.	M-LP, E-SR, MM-SR	M-SR, E-SR
Spatial Localization	Determining locations, line membership, same-line relationships, and relative positions.	M-SR, V-LP, V-SR	M-LP, V-SR
Numerical Reasoning	Performing counting, comparison, aggregation, averaging, and other numerical operations.	M-GP, M-LP, E-GP, E-SR, V-GP, V-LP, MM-GP, MM-LP	M-GP, E-GP, V-GP, V-LP, MM-SR

Table 1: Capability taxonomy and corresponding problem types in the proposed MapTab benchmark. Each problem may involve multiple capabilities. M, E, V, and MM denote Map, Edge_tab, Vertex_tab, and Map+Mix_tab, respectively.

Model	Type	Metromap						Travelmap
		Map-only			Map+Mix_tab			Map-only			Map+Mix_tab
		EMA	PMA	DS	EMA	PMA	DS	EMA	PMA	DS	EMA	PMA	DS
Human	–	85.1	–	–	62.2	–	–	91.7	–	–	72.8	–	–
Open-source Models
Qwen3-VL-8B-Instruct	No-Thinking	2.75	17.58	103	4.69	21.87	182	19.29	42.50	1040	15.65	40.97	804
Qwen3-VL-8B-Thinking	Thinking	5.12	20.99	188	6.38	22.93	270	22.62	45.94	4511	12.74	38.10	653

Table 2: Evaluation results of various Multimodal Large Language Models (MLLMs) on the MapTab route-planning task under the Metromap and Travelmap scenarios. EMA, PMA, and DS denote Exact Match Accuracy, Partial Match Accuracy, and Difficulty-aware Score, respectively. Map-only uses only the map image, while Map+Mix_tab uses the map image together with Mix_tab. Bold values indicate the best performance within the open-source and closed-source model groups, respectively.

Open-source Models
Model	Type	Metromap								Travelmap
Model	Type	VP	TU	CA	GT	SL	NR	PP	GR	VP	TU	CA	GT	SL	NR	PP	GR
Qwen3-VL-8B-Instruct	No-Thinking	48.5	54.4	20.6	21.3	70.6	29.4	30.6	19.7	40.1	52.2	53.4	49.1	60.7	25.7	43.3	72.9
Qwen3-VL-8B-Thinking	Thinking	44.2	78.0	17.5	40.2	75.6	46.1	22.5	21.6	54.2	87.9	48.6	46.1	85.4	68.1	40.9	66.1
Qwen3-VL-2B-Instruct	No-Thinking	25.4	32.7	11.7	11.7	46.3	8.2	17.5	13.4	27.0	41.9	45.4	37.2	48.5	14.1	45.6	66.1
Qwen2.5-VL-7B-Instruct	No-Thinking	43.5	53.0	20.0	19.6	71.5	27.7	29.4	17.2	41.5	61.6	39.5	50.9	72.6	32.7	44.1	51.2

Table 3: Capability-level performance of different MLLMs on the MapTab benchmark. Each score is the arithmetic mean accuracy over all problems associated with the corresponding capability. VP, TU, CA, GT, SL, NR, PP, and GR denote Visual Perception, Table Understanding, Cross-modal Alignment, Graph Topology Reasoning, Spatial Localization, Numerical Reasoning, Path Planning, and Global Reasoning, respectively. Bold values indicate the best performance within the open-source and closed-source model groups, respectively.

Metromap
Criteria Settings	FR		PR		NR		All
Criteria Settings	EMA	PMA	EMA	PMA	EMA	PMA	EMA	PMA	T	P	C	R
Time-Only	26.59	46.75	1.92	16.36	1.80	21.34	19.81	39.47	0.00	2.44	2.88	9.69
Price-Only	30.91	51.05	0.00	13.87	3.31	25.94	22.06	42.40	0.00	1.19	4.06	10.44
Comfort_Level-Only	25.31	46.24	0.00	16.20	2.56	23.81	18.25	38.97	0.00	2.44	4.88	10.19
Reliability-Only	25.73	45.57	0.00	13.84	3.59	27.69	19.38	39.53	0.00	1.19	4.06	10.44

Table 4: Ablation studies of route repetition and criteria understanding in the Metromap and Travelmap scenarios. Query types are categorized into Fully Repeat (FR), Partially Repeat (PR), and Not Repeat (NR), according to the deviation between the original shortest paths and the optimal paths after introducing different criteria. Under All, EMA and PMA report the overall route planning performance, while T, P, C, and R denote the understanding accuracy of Total Time, Total Price, Average Comfort Level, and Average Reliability, respectively.

Error Type	Count	Error (%)	Overall (%)
Perception	267	20.7	16.7
Transfer Indeterminate	109	8.4	6.8
Path Search	460	35.6	28.8
Topology Extraction	491	38.0	30.7
Output Format	34	2.6	2.1
Table Lookup	39	3.0	2.4

Table 5: Distribution of different error types of Qwen3-VL-8B on MetroMap. Error (%) denotes the percentage among all error cases, while Overall (%) denotes the percentage among the entire evaluation set. Multiple error labels may be assigned to the same prediction; therefore, percentages across categories do not sum to 100%.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Category / Representative Works	Input	G+T	RP	MC	Tools	Scale	Main Gap
Visual Reasoning V*Bench, VisualPuzzles, VisuLogic, R-Bench, VGRP-Bench	Images	$\times$	$\times$	$\times$	$\times$	Large	Focus on abstract visual and logical reasoning rather than route planning.
General Multimodal Reasoning MathVQA, MMMU, MathVerse	Images + Text	$\times$	$\times$	$\times$	$\times$	Large	Evaluate general multimodal reasoning rather than planning tasks.
Map Understanding MapEval, MapQA, MapIQ, CartoMark, MapBench	Maps	$\times$	Limited	$\times$	$\times$	Medium	Focus on map perception, OCR, and QA rather than constrained route planning.
Spatial Reasoning CityBench, DriveBench, FRIEDA, GeoX-Bench, CartoMapQA, MapLM	Maps	Partial	Partial	$\times$	$\times$	Medium	Evaluate spatial reasoning without explicit multi-criteria optimization.
Planning / Navigation PlanAgent, PReP, GeoNav, NavBench, GeoBenchX, ReasonMap, RewardMap, TraveLLM	Maps / Environment	Partial	✓	Partial	✓	Small–Medium	Focus on navigation, agent planning, or embodied execution, often with external tools.
MapTab (Ours)	Map + Table	✓	✓	✓	$\times$	Large	The first benchmark that evaluates native multimodal route planning over map images and structured tables under explicit multi-criteria constraints, requiring models to solve the task without external tools or tool-use assumptions.

Appendix

Table 6: Comparison of MapTab with existing multimodal reasoning and planning benchmarks. G+T , RP , and MC denote Graph+Table, Route Planning, and Multi-Criteria, respectively.

Model	Type	Map-only			E_tab-only			Map+E_tab			Map+E_tab+V_tab			Map+Mix_tab
		EMA	PMA	DS	EMA	PMA	DS	EMA	PMA	DS	EMA	PMA	DS	EMA	PMA	DS
Scenario: Metromap
Open-source Models
Qwen3-VL-8B-Instruct	No-Thinking	2.75	17.58	103	25.69	46.44	1153	21.25	41.30	921	19.31	39.31	855	4.69	21.87	182
Qwen3-VL-8B-Thinking	Thinking	5.12	20.99	188	31.69	49.76	1427	38.00	57.06	1771	23.75	41.69	1080	6.38	22.93	270
Qwen3-VL-2B-Instruct	No-Thinking	0.94	15.14	35	9.88	27.61	437	6.63	23.85	282	7.00	26.91	325	2.00	17.82	78

Appendix

Table 7: Evaluation results of various Multimodal Large Language Models (MLLMs) on the MapTab route planning task under the Metromap and Travelmap scenarios. EMA, PMA, and DS denote Exact Match Accuracy, Partial Match Accuracy, and Difficulty-aware Score, respectively. Map-only: map information only; Edge_tab-only: edge data only; Map+Edge_tab: map and edge data; Map+Edge_tab+Vertex_tab: map, edge, and vertex data; Map+Mix_tab: map and Mix_tab. We did not include Edge_tab+Vertex_tab because the comparison between it and Map+Mix_tab yielded conclusions consistent with those from the Map-only and Edge_tab-only control groups, without new findings. Bold values represent the best performance within open-source and closed-source groups, respectively.

Scenario: Metromap
Model	Type	Map			Edge_tab			Vertex_tab			Map+Mix_tab
Model	Type	GP	LP	SR	GP	LP	SR	GP	LP	SR	GP	LP	SR
Open-source Models
Qwen3-VL-8B-Instruct	No-Thinking	55.00	17.50	73.12	22.50	100.0	7.50	57.50	51.88	86.88	0.63	22.50	38.75
Qwen3-VL-8B-Thinking	Thinking	53.12	28.12	51.25	56.87	99.38	56.87	79.37	77.50	98.12	7.50	9.38	35.63
Qwen3-VL-2B-Instruct	No-Thinking	8.13	5.00	63.12	3.75	87.50	3.12	26.25	11.25	64.38	0.00	8.13	26.87

Appendix

Table 8: Performance of QA tasks across multiple MLLMs in the Metromap and Travelmap scenarios. In the Metromap scenario, the Mix_tab paired with the Map input has the Line column removed to prevent excessive table information from affecting the evaluation of map-table coordination. Task types are categorized into three classes: Global Perception-based Reasoning Tasks (GP), Local Perception-based Reasoning Tasks (LP), and Spatial Relationship Judgment Tasks (SR). Bold values in the table indicate the best performance among open-source and closed-source models, respectively.

Panel of Figure 3 — Figure 3: Impact of Image Resolution on RP and QA Tasks in Metromap and Travelmap Scenarios. Images are proportionally downsampled to $1/2$ , $1/4$ , $1/8$ , $1/16$ , and $1/32$ of the original resolution. For RP task, evaluation is conducted under 4 Map-based input settings: Map-Only (M), Map+Edge_tab (M+E_tab), Map+Edge_tab+Vertex_tab (M+E_tab+V_tab), and Map+Mix_tab (M+M_tab). For QA task, evaluation is conducted under 6 Map-based input settings: Map-GP (M-GP), Map-LP (M-LP), Map-SR (M-SR), Map+Mix_tab-GP (M+M-GP), Map+Mix_tab-LP (M+M-LP), and Map+Mix_tab-SR (M+M-SR).

Panel of Figure 4 — Figure 4: Distribution of model accuracy across Map Difficulty and Query Difficulty under different input modalities (Metromap and Travelmap scenarios).

Panel of Figure 5 — Figure 5: Model accuracy matrix under different combinations of Map Difficulty and Query Difficulty. The first row shows results in the Metromap scenario, while the second row corresponds to the Travelmap scenario. M denotes Map Only; E denotes Edge_Tab Only; M+E denotes Map + Edge_Tab; M+E+V denotes Map + Edge_Tab + Vertex_Tab; and M+M denotes Map + Mix_tab.

	Token Statistics			Map+E_tab+V_tab			Map+M_tab
Input format	E_tab	V_tab	M_tab	EMA	PMA	DS	EMA	PMA	DS
Metromap
csv	2651	2264	2496	19.75	40.16	876	6.31	24.48	255
json	6465	6193	6397	19.31	39.31	855	4.69	21.87	182
Travelmap
csv	577	462	462	33.39	55.10	1929	15.42	40.83	800

Appendix

Table 9: Token counts under CSV and JSON table formats and their performance comparison on RP queries.

Input format	E-GP	E-LP	E-SR	V-GP	V-LP	V-SR	M+M-GP	M+M-LP	M+M-SR
	(Acc)	(Acc)	(Acc)	(Acc)	(Acc)	(Acc)	(Acc)	(Acc)	(Acc)
Metromap
csv	20.00	100	3.12	47.50	58.75	78.75	1.25	21.25	70.00
json	22.50	100	7.50	57.50	51.88	86.88	0.63	22.50	38.75
Travelmap
csv	19.64	99.40	44.05	25.00	47.02	54.17	76.19	72.02	16.07

Appendix

Table 10: Performance comparison of CSV and JSON table formats on QA queries.

Model and Criteria Settings	Chinese-Chinese		English-English		Others-Others		Others-Chinese		Others-English		English-Others
Model and Criteria Settings	EMA	PMA	EMA	PMA	EMA	PMA	EMA	PMA	EMA	PMA	EMA	PMA
Qwen3-map	12.10	31.65	6.03	22.85	2.64	19.47	18.31	38.81	17.86	43.18	0	13.17
Qwen3-map+e_tab	30.78	47.77	31.03	56.41	29.92	53.61	41.36	60.79	57.14	71.25	4.17	33.70
Qwen3-map+e_tab+v_tab	26.29	44.86	32.76	52.13	23.80	48.53	31.36	54.22	60.71	76.29	4.17	29.93
Qwen3-map+m_tab	10.59	31.30	7.76	27.07	5.12	26.07	15.25	39.19	17.86	48.41	0	18.23
GPT4o-map	11.42	31.23	15.52	34.51	8.26	33.63	15.42	39.05	28.57	59.34	12.50	37.30

Appendix

Table 11: Experimental performance results under different language distributions in the Metromap and Travelmap scenarios.

Panel of Figure 6 — Figure 6: All images have the exact same height. Widths are adjusted automatically.

Error Type	Diagnostic Criterion	Capability Gap
Perception	The model fails to correctly recognize, localize, or associate task-critical visual entities (e.g., source and destination stations), resulting in incorrect inputs for subsequent route reasoning.	Visual Perception; Cross-modal Alignment
Transfer Indeterminate	The predicted transfer annotations are inconsistent with the actual line-switching status along the route.	Graph Topology Reasoning
Path Search	The model correctly identifies the optimization objective but fails to search for the optimal route, resulting in a path that deviates from the ground-truth solution.	Path Planning; Global Reasoning; Numerical Reasoning
Topology Extraction	The recognized stations are correct, but the predicted route contains invalid graph connections or violates the underlying map topology.	Graph Topology Reasoning; Spatial Localization
Output Format	The final response violates the required output specification, either because the model misunderstands the output requirements or because excessive reasoning prevents a valid final response.	Global Reasoning
Table Lookup	The model fails to correctly interpret or retrieve the required information from mix_tab , leading to an incorrect understanding of the tabular content.	Table Understanding

Appendix

Table 12: Diagnostic criteria and corresponding capability gaps for different error types. The capability taxonomy includes Visual Perception, Table Understanding, Cross-modal Alignment, Graph Topology Reasoning, Spatial Localization, Numerical Reasoning, Path Planning, and Global Reasoning.

Figure 7: Error Case 1: Perception Error

Figure 8: Error Case 2: Transfer Indeterminate Error

Figure 9: Error Case 3: Path Search Error

Figure 10: Error Case 4: Topology Extraction Error

Figure 11: Error Case 5: Output Format Error

Figure 12: Error Case 6: Table Lookup Error

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models
Mar 19, 2026Tianlong Wang, Pinqiao Wang, Weili Shi +1LLM Evaluation LLM Planning
University of Virginia
LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?
May 11, 2026Xueqi Cheng, Yushun DongEfficient Multimodal Inference Multimodal Large Language Models
Department of Computer Science Florida State University University
MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
Mar 2, 2026Jiachun Li, Shaoping Huang, Zhuoran Jin +5Multi-Image Reasoning Multimodal Reasoning
School of Artificial Intelligence, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences

Mar 19, 2026J/K move · Enter open · S save

ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models

Tianlong Wang, Pinqiao Wang, Weili Shi, Sheng li

University of Virginia

LLM Evaluation LLM Planning Spatial Reasoning Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) with advanced cognitive capabilities are emerging as agents for various reasoning and planning tasks. Traditional evaluations often focus on specific reasoning or planning questions within controlled environments. Recent studies have explored travel planning as a medium to integrate various verbal reasoning tasks into real-world contexts. However, reasoning tasks extend beyond verbal reasoning alone, and a comprehensive evaluation of LLMs requires a testbed that incorporates tasks from multiple cognitive domains. To address this gap, we introduce ItinBench, a benchmark that features one task of spatial reasoning, i.e., route optimization, into trip itinerary planning while keeping the traditional verbal reasoning tasks. ItinBench evaluates various LLMs across diverse tasks simultaneously, including Llama 3.1 8B, Mistral Large, Gemini 1.5 Pro, and GPT family. Our findings reveal that LLMs struggle to maintain high and consistent performance when concurrently handling multiple cognitive dimensions. By incorporating tasks from distinct human-level cognitive domains, ItinBench provides new insights into building more comprehensive reasoning testbeds that better reflect real-world challenges. The code and dataset: https://ethanwtl.github.io/IBweb/

Abstract

Multimodal Reasoning

Explore similar work

ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models

LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?

MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning