ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models

Tianlong Wang; Pinqiao Wang; Weili Shi; Sheng li

cs.AIMar 19, 2026

ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models

Authors: Tianlong Wang, Pinqiao Wang, Weili Shi, Sheng li

Organizations: University of Virginia

Abstract

Large language models (LLMs) with advanced cognitive capabilities are emerging as agents for various reasoning and planning tasks. Traditional evaluations often focus on specific reasoning or planning questions within controlled environments. Recent studies have explored travel planning as a medium to integrate various verbal reasoning tasks into real-world contexts. However, reasoning tasks extend beyond verbal reasoning alone, and a comprehensive evaluation of LLMs requires a testbed that incorporates tasks from multiple cognitive domains. To address this gap, we introduce ItinBench, a benchmark that features one task of spatial reasoning, i.e., route optimization, into trip itinerary planning while keeping the traditional verbal reasoning tasks. ItinBench evaluates various LLMs across diverse tasks simultaneously, including Llama 3.1 8B, Mistral Large, Gemini 1.5 Pro, and GPT family. Our findings reveal that LLMs struggle to maintain high and consistent performance when concurrently handling multiple cognitive dimensions. By incorporating tasks from distinct human-level cognitive domains, ItinBench provides new insights into building more comprehensive reasoning testbeds that better reflect real-world challenges. The code and dataset: https://ethanwtl.github.io/IBweb/

Figures & tables

Figure 1: An overview of ItinBench. One of the four tasks, “Tool Use with Route Optimization,” is chosen in this figure. The database with additional extracted information from user reviews, the human query, and a list of tools are integrated into a task-specific prompt. LLMs need to utilize their verbal and spatial reasoning ability to plan a trip itinerary based on the task constructed. The verbal and spatial reasoning aspects are evaluated to assess LLMs’ ability to simultaneously address tasks from multiple cognitive dimensions.

	Ours	TP	ITINERA	USC
Preference	✓	✓	✓	✓
Full Open Source	✓	✓
Full Real Data	✓		✓	✓
Day-Wise RO	✓		✓
Plan-Wise RO	✓
User Review	✓

Table 1: Downstream tasks comparison between ItinBench (ours), TravelPlanner (TP) ( xie2024travelplanner ) , ITINERA ( tang-etal-2024-itinera ) , and UnSatChristmas (USC) ( SMTUnSatChristmas ) . RO stands for Route Optimization. ItinBench is the only benchmark that covers both verbal and reasoning tasks and evaluations for LLMs.

	Base	Reviews	Review Attributes
Restaurants	500	49,972	cuisine, flavor, freshness, service, environment, value
Hotels	105	4,804	quality, location, service, safety
Attractions	322	10,146	family, history, activity, nature, food, shopping

Table 2: Entry number for base data and their user review records. Review attributes refer to the columns in the final dataset extracted from the user reviews.

Preference	Choices	Count	Probability
Day	“2 days”, "3 days", "4 days"	1	Equal
Price	"cheap budget", "moderate budget", "expensive budget"	1	Equal
Attraction Orientation	"family oriented", "history oriented", "activity oriented", "nature oriented", "food oriented", "shopping oriented"	1	Equal
Restaurant Related	"good flavor", "good freshness", "good service", "good environment", "good value"	[1, 2, 3]	[0.6, 0.3, 0.1]
Cuisine	"US", "Mexican", "Irish", "French", "Italian", "Greek", "Indian", "Chinese", "Japanese", "Korean", "Vietnamese", "Thai", "Asian Fusion", "Middle Eastern"	1	Equal
Hotel Related	"good quality", "good location", "good service", "good safety"	[1, 2, 3]	[0.6, 0.3, 0.1]

Table 3: Preference selection details for query construction. There are six main categories, each with their option lists. Besides restaurants and hotels selecting 1 to 3 preferences with the probability weights

[0.6,0.3,0.1]

, all other categories choose one preference.

	Verbal Reasoning					Spatial Reasoning
	OOP $\downarrow$	MI $\downarrow$	Micro $\uparrow$	Macro $\uparrow$	VR $\uparrow$	ARG $\downarrow$	DG $\downarrow$	Total-DG $\downarrow$	ECJ $\downarrow$
Greedy Approach	0.0	0.0	100.0	100.0	100.0	0 (4.00)	4.0	9.2	86.2
Task 1: Entire Dataset , No Request For Route Optimization (#100)
Llama 3.1 8B	45.0	11.0	60.1	0.0	0.0	24.3 (3.03)	9.2	24.6	99.2
Mistral-large (123B)	52.0	0.0	66.9	2.0	2.0	1.2 (3.95)	6.5	27.9	113.1
Gemini-1.5-Pro	18.0	52.0	77.0	5.0	5.0	13 (3.48)	7.7	25.2	124.3

Table 4: In percentage, the evaluation of LLM’s trip itinerary generation in different tasks is presented, with the best results highlighted in bold. The validated plan is around 7% and 65% with or without the filtered data accessible to the LLMs. When ask the LLM to perform the route optimization, the total distance gap is still around 20% for older models and 7% for newer models like o1 with and without the access to the spatial clustering information. Gemini’s spatial reasoning task result is "-" since it failed to provide a rational number of attractions.

Panel of Figure 2 — Figure 2: Visualizations of the main results for Validated Rate (VR) and Total Distance Gap (Total-DG). Task 1 and Task 2 (red) do not have access to filtered data. Task 3 and Task 4 (blue) have access to filtered data and spatial clustering information. The second-best result is shown in darker color, and the best result is shown in the darkest color.

	Llama	Mistral	Gemini	GPT
Parameter ACC	58.4	62.6	61.3	62.9
Delivery Rate	84.7	100	72.0	100
Order Dead Loop	48.9	–	0.0	–
Argument DL	51.1	–	100.0	–

Table 5: Tool-use performance for Llama 3.1 8B, Mistral Large, Gemini 1.5 Pro, and GPT-4o. DL denotes dead loop.

Figure 3: Error distribution for GPT-4o across four tasks. Errors primarily occur in out-of-pool, cuisine, restaurant, and hotel-related recommendations.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure B.2: Visualization of a case study about the itinerary generated by GPT 4o in tool-use mode, drawn in Google Map ( googlemap ) . Plan-wise (red markers and routes in Figure 2(c) ), one of the main issues is the route leads to the bottom right corner of the map (Cluster 13) while visiting the same area (Cluster 0) on the second day again. The mistake is corrected in the optimized route in Figure 2(d) . For this itinerary, the total distance gap ratio is 25.6%. Additionally, the extra cluster jump ratio is 100%.

Preference	Attributes
Hotels	"business_id", "name", "address", "latitude", "longitude", "stars", "price"
Restaurants	"business_id", "name", "address", "latitude", "longitude", "stars", "price", "good_for_meal", "cuisine_1", "cuisine_2"
Attraction	"business_id", "name", "address", "latitude", "longitude", "stars", "price"

Appendix

Table C.1: The attributes for basic data

	Base	Avg Reviews	Review Attributes
Restaurants	200	166	cuisine, flavor, freshness, service, environment, value
Hotels	50	92	quality, location, service, safety
Attractions	100	26	family, history, activity, nature, food, shopping

Appendix

Table C.2: Dataset statistic of Santa Barbara

	Task 1		Task 2		Task 3		Task 4
	VR $\uparrow$	Total DG $\downarrow$	VR $\uparrow$	Total DG $\downarrow$	VR $\uparrow$	Total DG $\downarrow$	VR $\uparrow$	Total DG $\downarrow$
LLaMA 3.1 8B	0	114.1	0	107.5	43	110.3	28	125.0
Gemini 1.5 pro	7	125.5	4	-	13	-	17	-
OpenAI o1	15	130.5	5	45.7	42	6.5	41.5	7.0

Appendix

Table C.3: Results for Tasks 1 to 4 (VR and Total DG) for Santa Barbara subset.

	Verbal Reasoning					Spatial Reasoning
	OOP $\downarrow$	MI $\downarrow$	Micro $\uparrow$	Macro $\uparrow$	VR $\uparrow$	ARG $\downarrow$	DG $\downarrow$	Total-DG $\downarrow$	ECJ $\downarrow$
Greedy Approach (#100)
$A^{*}$	0.0	0.0	100.0	100.0	100.0	0 (4.00)	0.0	0.0	0.0
MIP	0.0	0.0	100.0	100.0	100.0	0 (4.00)	0.0	0.0	0.0

Appendix

Table D.4: Alternative baselines; columns correspond to Table 4 in the main text. Except for the greedy + min distance approach, all other approaches achieve perfect performance in evaluation.

Panel of Figure 14 — Figure D.3: Alignments between human evaluation preferences (yellow), validate rate (blue), and total distance gap (green). For better visualization, Total-DG is inverted to see the trend.

	Mistral	GPT4o	Llama	Gemini
Evaluation 1	30.8	35.8	14.2	19.2

Appendix

Table D.5: Preferred rate across the plans generated by four different LLMs. Evaluation 1 refers to the comparison between the plans generated by four LLMs based on the same user query. Evaluation 2 refers to the comparison between a set of plans generated all by OpenAI-o1 with same user query, but within different task setting.

	AllDataNoRoute	AllDataRouteOP	FilteredData	ToolUse
Evaluation 2	20.5	19.8	33.2	30.5

Appendix

Table D.6: Preferred rate across the plans generated by four different LLMs. Evaluation 1 refers to the comparison between the plans generated by four LLMs based on the same user query. Evaluation 2 refers to the comparison between a set of plans generated all by OpenAI-o1 with same user query, but within different task setting.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

Revisiting the Travel Planning Capabilities of Large Language Models
May 5, 2026Bo-Wen Zhang, Jin Ye, Peng-Yu Hua +4LLM Self-Correction LLM Evaluation
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Trip+: Benchmarking Agents in Personalized Interactive Travel Planning
Jun 19, 2026Junle Chen, Wei Chen, Yehong Xu +6LLM Agent Evaluation AI Agent Benchmarks
HKUST · Tencent Hy · HKUST(GZ)
TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning Agents
May 31, 2026Weiyi Chen, Shuaixiong Wang, Ziyun Gao +5Spatiotemporal Reasoning LLM Agent Evaluation
Zhejiang University · Hangzhou, China · Hong Kong Polytechnic University +5

May 5, 2026J/K move · Enter open · S save

Revisiting the Travel Planning Capabilities of Large Language Models

Bo-Wen Zhang, Jin Ye, Peng-Yu Hua, Jia-Wei Cao+3

State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

LLM Self-Correction LLM Evaluation LLM Planning Travel Planning

Open paper AskPDF arXiv

Travel planning serves as a critical task for long-horizon reasoning, exposing significant deficits in LLMs. However, existing benchmarks and evaluations primarily assess final plans in an end-to-end manner, which lacks interpretability and makes it difficult to analyze the root causes of failures. To bridge this gap, we decompose travel planning into five constituent atomic sub-capabilities, including Constraint Extraction, Tool Use, Plan Generation, Error Identification, and Error Correction. We implement a decoupled evaluation protocol leveraging oracle intermediate contexts to rigorously isolate these components, thereby measuring the atomic performance boundary without the noise of cascading errors. Our results highlight a clear contrast in performance: while LLMs are proficient in extracting explicit constraints, they struggle to infer implicit, open-world requirements. Furthermore, they exhibit structural biases in plan generation and suffer from ineffective self-correction, characterized by excessive sensitivity and erroneous persistence. These findings offer precise directions for improving LLM reasoning and planning abilities.

Abstract

LLM Evaluation

Explore similar work

Revisiting the Travel Planning Capabilities of Large Language Models

Trip+: Benchmarking Agents in Personalized Interactive Travel Planning

TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning Agents