Large Language Models (LLMs) achieve strong performance on many reasoning benchmarks, yet these evaluations typically focus on isolated tasks that differ from real-world usage in task-oriented dialogue (TOD). In this setting, LLMs must perform reasoning inherently while generating text and adhering to instructions on role, format, and style. This mismatch raises concerns about whether benchmark performance accurately reflects models' reasoning robustness in TOD setting. We investigate how framing reasoning tasks within TOD affects LLM performance by introducing BOULDER, a new dynamic benchmark covering eight travel-related tasks that require arithmetic, spatial, and temporal reasoning with both commonsense and formal aspects. Each problem is presented in both isolated and dialogue-based variants, enabling controlled comparison while mitigating data contamination. Experiments on eight LLMs reveal a substantial and consistent performance gap between isolated and dialogue settings. Through ablations and qualitative analysis, we show that this gap is largely driven by the multi-turn nature of dialogue, with additional effects from role conditioning and tool-use requirements. Our results highlight the need to evaluate LLM reasoning in realistic interactive scenarios.
Figures & tables
Figure 1: An example from our Boulder benchmark, showing the same problem instance in two variants: as an isolated task and within a task-oriented dialogue.
Task
Domains
Example query
Extracted value
Metric
Ticket price
trains
Can you first calculate the total price for me? There are four of us. Three of us are buying return tickets, one standard class, and two first class - one with a 33% discount, the other without discount. Another person is buying a one-way ticket.
amount (float)
Accuracy
Booking price
hotels
Can you first calculate the total price for me? It’s for five people staying for four nights. One person will stay in a single room and the others will share double rooms. One person in a double room will check in one day later.
amount (float)
Accuracy
Departure time
trains
What time is the last departure that will get me to Cambridge before sunset?
time (HH:MM)
Accuracy
Departure frequency
trains
How often on average do trains run between 19:00 and midnight?
minutes (float)
MAE
Opening hours
restaurants
Which of them are open on Wednesday for the entire time between 2pm and 6:30pm?
venues (array[string])
Precision
Distance
hotels, restaurants
How far is it from the hotel to the restaurant?
distance (float)
MAE
Table 1: Overview of tasks in the Boulder benchmark, including corresponding domains, representative examples of user queries, types of values extracted from the generated answers, and primary evaluation metrics.
Task
Acc.
Model
Acc.
Ticket price
96.39%
Qwen3 4B
96.52%
Booking price
98.75%
Mistral Small 24B
98.47%
Departure time
98.61%
Qwen3 30B
98.05%
Departure freq.
97.50%
Command A 111B
98.61%
Opening hours
94.86%
Qwen3 235B
95.83%
Distance
96.11%
DeepSeek V3.2
95.97%
Table 2: Results of human evaluation of parsers aggregated by task and model. For each task, we sampled 90 examples for each model, one for each of the three main setups.
Model
Architecture
Open-weight
Qwen3 4B
Dense
✓
Mistral Small 24B
Dense
✓
Qwen3 30B A3B
MoE
✓
Command A 111B
Dense
✓
Qwen3 235B A22B
MoE
✓
DeepSeek V3.2 671B
MoE
✓
Table 3: Overview of the evaluated LLMs.
Figure 2: Evaluation results of 8 LLMs in the three main settings, averaged over all tasks. Asterisks indicate statistically significant differences between the Baseline and Dialogue , and Dialogue and Dialogue-concise settings (t-test, ∗:p<0.05 , ∗∗:p<0.01 , ∗∗∗:p<0.001 ).
Figure 3: Results for the Dialogue with reduced domains and Dialogue without tools ablations, micro-averaged over all tasks. Significance indication follows Figure 2 , and each ablation is compared to the main Dialogue setup.
Figure 4: Results for the Multi-turn baseline and Single-turn dialogue ablations, micro-averaged over all tasks. Significance indication follows Figure 2 and each ablation is compared with the corresponding main setup.
Figure 5: Results for the Baseline with dialogue role and Dialogue with reasoning instruction ablations, micro-averaged over all tasks. Significance indication follows Figure 2 and each ablation is compared with the corresponding main setup.
Figure 6: Detailed per-task results for the Baseline (b), Dialogue (d), and Dialogue-concise (c) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, ∗:p<0.05 , ∗∗:p<0.01 , ∗∗∗:p<0.001 ).
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Precision
Ollama/OpenRouter Tag
Qwen3 4B
FP16
qwen3:4b-instruct-2507-fp16
Mistral Small 24B
Q8_0
mistral-small:24b-instruct-2501-q8_0
Qwen3 30B A3B
Q8_0
qwen3:30b-a3b-instruct-2507-Q8_0
Command A 111B
Q4_K_M
command-a:111b-03-2025-q4_K_M
Qwen3 235B A22B
Q4_K_M
qwen3:235b-a22b-instruct-2507-q4_K_M
DeepSeek V3.2 671B
FP8
deepseek/deepseek-v3.2
Appendix
Table 4: Models used for the evaluation, including quantization levels and Ollama tags (for open-weight LLMs) or OpenRouter tags (for proprietary LLMs and DeepSeek V3.2).
Figure 7: Example conversation template in JSON format for the opening hours task. During test example generation, placeholders in curly brackets are replaced with actual values and placeholder messages are filled with specific tool calls and tool responses.
Figure 8: Prompt template used for generating paraphrases of conversation templates. Paraphrases generated in previous iterations are included in the prompt in subsequent inference calls to make the result more diverse.
Figure 9: Prompt template for the task-oriented dialogue system used in the Dialogue setup in our experiments, adapted from ( Xu et al., 2024 ) .
Figure 10: Prompt template for the task-oriented dialogue system used in the Dialogue-concise setup in our experiments, adapted from ( Xu et al., 2024 ) . Note that it differs from the prompt for the Dialogue setup (Figure 9 ) only in the last section (“General Instructions”).
Figure 11: Changes made to the prompt for the Dialogue with reduced domains ablation. Text highlighted in red indicates removed content. This example illustrates the changes made to the distance and opening hours tasks, where we remove the attractions and trains domains.
Figure 12: Changes made to the prompt and the message history for the Dialogue without tools ablation. Text highlighted in red indicates removed content, while a green highlight indicates added content. The ablation consists of the following modifications: tool schemas are removed, the data from tool responses are added to the system prompt, the instructions that refer to tools are modified to refer to the data.
Figure 13: Changes made to the prompt and the message history for the Single-turn dialogue ablation. Text highlighted in red indicates removed content, while a green highlight indicates added content. The ablation consists of the following modifications: all user and assistant messages are removed from the message history and replaced by a single user message that combines all previous user queries (if they are applicable); other modifications are the same as in Dialogue without tools .
Figure 14: Changes made to the prompt and the message history for the Multi-turn baseline ablation. Text highlighted in red indicates removed content, while a green highlight indicates added content. The ablation consists of the following modifications: the role of the original prompt is changed to system , the turns from the dialogue setup are added to the message history, and the query is moved from the original prompt to the last user message.
Figure 15: Changes made to the prompt and the message history for the Baseline with dialogue role ablation. Text highlighted in green indicates added content. Instructions related to the role of the task-oriented dialogue assistant are added to the baseline prompt.
Figure 16: Changes made to the prompt and the message history for the Reasoning instructions ablation. Text highlighted in green indicates added content. The prompt is the same as in the Dialogue setup, with the added instruction for the model to use more time for reasoning when needed.
Figure 17: Prompt template for the amounts parser used to extract values in the Train ticket price and Hotel booking price tasks. The answer is extracted from model output as a floating point number, representing the total amount for requested train tickets or hotel booking.
Figure 18: Prompt template for the time parser used to extract values in the Train departure time task. The answer is extracted from model output as a timestamp in ISO format (HH:MM), representing the departure time of the latest train that arrives to its destination before sunset.
Figure 19: Prompt template for the time parser used to extract values in the Train departure frequency task. The answer is extracted from model output as a floating point number representing the average time interval of departures in minutes.
Figure 20: Prompt template for the restaurant names parser used to extract values in the Restaurant opening hours task. The answer is extracted from model output as a list of restaurant names, representing the subset of restaurants that are open during the requested time interval.
Figure 21: Prompt template for the time parser used to extract values in the Distance between venues task. The answer is extracted from model output as an integer, representing the distance in meters between specified venues.
Figure 22: Prompt template for the time parser used to extract values in the Directional relations task. The answer is extracted from model output as one of three string values (yes/no/unknown), corresponding to a model’s answer to a question about a venue being in a specific cardinal direction of another venue.
Figure 23: Prompt template for the time parser used to extract values in the Shortest walking path task. The answer is extracted from model output as one a list of attraction names representing the optimal order to visit the venues from a hotel.
Figure 24: Detailed results for the Dialogue with reduced domains (d(d)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, ∗:p<0.05 , ∗∗:p<0.01 , ∗∗∗:p<0.001 ).
Figure 25: Detailed results for the Dialogue without tools (d(t)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, ∗:p<0.05 , ∗∗:p<0.01 , ∗∗∗:p<0.001 ).
Figure 26: Detailed results for the Multi-turn baseline (b(m)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, ∗:p<0.05 , ∗∗:p<0.01 , ∗∗∗:p<0.001 ).
Figure 27: Detailed results for the Single-turn dialogue (d(s)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, ∗:p<0.05 , ∗∗:p<0.01 , ∗∗∗:p<0.001 ).
Figure 28: Detailed results for the Baseline with dialogue role (b(r)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, ∗:p<0.05 , ∗∗:p<0.01 , ∗∗∗:p<0.001 ).
Figure 29: Detailed results for the Dialogue with reasoning instructions (d(r)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, ∗:p<0.05 , ∗∗:p<0.01 , ∗∗∗:p<0.001 ).
Figure 30: Detailed results for the parser evaluation by task, model, and evaluation setting. The error bars show 95% confidence intervals, calculated using the Wilson score method.
Figure 31: Scores and average response lengths in characters by LLM averaged over all tasks.
Figure 32: Comparison of scores and average response lengths in characters by LLM for the Baseline with dialogue role ablation.
Figure 33: Comparison of scores and average response lengths in characters by LLM for the Dialogue with reasoning instructions ablation.
Figure 34: Comparison of responses from Qwen3 30B A3B for the Hotel booking price task in the Baseline and Dialogue settings. In the Baseline setup, the LLM provides correct answer, while it is incorrect in the Dialogue setup. This is an example of the Post-hoc rationalization pattern (see Section E.1 ).
Figure 35: Comparison of responses from Command A 111B for the Train departure frequency task in the Baseline and Dialogue settings. In the Baseline setup, the LLM provides an answer withing a minute of the target, while it refuses to solve the task in the Dialogue setup. The response of an example of the Task avoidance pattern (see Section E.3 ).
Figure 36: Comparison of responses from Qwen3 235B A22B for the Directional relations task in the Baseline and Dialogue settings. In the Baseline setup, the LLM provides correct answer, while it is incorrect in the Dialogue setup due an inversion of the coordinate system mapping provided in the prompt. The response is an example of the Rule inversion pattern (see Section E.6 ).
Figure 37: Comparison of responses from Claude 4.5 Sonnet for the Train departure time task in the Baseline and Dialogue settings. In the Baseline setup, the LLM provides the optimal option, while it is suboptimal in the Dialogue setup. The response is an example of the Boundary failure pattern (see Section E.7 ).
The reasoning capability of large language models (LLMs), defined as their ability to analyze, infer, and make decisions based on input information, is essential for building intelligent task-oriented dialogue systems. However, existing benchmarks do not sufficiently reflect the complexity of real-world scenarios, which limits their effectiveness in evaluating and enhancing LLM reasoning in practical contexts. Many current reasoning datasets are overly simplistic and abstract, often disconnected from realistic task flows, domain constraints, and operational rules, making it difficult to effectively evaluate LLMs' logical reasoning ability. In addition, data contamination from pretraining corpora undermines the reliability of evaluation results, and traditional crowdsourcing methods for dataset construction are labor-intensive and difficult to scale. To address these challenges, we propose a LLM-driven framework for synthesizing multi-turn, task-oriented dialogues grounded in realistic reasoning scenarios, leveraging trilevel optimization to enhance dialogue quality. Our method generates dialogues grounded in authentic task scenarios, enriched with real-world information, and exhibiting strong contextual coherence. Corresponding reasoning tasks are carefully designed around these dialogues and iteratively refined to continuously improve the tasks' quality and challenge. The resulting dataset serves as a valuable benchmark for assessing and advancing the realistic logical reasoning capabilities of LLMs. Experimental results show that our synthetic data-based reasoning tasks introduce non-trivial reasoning challenges and provide meaningful support for improving the reasoning capabilities of LLMs.
Yu Zhu, Kai Yang
Department of Computer Science and Technology, Tongji University
We introduce a multi-turn interactive framework for reasoning evaluation that treats reasoning as active evidence acquisition and belief updating. Wherein, LLMs receive only the task rules, must issue targeted queries to a hidden environment, integrate partial observations over time, and decide when to submit a final answer. Beyond standard success rate and interaction efficiency, we evaluate contextual robustness under controlled contextual perturbations, and metacognitive adaptation through counterfactual revision and necessity judgment. We instantiate the framework as a benchmark of 474 executable games, each evaluated under five fixed configuration search spaces corresponding to five difficulty levels, and evaluate a broad set of frontier LLMs. Results show that the benchmark is highly discriminative, exposing large differences not only in success rate but also in interaction efficiency. Moreover, we empirically show that contextual perturbations cause moderate but consistent declines, whereas counterfactual revision and necessity judgment lead to much larger drops.
Characterizing LLM reasoning remains an open challenge, as many existing benchmarks isolate specific reasoning skills, rely on external knowledge, or are costly to extend. We introduce PetriBench, a compact, fully self-contained, and scalable benchmark for evaluating LLM reasoning over dynamic state spaces using Petri nets, a mature formalism for modeling real-world concurrent and distributed systems. PetriBench organizes reasoning into four task families varying by scope and temporal horizon, with Easy, Medium, and Hard levels generated by increasing structural complexity and evaluated against exact ground truth. Across a diverse set of proprietary and open-weight models, accuracy decreases consistently with difficulty, while harder instances expose increasingly distinct task-specific capability profiles. Additional analyses show that test-time compute improves performance but interacts differently with different reasoning tasks, and that procedural generation yields smooth scaling with structural complexity. Together, these results show that PetriBench provides a unified and extensible setting for probing the strengths, limits, and scaling behavior of LLM reasoning.