cs.CLMar 20, 2026
SaveReasoning Gets Harder for LLMs Inside A Dialogue
Organizations: Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Prague, Czechia
Abstract
Large Language Models (LLMs) achieve strong performance on many reasoning benchmarks, yet these evaluations typically focus on isolated tasks that differ from real-world usage in task-oriented dialogue (TOD). In this setting, LLMs must perform reasoning inherently while generating text and adhering to instructions on role, format, and style. This mismatch raises concerns about whether benchmark performance accurately reflects models' reasoning robustness in TOD setting. We investigate how framing reasoning tasks within TOD affects LLM performance by introducing BOULDER, a new dynamic benchmark covering eight travel-related tasks that require arithmetic, spatial, and temporal reasoning with both commonsense and formal aspects. Each problem is presented in both isolated and dialogue-based variants, enabling controlled comparison while mitigating data contamination. Experiments on eight LLMs reveal a substantial and consistent performance gap between isolated and dialogue settings. Through ablations and qualitative analysis, we show that this gap is largely driven by the multi-turn nature of dialogue, with additional effects from role conditioning and tool-use requirements. Our results highlight the need to evaluate LLM reasoning in realistic interactive scenarios.
Figures & tables
Figure 1: An example from our Boulder benchmark, showing the same problem instance in two variants: as an isolated task and within a task-oriented dialogue.
| Task | Domains | Example query | Extracted value | Metric |
| Ticket price | trains | Can you first calculate the total price for me? There are four of us. Three of us are buying return tickets, one standard class, and two first class - one with a 33% discount, the other without discount. Another person is buying a one-way ticket. | amount (float) | Accuracy |
| Booking price | hotels | Can you first calculate the total price for me? It’s for five people staying for four nights. One person will stay in a single room and the others will share double rooms. One person in a double room will check in one day later. | amount (float) | Accuracy |
| Departure time | trains | What time is the last departure that will get me to Cambridge before sunset? | time (HH:MM) | Accuracy |
| Departure frequency | trains | How often on average do trains run between 19:00 and midnight? | minutes (float) | MAE |
| Opening hours | restaurants | Which of them are open on Wednesday for the entire time between 2pm and 6:30pm? | venues (array[string]) | Precision |
| Distance | hotels, restaurants | How far is it from the hotel to the restaurant? | distance (float) | MAE |
Table 1: Overview of tasks in the Boulder benchmark, including corresponding domains, representative examples of user queries, types of values extracted from the generated answers, and primary evaluation metrics.
| Task | Acc. | Model | Acc. |
|---|---|---|---|
| Ticket price | 96.39% | Qwen3 4B | 96.52% |
| Booking price | 98.75% | Mistral Small 24B | 98.47% |
| Departure time | 98.61% | Qwen3 30B | 98.05% |
| Departure freq. | 97.50% | Command A 111B | 98.61% |
| Opening hours | 94.86% | Qwen3 235B | 95.83% |
| Distance | 96.11% | DeepSeek V3.2 | 95.97% |
Table 2: Results of human evaluation of parsers aggregated by task and model. For each task, we sampled 90 examples for each model, one for each of the three main setups.
| Model | Architecture | Open-weight |
|---|---|---|
| Qwen3 4B | Dense | ✓ |
| Mistral Small 24B | Dense | ✓ |
| Qwen3 30B A3B | MoE | ✓ |
| Command A 111B | Dense | ✓ |
| Qwen3 235B A22B | MoE | ✓ |
| DeepSeek V3.2 671B | MoE | ✓ |
Table 3: Overview of the evaluated LLMs.
Figure 2: Evaluation results of 8 LLMs in the three main settings, averaged over all tasks. Asterisks indicate statistically significant differences between the Baseline and Dialogue , and Dialogue and Dialogue-concise settings (t-test, , , ).
Figure 3: Results for the Dialogue with reduced domains and Dialogue without tools ablations, micro-averaged over all tasks. Significance indication follows Figure 2 , and each ablation is compared to the main Dialogue setup.
Figure 4: Results for the Multi-turn baseline and Single-turn dialogue ablations, micro-averaged over all tasks. Significance indication follows Figure 2 and each ablation is compared with the corresponding main setup.
Figure 5: Results for the Baseline with dialogue role and Dialogue with reasoning instruction ablations, micro-averaged over all tasks. Significance indication follows Figure 2 and each ablation is compared with the corresponding main setup.
Figure 6: Detailed per-task results for the Baseline (b), Dialogue (d), and Dialogue-concise (c) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, , , ).
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Precision | Ollama/OpenRouter Tag |
| Qwen3 4B | FP16 | qwen3:4b-instruct-2507-fp16 |
| Mistral Small 24B | Q8_0 | mistral-small:24b-instruct-2501-q8_0 |
| Qwen3 30B A3B | Q8_0 | qwen3:30b-a3b-instruct-2507-Q8_0 |
| Command A 111B | Q4_K_M | command-a:111b-03-2025-q4_K_M |
| Qwen3 235B A22B | Q4_K_M | qwen3:235b-a22b-instruct-2507-q4_K_M |
| DeepSeek V3.2 671B | FP8 | deepseek/deepseek-v3.2 |
Table 4: Models used for the evaluation, including quantization levels and Ollama tags (for open-weight LLMs) or OpenRouter tags (for proprietary LLMs and DeepSeek V3.2).
Figure 7: Example conversation template in JSON format for the opening hours task. During test example generation, placeholders in curly brackets are replaced with actual values and placeholder messages are filled with specific tool calls and tool responses.
Figure 8: Prompt template used for generating paraphrases of conversation templates. Paraphrases generated in previous iterations are included in the prompt in subsequent inference calls to make the result more diverse.
Figure 9: Prompt template for the task-oriented dialogue system used in the Dialogue setup in our experiments, adapted from ( Xu et al., 2024 ) .
Figure 10: Prompt template for the task-oriented dialogue system used in the Dialogue-concise setup in our experiments, adapted from ( Xu et al., 2024 ) . Note that it differs from the prompt for the Dialogue setup (Figure 9 ) only in the last section (“General Instructions”).
Figure 11: Changes made to the prompt for the Dialogue with reduced domains ablation. Text highlighted in red indicates removed content. This example illustrates the changes made to the distance and opening hours tasks, where we remove the attractions and trains domains.
Figure 12: Changes made to the prompt and the message history for the Dialogue without tools ablation. Text highlighted in red indicates removed content, while a green highlight indicates added content. The ablation consists of the following modifications: tool schemas are removed, the data from tool responses are added to the system prompt, the instructions that refer to tools are modified to refer to the data.
Figure 13: Changes made to the prompt and the message history for the Single-turn dialogue ablation. Text highlighted in red indicates removed content, while a green highlight indicates added content. The ablation consists of the following modifications: all user and assistant messages are removed from the message history and replaced by a single user message that combines all previous user queries (if they are applicable); other modifications are the same as in Dialogue without tools .
Figure 14: Changes made to the prompt and the message history for the Multi-turn baseline ablation. Text highlighted in red indicates removed content, while a green highlight indicates added content. The ablation consists of the following modifications: the role of the original prompt is changed to system , the turns from the dialogue setup are added to the message history, and the query is moved from the original prompt to the last user message.
Figure 15: Changes made to the prompt and the message history for the Baseline with dialogue role ablation. Text highlighted in green indicates added content. Instructions related to the role of the task-oriented dialogue assistant are added to the baseline prompt.
Figure 16: Changes made to the prompt and the message history for the Reasoning instructions ablation. Text highlighted in green indicates added content. The prompt is the same as in the Dialogue setup, with the added instruction for the model to use more time for reasoning when needed.
Figure 17: Prompt template for the amounts parser used to extract values in the Train ticket price and Hotel booking price tasks. The answer is extracted from model output as a floating point number, representing the total amount for requested train tickets or hotel booking.
Figure 18: Prompt template for the time parser used to extract values in the Train departure time task. The answer is extracted from model output as a timestamp in ISO format (HH:MM), representing the departure time of the latest train that arrives to its destination before sunset.
Figure 19: Prompt template for the time parser used to extract values in the Train departure frequency task. The answer is extracted from model output as a floating point number representing the average time interval of departures in minutes.
Figure 20: Prompt template for the restaurant names parser used to extract values in the Restaurant opening hours task. The answer is extracted from model output as a list of restaurant names, representing the subset of restaurants that are open during the requested time interval.
Figure 21: Prompt template for the time parser used to extract values in the Distance between venues task. The answer is extracted from model output as an integer, representing the distance in meters between specified venues.
Figure 22: Prompt template for the time parser used to extract values in the Directional relations task. The answer is extracted from model output as one of three string values (yes/no/unknown), corresponding to a model’s answer to a question about a venue being in a specific cardinal direction of another venue.
Figure 23: Prompt template for the time parser used to extract values in the Shortest walking path task. The answer is extracted from model output as one a list of attraction names representing the optimal order to visit the venues from a hotel.
Figure 24: Detailed results for the Dialogue with reduced domains (d(d)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, , , ).
Figure 25: Detailed results for the Dialogue without tools (d(t)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, , , ).
Figure 26: Detailed results for the Multi-turn baseline (b(m)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, , , ).
Figure 27: Detailed results for the Single-turn dialogue (d(s)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, , , ).
Figure 28: Detailed results for the Baseline with dialogue role (b(r)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, , , ).
Figure 29: Detailed results for the Dialogue with reasoning instructions (d(r)) ablation, compared with the Baseline (b) and Dialogue (d) setups. Asterisks indicate significant differences between setups in neighboring columns (t-test, , , ).
Figure 30: Detailed results for the parser evaluation by task, model, and evaluation setting. The error bars show 95% confidence intervals, calculated using the Wilson score method.
Figure 31: Scores and average response lengths in characters by LLM averaged over all tasks.
Figure 32: Comparison of scores and average response lengths in characters by LLM for the Baseline with dialogue role ablation.
Figure 33: Comparison of scores and average response lengths in characters by LLM for the Dialogue with reasoning instructions ablation.
Figure 34: Comparison of responses from Qwen3 30B A3B for the Hotel booking price task in the Baseline and Dialogue settings. In the Baseline setup, the LLM provides correct answer, while it is incorrect in the Dialogue setup. This is an example of the Post-hoc rationalization pattern (see Section E.1 ).
Figure 35: Comparison of responses from Command A 111B for the Train departure frequency task in the Baseline and Dialogue settings. In the Baseline setup, the LLM provides an answer withing a minute of the target, while it refuses to solve the task in the Dialogue setup. The response of an example of the Task avoidance pattern (see Section E.3 ).
Figure 36: Comparison of responses from Qwen3 235B A22B for the Directional relations task in the Baseline and Dialogue settings. In the Baseline setup, the LLM provides correct answer, while it is incorrect in the Dialogue setup due an inversion of the coordinate system mapping provided in the prompt. The response is an example of the Rule inversion pattern (see Section E.6 ).
Figure 37: Comparison of responses from Claude 4.5 Sonnet for the Train departure time task in the Baseline and Dialogue settings. In the Baseline setup, the LLM provides the optimal option, while it is suboptimal in the Dialogue setup. The response is an example of the Boundary failure pattern (see Section E.7 ).