cs.CLFeb 27, 2026

LLM-Driven Multi-Turn Task-Oriented Dialogue Synthesis for Realistic Reasoning

Authors: Yu Zhu, Kai Yang

Organizations: Department of Computer Science and Technology, Tongji University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The reasoning capability of large language models (LLMs), defined as their ability to analyze, infer, and make decisions based on input information, is essential for building intelligent task-oriented dialogue systems. However, existing benchmarks do not sufficiently reflect the complexity of real-world scenarios, which limits their effectiveness in evaluating and enhancing LLM reasoning in practical contexts. Many current reasoning datasets are overly simplistic and abstract, often disconnected from realistic task flows, domain constraints, and operational rules, making it difficult to effectively evaluate LLMs' logical reasoning ability. In addition, data contamination from pretraining corpora undermines the reliability of evaluation results, and traditional crowdsourcing methods for dataset construction are labor-intensive and difficult to scale. To address these challenges, we propose a LLM-driven framework for synthesizing multi-turn, task-oriented dialogues grounded in realistic reasoning scenarios, leveraging trilevel optimization to enhance dialogue quality. Our method generates dialogues grounded in authentic task scenarios, enriched with real-world information, and exhibiting strong contextual coherence. Corresponding reasoning tasks are carefully designed around these dialogues and iteratively refined to continuously improve the tasks' quality and challenge. The resulting dataset serves as a valuable benchmark for assessing and advancing the realistic logical reasoning capabilities of LLMs. Experimental results show that our synthetic data-based reasoning tasks introduce non-trivial reasoning challenges and provide meaningful support for improving the reasoning capabilities of LLMs.

Figures & tables

Figure 1: Schematic of the synthetic generation process.
Figure 2: The quality of generated dialogues.
MetricRealReasoning
Dialogues500
Total turns2398
Avg. turns per dialogue4.796
Avg. tokens per turn69.456
Unique tokens/Total tokens0.024
Table 1: RealReasoning Dataset Statistics
Figure 3: Iterative updating of the dataset.
MetricCoherenceFluencyDiversityAvg
Unoptimized4.333.082.963.46
distinct-N4.913.503.073.82
TF-IDF4.353.283.103.57
Embedding4.903.503.003.80
ours4.904.023.124.01
Table 2: The impact of different metrics on dialogue quality.
ModelRealReasoning math word reasoningRealReasoning common-sense reasoningRealReasoningGSM8KCODAH
qwen-plus40.6%59.5%48.4%69.1%94.8%
qwen-plus-thinking88.1%74.6%82.6%94.2%95.0%
qwen-turbo31.8%52.6%40.4%69.9%90.8%
qwen-turbo-thinking88.1%77.5%83.8%94.4 %86.3%
deepseek-r189.4 %85.3 %87.8 %94.2%95.2 %
deepseek-r1-distill-qwen-32b78.3%71.7%75.6%68.8%88.6%
Table 3: Answer accuracy of different models on datasets.
Figure 4: Impact of Ablation Components on Dialogue Topic Distribution.
Figure 5: Impact of Ablation Components on Cosine Similarity Distribution.
MethodCosine SimilarityTopic Diversity and Redundancy
 MaxMinMeanUnique TopicsMax Repetition
Direct Interaction1.000.8200.9216510
+ User Identity Info0.9860.6360.830757
+ Action Sequence Info0.9460.5280.813834
Table 4: Comparison of Multi-turn Dialogue Generation Results with Different Ablation Components Combined
More in this topic

Synthetic Benchmark Generation

Synthetic benchmark generation creates controlled, verifiable evaluation tasks and datasets to test models across diverse settings.

See all 281 papers

Explore similar work

CardsList
Mar 20, 2026

Reasoning Gets Harder for LLMs Inside A Dialogue

Large Language Models (LLMs) achieve strong performance on many reasoning benchmarks, yet these evaluations typically focus on isolated tasks that differ from real-world usage in task-oriented dialogue (TOD). In this setting, LLMs must perform reasoning inherently while generating text and adhering to instructions on role, format, and style. This mismatch raises concerns about whether benchmark performance accurately reflects models' reasoning robustness in TOD setting. We investigate how framing reasoning tasks within TOD affects LLM performance by introducing BOULDER, a new dynamic benchmark covering eight travel-related tasks that require arithmetic, spatial, and temporal reasoning with both commonsense and formal aspects. Each problem is presented in both isolated and dialogue-based variants, enabling controlled comparison while mitigating data contamination. Experiments on eight LLMs reveal a substantial and consistent performance gap between isolated and dialogue settings. Through ablations and qualitative analysis, we show that this gap is largely driven by the multi-turn nature of dialogue, with additional effects from role conditioning and tool-use requirements. Our results highlight the need to evaluate LLM reasoning in realistic interactive scenarios.

Ivan Kartáč, Mateusz Lango, Ondřej Dušek

Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Prague, Czechia

Spatial Reasoning in Language ModelsMulti-Turn Dialogue EvaluationTask-Oriented Dialogue SystemsNumerical Reasoning in Language Models
AskPDF
May 10, 2026

Absurd World: A Simple Yet Powerful Method to Absurdify the Real-world for Probing LLM Reasoning Capabilities

While extremely powerful and versatile at various tasks, the thinking capabilities of large language models (LLMs) are often put under scrutiny as they sometimes fail to solve problems that humans can systematically solve. However, recent literature focuses on breaking LLM reasoning with increasingly complex problems, and whether an LLM is robust in simple logical reasoning remains underexplored. This paper proposes Absurd World, a benchmarking framework, to test LLMs against altered realism, where scenarios are logically coherent, and humans can easily solve the tasks. Absurd World breaks a real-world model into symbols, actions, sequences, and events, which are automatically altered to create absurd worlds where the logic to solve the tasks remains the same. It evaluates a large collection of models with simple and advanced prompting techniques, and proves that it is an effective tool to determine LLMs' ability to think logically, ignoring the patterns learned from the real world. One can use this framework to extensively test an LLM against a real-world problem to verify whether the LLM's reasoning capability is robust against variations of the task.

Ryan Albright, Golam Md Muktadir, Zarif Ikram +3

The Nueva School, San Mateo, USA · University of Southern California · Notre Dame College, Dhaka, Bangladesh +2

LLM EvaluationLanguage Model Reasoning EvaluationLanguage Model RobustnessLLM Reasoning
AskPDF
Jun 18, 2026

QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making. As models improve, evaluation benchmarks should evolve to keep pace. However, existing benchmarks lack fine-grained control over logical complexity and struggle to balance semantic diversity with logical consistency. To address these issues, we propose QMFOL, an automated framework for generating monadic first-order logic reasoning tasks with quantifiable and controllable complexity. It constructs formal logical structures using conjunction and disjunction patterns, enabling precise control over reasoning depth, width, label types, and distractors. These structures are then translated into natural language via LLMs, with logical consistency ensured through round-trip verification using an external prover. Based on our framework, we build QMFOLBench, a benchmark comprising 2880 instances with 960 configurations across diverse logical and semantic dimensions. Evaluations on six large reasoning models (LRMs) and two LLMs show that performance degrades and computational overhead increases with rising logical complexity. Models perform better on True-labeled tasks than on False or Unknown ones, and exhibit sensitivity to semantic variation. Overall, QMFOL offers a scalable and reliable approach for constructing deductive reasoning benchmarks with controllable complexity, enabling more precise evaluation of reasoning capabilities in modern language models.

Xinyi Zheng, Ling Shi, Tianlong Yu +3

Huazhong University of Science and Technology Wuhan, China · Nanyang Technological University Singapore, Singapore · Hubei University Wuhan, China +2

Logical ReasoningLLM EvaluationSynthetic Benchmark GenerationLLM ReasoningLarge Reasoning Models
AskPDF