LLM-Driven Multi-Turn Task-Oriented Dialogue Synthesis for Realistic Reasoning
Organizations: Department of Computer Science and Technology, Tongji University
Abstract
The reasoning capability of large language models (LLMs), defined as their ability to analyze, infer, and make decisions based on input information, is essential for building intelligent task-oriented dialogue systems. However, existing benchmarks do not sufficiently reflect the complexity of real-world scenarios, which limits their effectiveness in evaluating and enhancing LLM reasoning in practical contexts. Many current reasoning datasets are overly simplistic and abstract, often disconnected from realistic task flows, domain constraints, and operational rules, making it difficult to effectively evaluate LLMs' logical reasoning ability. In addition, data contamination from pretraining corpora undermines the reliability of evaluation results, and traditional crowdsourcing methods for dataset construction are labor-intensive and difficult to scale. To address these challenges, we propose a LLM-driven framework for synthesizing multi-turn, task-oriented dialogues grounded in realistic reasoning scenarios, leveraging trilevel optimization to enhance dialogue quality. Our method generates dialogues grounded in authentic task scenarios, enriched with real-world information, and exhibiting strong contextual coherence. Corresponding reasoning tasks are carefully designed around these dialogues and iteratively refined to continuously improve the tasks' quality and challenge. The resulting dataset serves as a valuable benchmark for assessing and advancing the realistic logical reasoning capabilities of LLMs. Experimental results show that our synthetic data-based reasoning tasks introduce non-trivial reasoning challenges and provide meaningful support for improving the reasoning capabilities of LLMs.
Figures & tables
| Metric | RealReasoning |
| Dialogues | 500 |
| Total turns | 2398 |
| Avg. turns per dialogue | 4.796 |
| Avg. tokens per turn | 69.456 |
| Unique tokens/Total tokens | 0.024 |
| Metric | Coherence | Fluency | Diversity | Avg |
| Unoptimized | 4.33 | 3.08 | 2.96 | 3.46 |
| distinct-N | 4.91 | 3.50 | 3.07 | 3.82 |
| TF-IDF | 4.35 | 3.28 | 3.10 | 3.57 |
| Embedding | 4.90 | 3.50 | 3.00 | 3.80 |
| ours | 4.90 | 4.02 | 3.12 | 4.01 |
| Model | RealReasoning math word reasoning | RealReasoning common-sense reasoning | RealReasoning | GSM8K | CODAH |
| qwen-plus | 40.6% | 59.5% | 48.4% | 69.1% | 94.8% |
| qwen-plus-thinking | 88.1% | 74.6% | 82.6% | 94.2% | 95.0% |
| qwen-turbo | 31.8% | 52.6% | 40.4% | 69.9% | 90.8% |
| qwen-turbo-thinking | 88.1% | 77.5% | 83.8% | 94.4 % | 86.3% |
| deepseek-r1 | 89.4 % | 85.3 % | 87.8 % | 94.2% | 95.2 % |
| deepseek-r1-distill-qwen-32b | 78.3% | 71.7% | 75.6% | 68.8% | 88.6% |
| Method | Cosine Similarity | Topic Diversity and Redundancy | |||
| Max | Min | Mean | Unique Topics | Max Repetition | |
| Direct Interaction | 1.00 | 0.820 | 0.921 | 65 | 10 |
| + User Identity Info | 0.986 | 0.636 | 0.830 | 75 | 7 |
| + Action Sequence Info | 0.946 | 0.528 | 0.813 | 83 | 4 |
Synthetic Benchmark Generation
Synthetic benchmark generation creates controlled, verifiable evaluation tasks and datasets to test models across diverse settings.
See all 281 papers