cs.CLMar 2, 2026
SaveExtracting Training Dialogue Data from Large Language Model based Task Bots
Organizations: MOE Key Laboratory for Intelligent Networks and Network Security, Xi’an Jiaotong University, P.O. Box 1088, No. 28, Xianning West Road, Xi’an, Shaanxi 710049, China.
Abstract
Large Language Models (LLMs) have been widely adopted to enhance Task-Oriented Dialogue Systems (TODS) by modeling complex language patterns and delivering contextually appropriate responses. However, this integration introduces significant privacy risks, as LLMs, functioning as soft knowledge bases that compress extensive training data into rich knowledge representations, can inadvertently memorize training dialogue data containing not only identifiable information such as phone numbers but also entire dialogue-level events like complete travel schedules. Despite the critical nature of this privacy concern, how LLM memorization is inherited in developing task bots remains unexplored. In this work, we address this gap through a systematic quantitative study that involves evaluating existing training data extraction attacks, analyzing key characteristics of task-oriented dialogue modeling that render existing methods ineffective, and proposing novel attack techniques tailored for LLM-based TODS that enhance both response sampling and membership inference. Experimental results demonstrate the effectiveness of our proposed data extraction attack. Our method can extract thousands of training labels of dialogue states with best-case precision exceeding 70%. Furthermore, we provide an in-depth analysis of training data memorization in LLM-based TODS by identifying and quantifying key influencing factors and discussing targeted mitigation strategies.
Figures & tables
Fig. 1: (a) LLM-based task bot utilizes a single neural auto-regressive model to parameterize the sequential dialogue pipeline. (b) Our dialogue state extraction attack extracts the training labels of Dialogue States without access to the corresponding conditioning text of dialogue histories.
Fig. 2: (a) Workflow of the dialogue state extraction attack: We begin with Suffix Decoding , generating many samples from the LLM-based task bot by seeding the model with either empty inputs (for untargeted attack) or partial dialogue state prefixes (for targeted attack). Membership Inference metrics are then applied to rank each generation, with higher ranks indicating a greater likelihood of originating from the training data. For evaluation , we deduplicate generations, keep the top 100, and mark each generation as either memorized or not memorized by comparing them with the dialogue states from the training dataset. (b) Schema-Guided Sampling Section IV-B2 generates more valid results by constraining word choices for sampling domains and slots using the dialogue schema (i.e., service scope), which is automatically explored by simulating user interactions with ChatGPT. (c) Debiased Conditional Perplexity Section IV-C reduces bias toward favoring common generic dialogue state components.
Fig. 3: Schema extraction with ChatGPT. In (a) model-against-model loop , based on the collected schema (none for the first loop), ChatGPT is prompted to simulate a user asking the task bot questions to uncover additional domains and slots. To improve precision and exploration, we further propose to enhance the basic loop with (b) inter-temperature consistency . When decoding the answer from the task bot as in (a), we use high-temperature settings to generate diverse belief states, which may include incorrect domains and slots. We then filter these errors through a verification process, where ChatGPT is prompted to ask corresponding questions about the extracted (probably incorrect) states, and the task bot decodes answers under low-temperature settings to generate confident answers that exclude those illegal domains and slots.
| Generate questions for the dialogue system given the existing schema following these steps: Review the Existing Schema: Carefully examine the provided Validated and Confirmed Schema . Make sure you fully understand each domain, slot, and value listed within. Craft Simulated Task-oriented Queries: Based on each existing domain, create a series of questions that deviate slightly from the known scope. These queries should help probe potential new slots or values within those domains. For example, if you have a domain weather and a slot city , you might ask: “How’s the weather for outdoor activities tomorrow?” If there is a domain restaurant , you could say: “Can you suggest vegan restaurants near Tokyo?” Implement Cross-Domain Probing: Combining known slots and values, try constructing cross-domain queries. For example, you could ask: “Are there any music events happening in London when it’s forecasted to rain?” Or, “Can I get a taxi from Beijing Duck to the nearest cinema?” Initiate Random Task-oriented Queries: Design and propose a series of random task-oriented questions aiming to discover potential domains, slots, or values not covered by the current schema. The generated questions should be completely IRRELEVANT to the existing schema and based on a wide range of domains. Validated and Confirmed Schema: { schema: {domain: Restaurant, slots:[price_range, name]}} Your questions: |
TABLE I: LLM Prompt for Generating Schema-Probing Questions
| State Precision (%) | #Triplets | Value Precision (%) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Extraction Method | None | PPL% | PPL-zlib | None | PPL% | PPL-zlib | None | PPL% | PPL-zlib |
| Vanilla Sampling | 14.37 | 26.00 | 26.00 | 1.60 | 2.27 | 2.00 | 15.00 | 60.52 | 54.9 |
| Temperature Sampling | 4.59 | 19.00 | 11.00 | 1.29 | 1.53 | 1.45 | 5.76 | 38.56 | 33.00 |
| Beam Sampling | 11.95 | 23.00 | 21.00 | 1.95 | 2.52 | 2.71 | 10.73 | 66.97 | 68.1 |
TABLE II: Membership inference results for untargeted dialogue state extraction attack. Methods with fewer than 100 states are excluded, as the top 100 extractions are retained after ranking.
| Method | #States | SP% | #Triplets | #Values | VP% |
|---|---|---|---|---|---|
| Vanilla Sampling | 336 | 14.37 | 1.60 | 231 | 15.00 |
| Temperature Sampling | 207 | 4.59 | 1.29 | 234 | 5.76 |
| Decaying Temperature | 75 | 3.32 | 1.51 | 223 | 17.13 |
| Beam Search | 5 | 100 | 1.2 | 4 | 100 |
| Group Beam Search | 15 | 65.22 | 1.47 | 16 | 64.00 |
| Beam Sampling | 715 | 11.95 | 1.95 | 315 | 10.73 |
TABLE III: Untarget extraction using the strawman method
Fig. 4: Schema extraction performance over different temperatures. We report the mean values over three runs per method, along with the best-case results.
Fig. 5: Minimum number of questions required for ChatGPT to achieve the performance upper bound in schema extraction. For temperature sampling, samples are generated to reach the reported performance.
Fig. 6: Targeted state extraction results. We report the total number of successfully extracted training dialogue states, the memorized values within these states, and the extraction precision. The gray bars represent the number of deduplicated prefixes.
Fig. 7: Number of training dialogue states and values extracted under different sampling configurations. Results are reported for a prefix length of 3, with the largest sample size to provide statistically significant results.
Fig. 8: Repetitions per member under different prefix lengths: This measures how many times successfully extracted training dialogue states are repeated in the training dataset. Results are reported under a 99% confidence interval. Additionally, we include the state precision for different extraction methods (gray line).
Fig. 9: Gold answer ranking based on task bot decoding probability. We report results for slots, values, and the combined pairs. Avg. Rank (<5) refers to the average rank of examples that appear within the top 5. A lower rank indicates that the true answer is more confidently decoded by the task bot. Rank #1 and Rank >5 represent the percentage of true examples that rank at top-1 and those not within the top-5, respectively. A higher percentage of Rank #1 and a lower percentage of Rank >5 suggest greater certainty of the task bot in decoding the correct answers.
| State Precision (%) | Value Precision (%) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Extraction Method | None | PPL | PPL-zlib | C-PPL | DC-PPL | None | PPL | PPL-zlib | C-PPL | DC-PPL |
| Vanilla Sampling | 14.37 | 26.00 | 26.00 | 24.00 | 42.00 | 15.00 | 60.52 | 54.90 | 55.00 | 70.27 |
| Temperature Sampling | 4.59 | 19.00 | 11.00 | 14.00 | 23.00 | 5.76 | 38.56 | 33.00 | 36.95 | 46.45 |
| Beam Sampling | 11.95 | 23.00 | 21.00 | 29.00 | 40.00 | 10.73 | 66.97 | 68.10 | 70.19 | 76.34 |
TABLE IV: State and value precision for untargeted dialogue state extraction under various membership inference measures. “D” and “C” denotes “Debiased” and “conditional”, respectively.
Fig. 10: Dialogue state and slot value precision for targeted dialogue state extraction after membership inference. We report the results averaged across various extraction methods detailed in Section V-D . Shaded areas represent the performance range from the best to the worst extraction methods.
Fig. 11: Membership inference score distribution for extracted dialogue states. We compare dialogue states that appear in the training dataset (Members) with those excluded (Non-Members).
Fig. 12: PII score and marginal privacy score under different training data extraction (a) (b) and membership inference methods (c) (d).