cs.CLMar 31, 2026
SaveImpact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics
Organizations: Speech Interactive Group, University of the Basque Country (UPV/EHU), Sarriena s/n, 48940 Leioa, Spain
Abstract
Conversational systems should generate diverse language forms to interact fluently and accurately with users. In this context, Natural Language Generation (NLG) engines convert Meaning Representations (MRs) into sentences, directly influencing user perception. These MRs usually encode the communicative function (e.g., inform, request, confirm) via DAs and enumerate the semantic content with slot-value pairs. In this work, our objective is to analyse whether providing a task demonstrator to the generator enhances the generations of a fine-tuned model. This demonstrator is an MR-sentence pair extracted from the original dataset that enriches the input at training and inference time. The analysis involves five metrics that focus on different linguistic aspects, and four datasets that differ in multiple features, such as domain, size, lexicon, MR variability, and acquisition process. To the best of our knowledge, this is the first study on dialogue NLG implementing a comparative analysis of the impact of MRs on generation quality across domains, corpus characteristics, and the metrics used to evaluate these generations. Our key insight is that the proposed enriched inputs are effective for complex tasks and small datasets with high variability in MRs and sentences. They are also beneficial in zero-shot settings for any domain. Moreover, the analysis of the metrics shows that semantic metrics capture generation quality more accurately than lexical metrics. In addition, among these semantic metrics, those trained with human ratings can detect omissions and other subtle semantic issues that embedding-based metrics often miss. Finally, the evolution of the metric scores and the excellent results for Slot Accuracy and Dialogue Act Accuracy demonstrate that the generative models present fast adaptability to different tasks and robustness at semantic and communicative intention levels.
Figures & tables
| MR | Sentence |
|---|---|
| inform (name = The Vaults ; eatType = pub ; priceRange = more than £30 ; customerRating = 5 out of 5 ; near = Café Adriatic) | The Vaults pub near Café Adriatic has a 5 star rating. Prices start at £30. |
| inform ( name = The Cambridge Blue ; eatType = pub ; food = English ; priceRange = cheap ; near = Café Brazil ) | Close to Café Brazil, The Cambridge Blue pub serves delicious Tuscan Beef for the cheap price of £10.50. Delicious Pub food. |
Table 1: Samples of the E2E dataset.
| MR | Sentence |
|---|---|
| inform ( name = The Forest of Doom ; release_year = 2014 ; genres = role-playing, text adventure ; has_multiplayer = no ) | The Forest of Doom is a role-playing text adventure game released in 2014. It is a single-player only game. |
| give_opinion ( name = Undertale ; esrb = E (for Everyone) ; rating = excellent ) | Undertale is a downright excellent game, and it’s rated E so everyone can enjoy it. |
| request_attribute ( has_multiplayer = ? ) | Do you prefer to game with others, I mean, multiplayer? |
Table 2: Samples of the ViGGO corpus
| MR | Sentence |
|---|---|
| SYSTEM_Booking_Book ( bookday = Wednesday ; bookpeople = five ; booktime = 11 am ; name = this restaurant ; ref = DU8IWQZ2 ) & SYSTEM_Restaurant_Inform ( address = 2 Rose Crescent City Centre ; name = The Gardenia ) | The Gardenia is located at 2 Rose Crescent City Centre. I have booked your table at this restaurant for five people, Wednesday at 11 am. Your reference number is DU8IWQZ2. |
| USER_general_thank ( ) | No that will be all, thank you. |
Table 3: Samples of the MultiWOZ corpus
| MR | Sentence |
|---|---|
| Gen_Hello ( user_name = Elisabeth ) | Hello Elisabeth! |
| Int_know_coaching ( ) | Do you know what coaching is? |
| GSQ_what_obj ( action = change ) | Would you like to change anything? |
| RQ_curr_sit ( food = water ) | Do you think you’re drinking enough water? |
Table 4: Samples of the EMPATHIC corpus
| Dataset | Domains | Acquisition process | Number of DAs | Number of attributes | Number of MRs | Corpus size | Running words | Vocabulary |
|---|---|---|---|---|---|---|---|---|
| E2E | Restaurants | Crowd | 1 | 8 | 6.0k | 51k | 1.0M | 5.1k |
| ViGGO | Video games | Crowd | 9 | 14 | 2.3k | 6.9k | 150k | 4.4k |
| MultiWOZ | 7 domains | WOZ + Crowd | 798 (52) | 25 | 54k | 140k | 1.9M | 30k |
| EMPATHIC | Coaching | WOZ | 78 | 21 | 1.2k | 2.3k | 13k | 1.7k |
Table 5: Characteristics of the four selected corpora. Domains stands for the topics the datasets cover. Acquisition process means the methodology to obtain the datasets. Numbers of DAs, attributes, and MRs quantify the diversity of inputs. Corpus size stands for the number of MR-sentence pairs. Running words include all the words in the corpus, whereas Vocabulary is the number of different words. In the Number of DAs for MultiWOZ, 798 represents the total number of DAs that appear as a combination of the 52 simple DAs defined for the corpus.
Figure 1: Meaning representations overview. One Baseline representation and three Prompt representations. The Baseline representation, which is the MR found in the original datasets, is the input MR (orange boxes) for the generation in all the representations. Prompt representations are created by connecting a task demonstrator with the input MR . This task demonstrator includes an MR, the demonstator MR (green), and its corresponding example sentence, the demonstrator sentence (grey). The colours of the boxes, the letters n and m for the number of the attributes and the use (or not) of apostrophes inside Input and Demonstrator MR boxes are used to indicate which elements are necessarily identical (or not) among the representations.
| Dataset | Example type | Examples |
| E2E | Input MR | inform ( name = Zizzi ; eatType = coffee shop ; customer rating = high ; near = Burger King ) |
| Prompt 2 demonstrator | inform ( name = Loch Fyne ; eatType = restaurant ; familyFriendly = yes ; food = English ) + A child friendly restaurant that has English food is Loch Fyne. | |
| Prompt 3 demonstrator | inform ( name = Wildwood ; near = Café Rouge ; customer rating = low ; eatType = restaurant ) + Wildwood, a restaurant with a low customer rating, is near Café Rouge. | |
| ViGOO | Input MR | give_opinion ( name = Super Mario World ; player_perspective = side view ; rating = excellent ; has_multiplayer = yes ) |
| Prompt 1 demonstrator | give_opinion ( name = Guitar Hero: Smash Hits ; rating = poor ; developer = Beenox ) + Guitar Hero: Smash Hits is just not my kind of game. I find Beenox games are often not very fun. | |
| Prompt 2 demonstrator | give_opinion ( name = Worms: Reloaded ; player_perspective = side view ; rating = average ; available_on_Steam = yes ) + Worms: Reloaded is an average quality side view game you can at least find on Steam. |
Table 6: Examples of task demonstrators linked to an input MR for each dataset. DAs and attributes are in bold and coloured to highlight identical elements between the demonstrator MRs and the input MR . Note that brown colour is employed for those attributes that do not appear in the input MR , but they do in any demonstrator MR . Prompt 1 was discarded for E2E because the selection methodology would assign the same task demonstrator for all the E2E MRs.
| Dataset | Prompt 1 | Prompt 2 | Prompt 3 |
|---|---|---|---|
| E2E | - 7 7 7 There is no Prompt 1 representation for E2E, because E2E only has one DA, so all the MRs from the original corpus would be connected to the same demonstrator for this representation. | 6 | 120 |
| ViGOO | 9 | 21 | 473 |
| MultiWOZ | 798 | 2281 | 11644 |
| EMPATHIC | 78 | 197 | 537 |
Table 7: Number of different demonstrators in each dataset for the three Prompt representations.
Table 8: Average scores after fine-tuning for each reference-driven metric, representation, and dataset. Base corresponds to the baseline representation, whereas Pi is the Prompt i representation with i=1,2,3. The datasets (Data) are E2E, ViGGO (ViG), MultiWOZ (MWOZ), and EMPATHIC (EMP). No Prompt 1 for E2E. The best score for each dataset is denoted in bold.
Figure 2: Distribution of the MR scores of each reference-driven metric after the fine-tuning for each representation in each dataset.
Figure 3: Evolution of the average score of each reference-driven metric during the fine-tuning for each representation in each dataset.
| Data | Base | P1 | P2 | P3 |
|---|---|---|---|---|
| E2E | 0.79 | - | 0.65 | 0.67 |
| ViG | 0.80 | 0.69 | 0.72 | 0.72 |
| MWOZ | 0.94 | 0.91 | 0.91 | 0.91 |
| EMP | 0.88 | 0.86 | 0.87 | 0.90 |
Table 9: Slot accuracy average scores after fine-tuning for each representation and dataset. Base corresponds to the Baseline representation, whereas Pi is the Prompt i representation with i=1,2,3. The datasets (Data) are E2E, ViGGO (ViG), MultiWOZ (MWOZ) and EMPATHIC (EMP). No Prompt 1 for E2E. The best score for each dataset is denoted in bold.
Figure 4: Distribution of the Slot Accuracy MR scores after fine-tuning for each representation in each dataset.
Figure 5: Evolution of the average scores of Slot Accuracy during the fine-tuning for each representation in each dataset.
| Data | Val | Base | P1 | P2 | P3 |
|---|---|---|---|---|---|
| ViG | 0.98 | 0.99 | 0.65 | 0.89 | 0.94 |
| MWOZ | 0.85 | 0.86 | 0.52 | 0.68 | 0.83 |
| EMP | 0.68 | 0.65 | 0.72 | 0.70 | 0.70 |
Table 10: Dialogue Act Accuracy average scores after fine-tuning for each representation in each dataset. Val and Base correspond to the Validation dataset and Baseline representation, respectively, whereas Pi are the Prompt i representations with i=1,2,3. The datasets (Data) are ViGGO (ViG), MultiWOZ (MWOZ) and EMPATHIC (EMP). The best score for each dataset is denoted in bold, the Validation result is excluded from this comparison.
Figure 6: Evolution of the Dialogue Act Accuracy average scores during the fine-tuning for each representation in each dataset. No E2E as it only contains the inform DA.
Figure 7: MR distributions in terms of the number of references/attributes. The number of references/attributes is given on the x-axis with specific numbers of references/attributes or intervals. The distributions are in percentages.
Figure 8: Results for BLEU (top) and LaBSE (bottom) scores in terms of the number of references per MR with the fine-tuned models. The x-axis establishes specific numbers of references or intervals of numbers of references.
Figure 9: Results for BLEU (top) and LaBSE (bottom) scores in terms of the number of attributes per MR with the fine-tuned models.
| MR | inform ( name = Crysis ; release_year = 2007 ; esrb = M (for Mature) ; rating = good ; genres = action-adventure, shooter ; has_multiplayer = yes ) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| References | 3 references: - A solid M-rated action-adventure shooter, Crysis, came out in 2007. The game offers multiplayer. - An action-adventure shooter game for mature audiences is the 2007 game, Crysis. It was a fun game offering also a multiplayer mode. - Crysis is an action-adventure shooter that was released in 2007, rated M (for Mature). It is considered good quality, and has multiplayer mode. | ||||||||
| Generation | Crysis is an action-adventure shooter released in 2007. It has received good reviews. | ||||||||
| Dataset | ViGOO | BLEU | 0.06 | BLEURT | 0.66 | LaBSE | 0.81 | Slot acc | 0.80 |
| MR | Gen_Open_question ( action = cook ; food = fish ) | ||||||||
| References | 1 reference: What’s your favourite fish to cook? | ||||||||
| Generation | Do you like to cook fish? | ||||||||
Table 11: Examples to compare BLEU with BLEURT and LaBSE . Slot Accuracy (Slot acc) is also included. Each example contains, from top to bottom: MR, references, a generation, as well as the dataset and scores for all the metrics in the last row. Relevant elements are highlighted in bold and different colours as follows: orange for the headers related to the inputs, purple for those related to the outputs, blue for the datasets, red for BLEU scores, and green for BLEURT and LaBSE scores.
| MR | inform ( name = Tony Hawk’s Pro Skater 3 ; release_year = 2001 ; esrb = T (for Teen) ; genres = sport ; player_perspective = third person ) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| References | 3 references: - Tony Hawk’s Pro Skater 3 was released in 2001. It is a third person sports game that is rated T (for Teen). - Tony Hawk’s Pro Skater 3 was a third person sport game released in 2001. It was rated T (for Teen). - Another third person perspective sport game is Tony Hawk’s Pro Skater 3. It came out in 2001 and is rated T (for Teen). | ||||||||
| Generation | The 2001 sport game, Tony Hawk’s Pro Skater 3, was a third person game. It has a t (for teen) rating and is rated t (for physical). | ||||||||
| Dataset | ViGGO | BLEU | 0.34 | BLEURT | 0.61 | LaBSE | 0.92 | Slot acc | 1.00 |
| MR | Gen_Agreement ( ) | ||||||||
| References | 91 references: - Absolutely - Of course. - Yeah - Right - I totally understand. - (86 references more) | ||||||||
| Generation | Sure. | ||||||||
Table 12: Examples to compare BLEURT with LaBSE . BLEU and Slot Accuracy (Slot acc) are also included. Each example contains, from top to bottom: MR, references, a generation, as well as the dataset and scores for all the metrics in the last row. Relevant elements are highlighted in bold and different colours as follows: orange for the headers related to the inputs, purple for those related to the outputs, blue for the datasets, red for BLEURT scores, and green for LaBSE scores.
| MR | inform ( name = HSS ; genres = adventure, hack-and-slash, indie ; platforms = PlayStation, Xbox, PC ; available_on_steam = yes ; has_linux_release = no ; has_mac_release = no ) | ||||||||
| References | 3 references: - HSS is an indie adventure, hack-and-slash game for PlayStation, Xbox, and PC. It is available for Steam, however, it is not out for Linux or even Mac. - HSS is an indie hack-and-slash adventure game available on Steam. You can play it on PlayStation, Xbox, and PC, but not on Linux or Mac. - HSS is an indie, adventure, hack-and-slash game that’s available on PlayStation, Xbox and PC. You can also get it from Steam, but it does not have Linux or Mac releases. | ||||||||
| Prompt 3 task demonstrator | inform ( name = The Sims ; genres = simulation, strategy ; platforms = PlayStation, Xbox, Nintendo, PC ; available_on_steam = no ; has_linux_release = no ; has_mac_release = yes ) + The Sims is a simulation-strategy game that is available for PlayStation, Xbox, Nintendo, and PC. It can operate on Mac, but it is not released for Linux. It is not available on Steam. | ||||||||
| Baseline Generations | 15 generations: - HSS is an indie adventure hack-and-slash game. It’s available on the following platforms: PlayStation, Xbox, and PC. Steam is also the home of the game, though it isn’t available for Linux or Mac. - If you’re into indie adventure and hack-and-slash games, try HSS. It’s an indie game released for PlayStation, Xbox, PC, and Mac. You can get it on Steam now. - If you like indie hack-and-slash adventures on the PC, try HSS. It’s available on Steam, and you can play it on all platforms, such as the PlayStation, Xbox, and PC. There was no Linux or Mac release. - (12 generations more) | ||||||||
| Prompt 3 Generations | 15 generations: - HSS is an indie adventure-shooter with puzzle solving elements. - HSS is an indie adventure-hack and slash with a Mac twist. - HSS is an indie adventure-hack-and-slash on the PC that has been released on Steam. - (12 generations more) | ||||||||
| Dataset | ViGGO | BLEU | 0.31 / 0.15 | BLEURT | 0.70 / 0.56 | LaBSE | 0.89 / 0.68 | Slot acc | 0.82 / 0.33 |
| MR | Gen_Open_topic ( hobbies = travel ) | ||||||||
Table 13: Examples of the generations obtained with the Baseline and Prompt 3 representations. Each example contains from top to bottom: MR, references, the task demonstrator linked to the MR in the Prompt 3 representation, generations obtained with the Baseline representation, generations obtained with the Prompt 3 representation, and dataset and scores for all the metrics for both representations (Baseline/Prompt 3) in the last row. The highest score for each metric is highlighted in bold.
More in this topic
Text Generation
Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.
See all 422 papers