cs.CLApr 13, 2026
SaveYIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents
Organizations: InfoSense Lab, Georgetown University
Abstract
Most conversational agents (CAs) are designed to satisfy user needs through user-driven interactions. However, many real-world settings, such as academic interviewing, judicial proceedings, and journalistic investigations, involve broader institutional decision-making processes and require agents that can elicit information from users. In this paper, we introduce Information Elicitation Agents (IEAs) in which the agent's goal is to elicit information from users to support the agent's institutional or task-oriented objectives. To enable systematic research on this setting, we present YIELD, a 26M-token dataset of 2,281 ethically sourced, human-to-human dialogues. Moreover, we formalize information elicitation as a finite-horizon POMDP and propose novel metrics tailored to IEAs. Pilot experiments on multiple foundation LLMs show that training on YIELD improves their alignment with real elicitation behavior and findings are corroborated by human evaluation. We release YIELD under CC BY 4.0. The dataset, project code, evaluation tools, and fine-tuned model adapters are available at: https://github.com/infosenselab/yield.
Figures & tables
Figure 1: Traditional vs. information elicitation agents.
| Oral History Oglesby (2009) | Judicial Proceeding Supreme Court of the USA (2024) |
| Elicitor : "OK, then think back to when you went into the Peace Corps. What were you doing the year before you went in?" Respondent : "I was in high school when I went in. I was one of the eighteen year olds that went into the Peace Corps." Elicitor : "Did you have any particular country or any place that you wanted to go?" Respondent : "South America always had a calling for me. I didn’t have any particular country at all." | Elicitor : "Is there any aspect of social media that you think is protected by the First Amendment?" Respondent : "Yes, Your Honor. I can certainly imagine platforms that would be subject to this law that indeed have First Amendment rights." Elicitor : "But why is it different?" Respondent : "Well – well, there’s a lot in there, Your Honor. Maybe I can start with the Hurley case." |
| Academic Interview Taherzadeh (2016) | Journalistic Investigation Van Susteren (2018) |
| Elicitor : "I was just wondering your opinion on whether biodiversity offsetting is likely to lead to more stringent or less stringent regulations around infrastructure projects?" Respondent : "It’s ill-informed people not understanding the system properly because at the end of the day biodiversity offsetting doesn’t affect any existing protections." Elicitor : "What’s been the reception of businesses to biodiversity offsetting extending the mitigation hierarchy?" Respondent : "Well, the reception that I’ve had when I’ve been speaking to them has been really quite positive." | Elicitor : "This is your second trip to Mexico, but why are you here now in Mexico?" Respondent : "So, as the new government makes this transition beginning on Dec. 1, we’re working diligently to make sure we have a solid foreign policy relationship with them." Elicitor : "That caravan that is coming up from Guatemala heading to Mexico and then presumably maybe some place else, maybe the United States: What’s Mexico say about that?" Respondent : "So Mexico agrees that it is not constructive to have unlawful migration transit from these Northern Triangle countries through Mexico into the United States." |
Table 1: Shortened examples of information elicitation dialogues. Sources are included next to domain names.
| Dataset | # Domains | # Dialogues | Avg. Turns Per Dialogue | Avg. Tokens Per Turn | # Turns | # Tokens |
| DSTC2 | 1 | 1,612 | 14.49 | 8.54 | 23,354 | 199,431 |
| MultiWOZ | 7 | 8,438 | 13.46 | 13.13 | 113,556 | 1,490,615 |
| SGD | 16 | 16,142 | 20.44 | 9.75 | 329,964 | 3,217,149 |
| YIELD | 4 | 2,281 | 171.07 | 67.27 | 390,205 | 26,249,014 |
Table 2: Commonly used dialogue dataset statistics.
| Field | Description |
| dialogue_id | Unique dialogue identifier. |
| metadata | Metadata varies by collection. |
| broad_source | Collection’s primary archive. |
| domain | General content category. |
| title | Dialogue name (if available). |
| elicitors | Speaker’s names/identifiers. |
Table 3: Dataset schema.
| Domain | Broad Source | Dia- logues | Turns ( ) | Tokens ( ) |
| Academic | Harvard | 136 | 26.3 | 654.4 |
| Academic | Other | 12 | 1.2 | 56.7 |
| Journalistic | VOA | 73 | 2.2 | 150.2 |
| Journalistic | Wikinews | 56 | 3.2 | 172.5 |
| Judicial | Oyez | 621 | 162.9 | 7,655.5 |
| Oral | JFK | 270 | 83.0 | 3,056.4 |
Table 4: Domain and source statistics.
| Domain | Segments | Tokens | Tokens / Segment | Token Range | Reward Avg | Reward Range | RTG Avg | RTG Range |
| Academic | 11,036 | 2,255,573 | 204.4 | 81-512 | 0.3 | 0-29 | 2.5 | 0-36.5 |
| Journalistic | 1,331 | 414,870 | 311.7 | 99-511 | 1.3 | 0-26 | 8.7 | 0-117.1 |
| Judicial | 48,679 | 14,920,246 | 306.5 | 91-512 | 0.5 | 0-18 | 4.7 | 0-40.4 |
| Oral | 42,408 | 12,156,466 | 286.7 | 83-512 | 1.6 | 0-223 | 13.6 | 0-453.0 |
| Total | 103,454 | 29,747,155 | 277.3 | 81-512 | 0.9 | 0-223 | 7.4 | 0-453.0 |
Table 5: Experimental YIELD dataset statistics. Reward outliers in Oral History are due to prompts that led to the respondent sharing stories of significant length.
| Model | Ac | Jo | Ju | Or |
| Micro Perplexity (models vs real utterances) | ||||
| Llama-3.1-8B Base | 46.9 | 29.3 | 22.6 | 30.4 |
| Llama-3.1-8B SFT | 10.9 | 13.0 | 10.9 | 11.0 |
| Llama-3.1-8B ORL | 12.5 | 13.2 | 11.3 | 11.6 |
| DeepSeek-R1 Base | 78.5 | 60.5 | 38.0 | 62.8 |
| DeepSeek-R1 SFT | 12.4 | 15.6 | 12.3 | 12.4 |
Table 6: Conformity results (Ac = Academic, Jo = Journalistic, Ju = Judicial, Or = Oral).
| Model | Ac | Jo | Ju | Or |
| Progression | ||||
| Real | 0.713 | 0.751 | 0.744 | 0.755 |
| Shuffled | 0.824 | 0.848 | 0.800 | 0.821 |
| Llama-3.1-8B Prompt | 0.660 | 0.629 | 0.663 | 0.653 |
| Llama-3.1-8B SFT | 0.688 | 0.776 | 0.732 | 0.743 |
| Llama-3.1-8B ORL | 0.680 | 0.734 | 0.725 | 0.734 |
Table 7: Progression results (Ac = Academic, Jo = Journalistic, Ju = Judicial, Or = Oral). Real and Shuffled denote human dataset utterances.
| Model | Ac | Jo | Ju | Or |
| Turn-length Ratio | ||||
| Real | 1.672 | 3.051 | 1.227 | 3.187 |
| Shuffled | 0.969 | 1.002 | 0.883 | 0.954 |
| Llama-3.1-8B Prompt | 1.160 | 2.071 | 1.088 | 2.222 |
| Llama-3.1-8B SFT | 1.878 | 3.496 | 1.346 | 3.671 |
| Llama-3.1-8B ORL | 1.864 | 3.231 | 1.270 | 3.595 |
Table 8: Turn-length ratio results (Ac = Academic, Jo = Journalistic, Ju = Judicial, Or = Oral). Real and Shuffled denote human dataset utterances.
| P | CC | OR | PE | C | |
| Real | 3.89 | 3.75 | 3.92 | 3.71 | 3.97 |
| Prompted | 4.25 | 4.07 | 4.25 | 4.17 | 4.20 |
| SFT | 3.65 | 3.64 | 4.01 | 3.57 | 3.96 |
| ORL | 3.70 | 3.67 | 3.89 | 3.45 | 3.99 |
Table 9: Human evaluation results per response source. Metrics: P = Progression, CC = Conversational Control, OR = Outcome Relevance, PE = Probing Effectiveness, C = Conformity.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Copyright | Collections |
| CC0 1.0 | Covid Threshold Project Dataverse, Healthworker Interview Transcripts, Drivers of Food Choice, AI feedback: Moving beyond the hype, Relationship Building Around Farmers’ Market Incentive Programs (D.C. Market Manager Relationships), Leaders Leading Organizational Change, Biodiversity Offsetting |
| CC BY 4.0 | Flu Vaccination Interviews, Wikinews |
| CC BY-NC 4.0 | Oyez Supreme Court |
| Public Domain | National Archives History Office Interviews, National Personnel Records Center Oral History Interviews, Veterans Oral History Interviews, National Archives Assembly’s Legacy Project, Returned Peace Corps Volunteer Collection, VOA News |
Table A.1: Collections by copyright status.