cs.CLOct 6, 2026
SaveWiki-Talkie: Multilingual Benchmarking of Persona-Based Agents on Real-World Discussions
Organizations: Amazon
Abstract
LLMs are increasingly deployed as autonomous agents in social environments, making it critical to study their ability to faithfully simulate human interactions. Central to this is grounding agents in realistic user personas, yet existing datasets rely on fictional personas and are limited to a handful of languages, lacking the empirical grounding necessary to evaluate behavioral fidelity across diverse populations. We introduce Wiki-Talkie, a multilingual dataset of real-world conversations from Wikipedia Talk pages across five languages spanning two language families: Germanic (German, English) and Romance (Spanish, French, Italian), paired with personas derived from real user communities and encompassing sociodemographic attributes, self-descriptions, and behaviorally grounded interaction traits. Using Wiki-Talkie, we evaluate agent interactional behavior on a next-turn generation task across various persona conditioning strategies. Our evaluation assesses whether agents collectively reproduce the distributional behavioral patterns observed in human discussions. Results show that user's comment history exemplifying interaction behavior consistently outperforms explicit persona information. In addition, models systematically underproduce negative or extreme sentiments, while over producing references and suggestions, revealing biases toward agreeableness and positivity. Crucially, these patterns hold robustly across languages, with small cross-lingual differences.
Figures & tables
Figure 1: Example from Wiki-Talkie with its three core components: article , thread , and persona .
| # Threads | # Turns | # Users | Turns/Thread | Turns/User | Turn Length | Users/Thread | |
| de | 38,711 | 123,520 | 6,282 | 3.19 1.92 | 19.93 53.93 | 48.09 56.80 | 2.11 0.35 |
| en | 17,645 | 59,537 | 4,580 | 3.37 2.33 | 13.19 40.01 | 67.30 81.68 | 2.06 0.24 |
| es | 5,229 | 17,019 | 2,307 | 3.25 2.13 | 7.55 19.41 | 79.80 96.36 | 2.14 0.42 |
| fr | 23,983 | 86,041 | 4,952 | 3.59 2.37 | 17.59 58.68 | 66.32 79.56 | 2.26 0.61 |
| it | 5,453 | 18,956 | 1,850 | 3.48 2.35 | 10.39 27.04 | 66.00 84.43 | 2.18 0.46 |
Table 1: Summary statistics of Wiki-Talkie ground-truth threads, turns, users, and average metrics (mean std).
Figure 2: ROUGE-L and rescaled BERTScore between humans and agents across the five conditioning methods. Left: averages across models for each language. Right: averages across languages for each model.
Figure 3: JSD between human and agent dialog act distributions across conditioning methods. Left: averages across models for each language. Right: averages across languages for each model.
Figure 4: Proportion difference between human and agent dialog acts. Positive values indicate human over-production; negative values indicate agent over-production. Differences are averaged across languages.
Figure 5: Distribution of sentiment scores for human and agent (Model A under history ) comments for en .
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Distributions of words per turn, turns per user, turns per thread, users per thread across the five languages.
Figure 7: Word distributions for user pages and profiles.
| Total Users | with Topics | with Gender | with Profession | with >= 5 Turns | Profile Length | |
| de | 6,282 | 6,213 | 2,233 | 1,620 | 2,983 | 60.14 27.42 |
| en | 4,580 | 4,580 | 3,062 | 3,038 | 1,874 | 98.07 46.22 |
| es | 2,307 | 1,309 | 891 | 446 | 671 | 57.07 30.73 |
| fr | 4,952 | 2,646 | 2,280 | 1,494 | 1,986 | 64.80 35.23 |
| it | 1,850 | 1,089 | 882 | 381 | 678 | 56.06 31.95 |
Table 2: Summary statistics of users, their attributes, and activity metrics (mean std for profile length).
Figure 8: Proportion of unavailable trait values across all users, broken down by trait and language.
| # Comments | Avg. Prev. Len ( Std) | # Unique Users | # Men | # Women | # Men Sent. | # Women Sent. | |
| de | 5,187 | 3.05 ( 2.59) | 564 | 195 | 55 | 1,638 | 1,212 |
| en | 8,658 | 3.39 ( 3.05) | 561 | 296 | 83 | 3,268 | 3,277 |
| es | 4,310 | 3.03 ( 2.69) | 298 | 83 | 20 | 1,129 | 458 |
| fr | 9,827 | 3.29 ( 2.88) | 682 | 250 | 61 | 2,535 | 2,566 |
| it | 4,706 | 3.19 ( 2.92) | 325 | 131 | 15 | 1,989 | 178 |
Table 3: Statistics of the sampled data used for experiments.
| Original Label | Our Label |
| CM, CW, CU, CS, CL, COBJ, CO | Critique |
| PSR | Request |
| PREF | Reference |
| PFC | Commit |
| PPC | Report |
| IP, IC | Inform |
Table 4: Mapping from original SEWD labels ( Ferschke et al., 2012 ) to our dialog act taxonomy.
| # Comments | Micro-F1 | Ask | Commit | Critique | Inform | Negative | Partial | Positive | Reference | Report | Request |
| All | 0.74 | 0.82 | 0.75 | 0.77 | 0.59 | 0.68 | 0.35 | 0.80 | 0.49 | 0.97 | 0.81 |
| Three | 0.74 | 0.84 | 0.79 | 0.78 | 0.58 | 0.65 | 0.19 | 0.79 | 0.55 | 0.98 | 0.83 |
| Two | 0.76 | 0.85 | 0.80 | 0.81 | 0.59 | 0.65 | 0.42 | 0.79 | 0.50 | 0.97 | 0.84 |
| One | 0.75 | 0.83 | 0.83 | 0.82 | 0.59 | 0.67 | 0.36 | 0.78 | 0.49 | 0.97 | 0.84 |
Table 5: Micro-F1 and F1 scores per label across different training settings.
| Language: German Page: Diskussion:Schwicheldt (Adelsgeschlecht) Thread: “Bezug” | |
| User | Profile |
| 8bfc7f759c | Der Benutzer ist seit dem 17. September 2004 bei Wikipedia registriert und feierte am 17. September 2025 seinen 21. Wiki-Geburtstag. Er hat insgesamt rund 284.000 Bearbeitungen vorgenommen, etwa 1100 Artikel verfasst und ungefähr 8400 Fotos hochgeladen. Er ist unter den 8000 aktivsten deutschsprachigen Wikipedianern. Der Benutzer erhielt 2016 den Denkmalpreis ’Silberne Halbkugel’. Er ist Mitglied im Verein Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens. Seine Beiträge konzentrieren sich auf Themen wie Bessarabien, den Wolfsburger und Helmstedter/Gifhorner Raum, Hannover, Hann. Münden sowie Archäologie, Geschichte, Burgen, Schlösser, Kirchen und lokale Denkmäler in Niedersachsen. |
| 4f3806c190 | Der Benutzer ist ein deutscher Muttersprachler mit fortgeschrittenen Englischkenntnissen. Er hat sich am 19. Dezember 2004 bei Wikipedia registriert und kümmert sich hauptsächlich um historischen Bergbau, insbesondere um den Zwickauer Steinkohlenbergbau. Er hat zahlreiche Artikel zu bergbauhistorischen und bergbautechnischen Themen erstellt sowie zu verschiedenen anderen Themen wie Waffen, Militärtechnik und Flugzeugen beigetragen. |
| Author | Comment |
| 8bfc7f759c | Besteht ein Bezug zum Ort [[Schwicheldt]]? |
| 4f3806c190 | Wenn man dem Ortsartikel glauben darf: ja. |
Table 6: German example: persona profiles and thread.
| Language: English Page: Talk:Cleveland Bay Thread: “Merge” | |
| User | Profile |
| 5f962a065d | The user lives in Montana and owns horses. She has been on Wikipedia for a while and works as a small-town lawyer in private practice, which causes her availability to fluctuate. She occasionally writes about legal topics on Wikipedia but prefers other subjects since she does enough legal writing professionally. She also writes books and does freelance writing outside of Wikipedia. She is not compensated for her Wikipedia contributions. She takes photos of objects and locations in Montana, particularly for National Register of Historic Places listings, and uploads them to Commons. She can access documents at the Montana Historical Society. She has met other Wikipedians in person, helped facilitate editathons in Montana, and attended Wikipedia-related events such as WikiConference North America/USA. Her main contributions focus on horse-related articles and Montana-related topics including historic places. |
| 95725d4ec5 | The user is a woman whose main focus on Wikipedia is equine articles, although she also contributes to topics related to history, astronomy, books, endurance sports, the outdoors, and labor unions. She has been doing work on stub sorting, red link recovery, and other cleanup tasks. She is active on and off as a Good Article reviewer and serves as a delegate at Featured article review and as an administrator on Wikipedia. She is married. |
| Author | Comment |
| 5f962a065d | I’m only wondering if the Chapman horse contributed significantly to anything other than the Cleveland Bay, thus warranting its continued independent article. (Thinking of the Narragansett Pacer or Old English Black horse). Other than that, no problem with the merge, content seems similar in both. |
| 95725d4ec5 | I’ve never found a source that references the Chapman horse in relation to any breed other than the Cleveland Bay. All of the sources I combed through for this article seemed to think that the Chapman was basically the earliest incarnation of the Cleveland Bay, rather than a separate breed that they were descended from. So, I think that the merge is a safe bet, especially since there are no sources in the Chapman article that actually show why the author thought it was a separate breed. |
Table 7: English example: persona profiles and thread.
| Language: Spanish Page: Discusión:Calor específico Thread: “Número de citas” | |
| User | Profile |
| e1c14a406a | El usuario expresa interés en tres áreas principales: la lingüística y las lenguas del mundo, la física matemática y la ingeniería, y la historia económica, economía y sociología. Tiene enlaces a sus páginas de usuario en las Wikipedias en alemán, inglés, francés y portugués. |
| 94a136608d | El usuario es un antiguo bibliotecario peruano que se identifica como miembro de ACAD. Está agradecido con la Wikipedia en español. Sus intereses incluyen Moquegua, las Olimpiadas y la Química. |
| 4901fd71f4 | El usuario es un hombre de Quito, Ecuador, que trabaja para el experimento CMS del CERN. Tiene un doctorado en Física Experimental de Partículas. En el pasado dedicaba mucho tiempo a editar Wikipedia en español, pero actualmente lo hace cuando puede debido a la falta de tiempo libre. Ha contribuido a numerosos artículos relacionados con física, incluyendo mecánica cuántica, teoría de la relatividad, física de partículas y electromagnetismo. |
| Author | Comment |
| e1c14a406a | A petición de otro wikipedista leí extensamente el artículo, fijándome en especial en el número de afirmaciones que podrían requerir cita. Aunque las citas son algo deseable y casi siempre necesarias en este caso concreto no veo demasiadas afirmaciones extraordinarias o polémicas que requieran citas. En un artículo sobre temas polémicos las citas son más necesarias, pero creer que todo artículo sea del tema que se requiere el mismo número de citas es erróneo. En este caso el artículo trata de un tema de ciencia convencional muy asentado que podríamos considerar un tema ya muy asentado, en ese caso creo que al artículo no le faltan ya más citas. Tener citas por tener citas, no garantiza la calidad o correcta referenciación del artículo, el número de citas debe depender el tipo de artículo, la temática y lo polémico del tema. Mi apreciación es que en este caso el artículo tiene el número de citas adecuadas. En cualquier caso si alguien considera que no es así debería añadir QUÉ afirmaciones concretas considera que deben ser referenciadas. |
Table 8: Spanish example: persona profiles and thread.
| Language: French Page: Discussion:Pfalz-ardenner/Bon article Thread: (untitled) | |
| User | Profile |
| 702383d34d | L’utilisateur a commencé à contribuer sur Wikipédia en décembre 2006. Il se décrit comme un wikignome à ses heures perdues et travaille principalement sur des articles liés à la physique nucléaire, aux accélérateurs de particules, aux biographies de scientifiques et à l’Irlande. Il a significativement contribué à des articles sur la capture électronique, la désintégration du neutron libre et les noyaux p. |
| d637e7f14a | L’utilisatrice est une femme inscrite sur Wikipédia depuis 2007. Elle est une contributrice très active, ayant lancé environ 230 propositions de labellisation d’articles, dont 83% concernent le cheval. Elle a participé à plusieurs publications scientifiques. Ses domaines de contribution principaux sont les créatures légendaires (fées, lutins, malebestes), les chevaux (aspects culturels, sociologiques et génétiques), et l’autisme. Elle s’intéresse de plus en plus à la génétique. Elle apprécie les tableaux préraphaélites. Depuis 2017, elle pratique la photographie animalière comme hobby, cherchant à photographier un maximum d’espèces et de races animales. Elle préfère être tutoyée. |
| 75d8423c61 | L’utilisateur est passionné de sport automobile, plus particulièrement d’endurance. Il est né le jour du Grand Prix automobile du Portugal 1995. Il s’intéresse aux histoires oubliées du sport automobile. Sur Wikipédia, il contribue activement en créant et améliorant de nombreux articles liés au sport automobile, notamment sur les automobiles de course, les courses d’endurance, les écuries, les pilotes, ainsi que sur d’autres sujets comme la course au large, le football, l’histoire militaire et l’architecture. |
| Author | Comment |
| 702383d34d | Très beau travail de recherche {{u-|user_d637e7f14a}}. Je n’ai fait que survoler l’article mais une remarque me vient au passage. Étant donné que quasiment toutes les sources sont en anglais ou en allemand, d’où vient le nom en français de cette race. Est-ce qu’il est mentionné dans Guide des chevaux d’Europe qui est la seule source en français ? Ou bien est-ce qu’il s’agit de ta propre traduction ? Je demande cela car les seuls résultats que je trouve sur le Web pour ce mot proviennent de Wikipédia ou de clones. |
Table 9: French example: persona profiles and thread.
| Language: Italian Page: Discussione:Nevicata del 1985 Thread: “Leggiamo l’articolo” | |
| User | Profile |
| 6aa9673768 | L’utente si è registrato su Wikipedia l’8 agosto 2006. Contribuisce principalmente nell’ambito delle voci scientifiche, con particolare attenzione alle Scienze della Terra, e interviene anche su voci riguardanti storiografia, filosofia e cinema, che sono sue vecchie passioni. È anche appassionato di fotografia e inserisce immagini originali di sua produzione. Partecipa al Progetto sulle persecuzioni, deportazioni e crimini del periodo nazi-fascista, in particolare sul tema del Negazionismo dell’Olocausto. È amministratore di Wikipedia dal 2007. |
| 677e695641 | L’utente è un uomo over 60, curioso e intellettualmente aperto a nuovi apprendimenti. Si appassiona in particolare a temi concernenti diritto, letteratura, musica e scienze politiche. Si è registrato su Wikipedia verso la fine di febbraio 2006. È laureato in Giurisprudenza. Vive a La Spezia. Su Wikipedia ha creato o modificato consistentemente voci riguardanti società, diritto, economia, storia, istituzioni, politica, musica, letteratura, arte, religione, filosofia e sport. |
| 1363852f3c | L’utente è attivo su Wikipedia dal 21 agosto 2006, quando ha effettuato la sua prima modifica. L’11 febbraio 2008 è stato eletto amministratore. Ha raggiunto diversi traguardi di contributi: 100 edit il 24 marzo 2007, 500 il 19 giugno 2007, 1000 il 21 luglio 2007, 2000 l’8 agosto 2007, 4000 il 22 settembre 2007, 5000 il 17 ottobre 2007, 7500 il 18 gennaio 2008, 10000 il 13 aprile 2008 e 20000 il 16 aprile 2009. Ha creato 100 nuove voci entro il 21 ottobre 2007. Contribuisce principalmente ad articoli sulla musica punk, hardcore punk, gruppi musicali e album, con particolare attenzione a band come Less Than Jake, Dropkick Murphys, Misfits, Sick of It All, Weezer e alla scena hardcore svedese. |
| Author | Comment |
| 6aa9673768 | Nell’ ultimo intervento user_677e695641 ha richiesto di inserire una citazione gia’ presente da almeno una settimana nelle fonti/bibliografia. Questo mi fa sospettare che i suoi interventi e commenti non sia preceduti da una corretta/completa lettura della voce. Lascio a chi legge i commenti del caso. |
Table 10: Italian example: persona profiles and thread.
Figure 9: Prompt template for thread split revision. Few-shot example content is summarized for brevity.
Figure 10: Prompt template for user page profiling.
Figure 11: Prompt template for editorial behavior extraction (system prompt and user prompt, part 1).
Figure 12: Prompt template for dialog act labeling.
Figure 13: Prompt template for editorial behavior extraction (user prompt, part 2).
Figure 14: Prompt template for base .
Figure 15: Prompt template for profile .
Figure 16: Prompt template for behavior ( {user_information} without reasoning) and behavior+r ( {user_information} with reasoning).
Figure 17: Prompt template for history .
Figure 18: Proportion difference between human and agent dialog acts split across genders. Positive values indicate human over-production; negative values indicate agent over-production.
Figure 19: Sentiment scores distributions for human agent comments on en for selected models and conditioning methods.
| Setup | Lang | Model | Avg Human Words | Avg Agent Words | BERT | rBERT | ROUGE |
| base | de | Model A | 47.9 (56.3) | 80.2 (34.2) | 0.849 | 0.065 | 0.096 |
| Model B | 47.7 (56.3) | 205.1 (74.3) | 0.822 | -0.102 | 0.061 | ||
| Model C | 47.9 (56.3) | 87.7 (32.5) | 0.847 | 0.051 | 0.088 | ||
| Model D | 47.9 (56.3) | 121.7 (47.4) | 0.837 | -0.013 | 0.077 | ||
| Avg models | 47.8 0.1 | 123.7 57.2 | 0.839 0.012 | 0.000 0.076 | 0.081 0.015 | ||
| en | Model A | 62.4 (78.7) | 105.4 (54.3) | 0.854 | 0.013 | 0.124 |
Table 11: Full results (part 1). Rows grouped by setup and language. Avg Human Words varies slightly across models and setups due to rare cases where human comments were excluded from evaluation owing to missing generated counterparts (generation errors or hallucinations).
| Setup | Lang | Model | Avg Human Words | Avg Agent Words | BERT | rBERT | ROUGE |
| behavior | fr | Model A | 63.9 (77.2) | 133.6 (67.5) | 0.846 | 0.024 | 0.107 |
| Model B | 63.7 (77.2) | 199.8 (78.2) | 0.829 | -0.081 | 0.086 | ||
| Model C | 63.9 (77.2) | 124.6 (54.2) | 0.847 | 0.032 | 0.106 | ||
| Model D | 63.9 (77.2) | 194.5 (115.4) | 0.836 | -0.037 | 0.092 | ||
| Avg models | 63.8 0.1 | 163.1 39.6 | 0.839 0.008 | -0.016 0.054 | 0.098 0.011 | ||
| it | Model A | 64.1 (79.1) | 130.8 (70.9) | 0.842 | 0.042 | 0.090 |
Table 12: Full results (part 2). Rows grouped by setup and language. Avg Human Words varies slightly across models and setups due to rare cases where human comments were excluded from evaluation owing to missing generated counterparts (generation errors or hallucinations).
| Sentiment | Dialog Acts | ||||||||||||||
| WD | JS | Density Ratio | CRIT | NEG | POS | PART | INF | REP | COM | REQ | ASK | REF | |||
| base | de | Model A | 0.236 | 0.147 | 0.700 | 0.002 | 0.051 | -0.027 | -0.009 | 0.042 | 0.040 | 0.005 | -0.107 | 0.003 | 0.000 |
| Model B | 0.565 | 0.270 | 0.549 | 0.018 | 0.077 | 0.034 | 0.007 | 0.078 | 0.057 | -0.017 | -0.165 | 0.036 | -0.123 | ||
| Model C | 0.329 | 0.259 | 0.733 | 0.028 | 0.073 | -0.061 | -0.021 | 0.069 | 0.081 | -0.027 | -0.151 | -0.019 | 0.029 | ||
| Model D | 0.459 | 0.207 | 0.733 | -0.010 | 0.035 | -0.000 | -0.000 | -0.020 | 0.066 | -0.028 | -0.100 | 0.069 | -0.013 | ||
| Avg | 0.397 0.145 | 0.221 0.056 | 0.679 0.088 | 0.009 | 0.059 | -0.014 | -0.006 | 0.042 | 0.061 | -0.017 | -0.131 | 0.022 | -0.027 | ||
Table 13: Dialog acts (part 1). JS = Jensen-Shannon distance on label proportions. Density Ratio = human labels/sentence agent labels/sentence. Dialog Acts = human proportion agent proportion for each label (positive agent under-produces).
| Sentiment | Dialog Acts | ||||||||||||||
| WD | JS | Density Ratio | CRIT | NEG | POS | PART | INF | REP | COM | REQ | ASK | REF | |||
| behavior | fr | Model A | 0.358 | 0.222 | 0.615 | 0.007 | 0.055 | -0.004 | -0.001 | 0.115 | 0.075 | -0.007 | -0.120 | -0.069 | -0.051 |
| Model B | 0.508 | 0.246 | 0.511 | 0.007 | 0.073 | 0.035 | 0.030 | 0.125 | -0.011 | -0.040 | -0.124 | 0.025 | -0.120 | ||
| Model C | 0.323 | 0.280 | 0.689 | 0.038 | 0.076 | -0.057 | -0.050 | 0.106 | 0.088 | -0.012 | -0.127 | -0.070 | 0.009 | ||
| Model D | 0.457 | 0.291 | 0.524 | 0.004 | 0.064 | 0.034 | 0.008 | 0.114 | 0.068 | -0.108 | -0.133 | 0.054 | -0.106 | ||
| Avg | 0.411 0.086 | 0.260 0.031 | 0.585 0.084 | 0.014 | 0.067 | 0.002 | -0.003 | 0.115 | 0.055 | -0.042 | -0.126 | -0.015 | -0.067 | ||
Table 14: Dialog acts (overall). JS = Jensen-Shannon distance on label proportions. Density Ratio = human labels/sentence agent labels/sentence. Dialog Acts = human proportion agent proportion for each label (positive agent under-produces).
Figure 20: ROUGE-L between agent and human comments under profile conditioning, stratified by profile word count. Each row is a language; each column is a model. Brown solid line: ROUGE-L (left axis). Red dashed line: proportion of samples in each bucket (right axis).
Figure 21: Rescaled BERTScore (rBERT) between agent and human comments under profile conditioning, stratified by profile word count. Conventions as in Figure 20 .
Figure 22: Dialog act JSD between agent and human distributions under profile conditioning, stratified by profile word count. Lower is better. Conventions as in Figure 20 .
Figure 23: Sentiment JSD between agent and human distributions under profile conditioning, stratified by profile word count. Lower is better. Conventions as in Figure 20 .