Spotting the Artificial: Dialogic Humor Improvisation in Humans and LLMs
Organizations: Posgrado de lingüística, UNAM, Mexico · CEIICH, UNAM, Mexico
Abstract
We propose a methodology for investigating the differences between natural and synthetic language in a challenging setting: humorous improvisational dialogue. Our experimental setting compares human-produced comedy improvisations with the dialogues generated by LLMs in a multi-agent environment that we designed. The analysis combines automated linguistic measures with manual annotation grounded in Systemic Functional Linguistics (SFL). The results show that, although LLMs can create coherent and creative scenes, they often fail to reproduce key characteristics of spontaneous spoken interaction. In addition, LLM-generated interactions display a strong tendency toward politeness and positive evaluations, contrasting with the broader range of attitudes found in human humorous improvisation. Our evidence highlights both the strengths and limitations of current LLMs in modeling dialogic humor and demonstrate the value of integrating linguistic theory with NLP methods for the analysis of machine-generated language
Figures & tables
| Improvisation act | Turns |
|---|---|
| Guess Who | |
| All situations take place inside a Mexico City taxi. The driver must take each passenger to their destination and, over the course of the conversation, guess which character the passenger is portraying | 82 |
| Four Scenarios | |
| A groom celebrates his bachelor party, which unexpectedly takes place across three locations: a dog grooming salon, a massage parlor, and a funeral home. | 67 |
| Semantic Field | |
| The story takes place in an Argentine restaurant in Mexico, where a couple are celebrating their relationship. However, they unexpectedly encounter an ex-boyfriend | 60 |
| Improvisation act | Turns |
|---|---|
| Guess Who | 36 |
| Four Scenarios | 12 |
| Semantic Field | 12 |
| Talk show | 12 |
| Total tokens: | 23037 |
| Total number of tokens across repeated dialogue generations. | 155178 |
| Window size | ||||
|---|---|---|---|---|
| MATTR | 25 | 50 | 100 | 150 |
| Spontaneous (natural) | 0.80 | 0.71 | 0.62 | 0.56 |
| Synthetic | 0.91 | 0.82 | 0.70 | 0.62 |
| Natural | Synthetic | |
|---|---|---|
| Affect | 55.6% | 32.8% |
| Appreciation | 24.4% | 51.1% |
| Judgement | 20% | 16.1% |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Language Model Generation Evaluation
Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.
See all 420 papers