We propose a methodology for investigating the differences between natural and synthetic language in a challenging setting: humorous improvisational dialogue. Our experimental setting compares human-produced comedy improvisations with the dialogues generated by LLMs in a multi-agent environment that we designed. The analysis combines automated linguistic measures with manual annotation grounded in Systemic Functional Linguistics (SFL). The results show that, although LLMs can create coherent and creative scenes, they often fail to reproduce key characteristics of spontaneous spoken interaction. In addition, LLM-generated interactions display a strong tendency toward politeness and positive evaluations, contrasting with the broader range of attitudes found in human humorous improvisation. Our evidence highlights both the strengths and limitations of current LLMs in modeling dialogic humor and demonstrate the value of integrating linguistic theory with NLP methods for the analysis of machine-generated language
Figures & tables
Figure 1: Overall framework for analyzing Dialogic Humor Improvisation in Humans and LLMs
Improvisation act
Turns
Guess Who
All situations take place inside a Mexico City taxi. The driver must take each passenger to their destination and, over the course of the conversation, guess which character the passenger is portraying
82
Four Scenarios
A groom celebrates his bachelor party, which unexpectedly takes place across three locations: a dog grooming salon, a massage parlor, and a funeral home.
67
Semantic Field
The story takes place in an Argentine restaurant in Mexico, where a couple are celebrating their relationship. However, they unexpectedly encounter an ex-boyfriend
60
Table 1: Spontaneous (natural) corpus. Each improvisation act represents a scenario introduced by the TV host, providing the prompt from which the actors develop their humorous improvised dialogues.
Figure 2: Proposed multi-agent setting for humorous dialogue improvisation
Improvisation act
Turns
Guess Who
36
Four Scenarios
12
Semantic Field
12
Talk show
12
Total tokens:
23037
Total number of tokens across repeated dialogue generations.
155178
Table 2: Synthetic corpus. Each improvisation act was generated using LLMs following the scenario and character roles defined for the scene. The table also reports the corpus size when dialogue generation is repeated multiple times.
Figure 3: Histograms of conversational turn lengths in words and characters.
Window size
MATTR
25
50
100
150
Spontaneous (natural)
0.80
0.71
0.62
0.56
Synthetic
0.91
0.82
0.70
0.62
Table 3: Moving average type token ratio (MATTR) across different window sizes for the two corpora
Figure 4: Distribution of POS tags and syntactic dependencies across corpora
Figure 5: NER distributions across corpora.
Natural
Synthetic
Affect
55.6%
32.8%
Appreciation
24.4%
51.1%
Judgement
20%
16.1%
Table 4: Total percentage of elements found in the attitude subsystem
Figure 6: Analysis of the attitudes system based on the appraisal theory
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Histograms of conversational turn lengths in words and characters.
Figure 8: Distribution of POS tags across corpora
Figure 9: Distribution of syntactic dependencies across corpora
Do LLMs talk like us? This question intrigues a multitude of scholar and it is relevant in many fields, from education to academia. This work presents an interpretable statistical feature for distinguishing human written and LLMs generated dialogue. We introduce a lightweight metric derived from semantic categories distribution. Using the Empath lexical analysis framework, each text is mapped to a set of thematic intensity scores. We define semantic delta as the difference between the two most dominant category intensities within a dialogue, hypothesizing that LLM outputs exhibit stronger thematic concentration than human discourse. To evaluate this hypothesis, conversational data were generated from multiple LLM configurations and compared against heterogeneous human corpora, including scripted dialogue, literary works, and online discussions. A Welch t-test was applied to the resulting distributions of semantic delta values. Results show that AI-generated texts consistently produce higher deltas than human texts, indicating a more rigid topics structure, whereas human dialogue displays a broader and more balanced semantic spread. Rather than replacing existing detection techniques, the proposed zero-shot metric provides a computationally inexpensive complementary signal that can be integrated into ensemble detection systems. These finding also contribute to the broader empirical understanding of LLM behavioural mimicry and suggest that thematic distribution constitutes a quantifiable dimension along which current models fall short of human conversational dynamics.
In most existing AI humor research, humor was treated as either "present" or "not present." We explore the concept of humor as a social interaction with context and explanations. During this project, we defined a humor reasoning data object and developed a way to prompt LLMs to generate an explanation of humor effective for general population. We iterated from an earlier prompt to an improved prompt, found that the later version reduced important errors, and then scaled generation to a large number of data objects which have the potential to enable data synthesis and data augmentation for AI humor research. Our main takeaway is that better prompting of an LLM improves humor explanation quality, especially by handling missing context, multi-modality, and transcript issues more carefully. These results establish a strong foundation for future work on AI understanding of humor as social behavior. All code and data are available at: https://github.com/anna-arnett/ai-humor/ .
Anna Arnett, Bang Nguyen, Meng Jiang
Department of Computer Science and Engineering, University of Notre Dame
Humor generation poses a significant challenge for Large Language Models (LLMs), because their standard training objective (next-token prediction) inherently conflicts with the surprise and incongruity required for comedy. To bridge this gap, we introduce the Cognitive Synergy Framework, a methodology for generating highquality humor data inspired by psychological theories of humor. Utilizing a Mixtureof-Thought (MoT) approach, we deploy six cognitive personas (e.g., The Absurdist, The Cynic) to synthesize diverse comedic perspectives for a given prompt. This framework produces a theory-grounded dataset, which we use to fine-tune a 7B-parameter student model. We further evaluate two alignment strategies, Direct Preference Optimization (DPO) and an offline group-relative variant O-GRPO, finding that neither improves over SFT. However, our 7B HumorGen model variants significantly outperform larger instruction-tuned baselines and achieve top-tier open-weight performance while remaining competitive with frontier proprietary systems. These results suggest that cognitively driven data curation is more critical than alignment algorithms or model scale for humor generation.