Cards Against LLMs: Benchmarking Humor Alignment in Large Language Models

Yousra Fettach; Guillaume Bied; Hannu Toivonen; Tijl De Bie

cs.CLApr 9, 2026

Cards Against LLMs: Benchmarking Humor Alignment in Large Language Models

Authors: Yousra Fettach, Guillaume Bied, Hannu Toivonen, Tijl De Bie

Organizations: Ghent University, Belgium · University of Helsinki, Finland

Abstract

Humor is one of the most culturally embedded and socially significant dimensions of human communication, yet it remains largely unexplored as a dimension of Large Language Model (LLM) alignment. In this study, five frontier language models play the same Cards Against Humanity games (CAH) as human players. The models select the funniest response from a slate of ten candidate cards across 9,894 rounds. While all models exceed the random baseline, alignment with human preference remains modest. More striking is that models agree with each other substantially more often than they agree with humans. We show that this preference is partly explained by systematic position biases and content preferences, raising the question whether LLM humor judgment reflects genuine preference or structural artifacts of inference and alignment.

Figures & tables

Figure 1: Framework overview. Given a black card prompt and a slate of 10 white card candidates, five frontier LLMs and a human player independently select the card they deem funniest.

Figure 2: Pairwise agreement rate between models, measured as the proportion of rounds in which two models select the same white card. Off-diagonal cells show inter-model agreement (yellow-red scale); diagonal cells show intra-model consistency across the two replicates (blue scale).

Model	$\chi^{2}$	p -value	Dominant Position
GPT	356	$<$ 0.001	—
Gemini	282	$<$ 0.001	—
Claude	678	$<$ 0.001	Early positions
DeepSeek	1851	$<$ 0.001	Position 3
Grok	658	$<$ 0.001	Position 10

Table 1: Chi-square test of uniformity for position bias per model (df=9). Higher

\chi^{2}

indicates stronger deviation from a uniform pick distribution across the 10 slate positions.

Figure 3: Shares of human (column 1) and LLM (columns 2-5) white card picks involving different topics (a card can have several topic tags). Column 6 provides the shares of topics among all available cards for reference.

Model	Round-level accuracy
GPT	0.171
Gemini	0.339
Claude	0.244
DeepSeek	0.351
Grok	0.361

Table 2: Share of rounds for which surrogate logistic models correctly predict LLMs’ answers.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Definition
Bodily/Gross	Anatomy, bodily fluids, gross-out physical humor
Sexual	Sexual content: innuendo, explicit acts, relationships
Violence/Crime	Physical harm, mortality, criminal acts, threats
Politics/Society	Government, activism, social norms, cultural commentary
Drugs/Risk	Substance use, addiction, reckless actions
Pop Culture	Celebrities, movies, memes, brands, viral trends

Appendix

Table 3: Topic Labels and Definitions

Figure 4: Distribution of model picks across slate positions (N=10). Each bar represents the proportion of rounds in which a model selected the card at a given position, aggregated across all rounds and replicates. Under the null hypothesis of no positional bias, picks would be uniformly distributed at 0.10 (dashed line).

Hyperparameter	Search Values
n_estimators	100,150,200,250,300
max_depth	4,5,6,7,8
learning_rate	0.05,0.08,0.1,0.15,0.2
min_child_weight	3,5,7,10
scale_pos_weight	7,8,9,10,11
subsample	0.7,0.8,0.9,1.0

Appendix

Table 4: Hyperparameter Search Space for XGBoost Baseline

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

Who Laughs with Whom? Disentangling Influential Factors in Humor Preferences across User Clusters and LLMs
Jan 6, 2026Soichiro Murakami, Hidetaka Kamigaito, Hiroya Takamura +1LLM Evaluation Human Preference Modeling
CyberAgent · Nara Institute of Science and Technology · Institute of Science Tokyo
HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models
Mar 31, 2026Edward Ajayi, Prasenjit MitraHumor Explanation Generation LLM Evaluation
Carnegie Mellon University Africa Kigali, Rwanda
LLMs Can Better Capture Human Judgments--With the Right Prompts
Jun 10, 2026Danica Dillion, Chen Cecilia Liu, Baihui Wang +5Human Preference Evaluation LLM Alignment
Complexity Science Hub, Vienna · The Ohio State University · University of Cambridge +2

Jan 6, 2026J/K move · Enter open · S save

Who Laughs with Whom? Disentangling Influential Factors in Humor Preferences across User Clusters and LLMs

Soichiro Murakami, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura

CyberAgent · Nara Institute of Science and Technology · Institute of Science Tokyo

LLM Evaluation Human Preference Modeling Persona Prompting User Preference Modeling

Open paper AskPDF arXiv

Humor preferences vary widely across individuals and cultures, complicating the evaluation of humor using large language models (LLMs). In this study, we model heterogeneity in humor preferences in Oogiri, a Japanese creative response game, by clustering users with voting logs and estimating cluster-specific weights over interpretable preference factors using Bradley-Terry-Luce models. We elicit preference judgments from LLMs by prompting them to select the funnier response and found that user clusters exhibit distinct preference patterns and that the LLM results can resemble those of particular clusters. Finally, we demonstrate that, by persona prompting, LLM preferences can be directed toward a specific cluster. The scripts for data collection and analysis are publicly available to support reproducibility.

Abstract

LLM Alignment

Explore similar work

Who Laughs with Whom? Disentangling Influential Factors in Humor Preferences across User Clusters and LLMs

HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

LLMs Can Better Capture Human Judgments--With the Right Prompts