HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

Edward Ajayi; Prasenjit Mitra

cs.CLMar 31, 2026

HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

Authors: Edward Ajayi, Prasenjit Mitra

Organizations: Carnegie Mellon University Africa Kigali, Rwanda

Abstract

Evaluating humor in large language models (LLMs) is an open challenge because existing approaches yield isolated, incomparable metrics rather than unified model rankings, making it difficult to track progress across systems. We introduce HumorRank, a tournament-based evaluation framework and leaderboard for textual humor generation. On two public benchmarks (SemEval-2026 MWAHAHA and Humor Transfer Bench), we conduct extensive automated pairwise evaluation across nine models spanning proprietary, open-weight, and specialized systems. Pairwise judgments are produced by LLM judges grounded in the General Theory of Verbal Humor (GTVH): each judge integrates structured comedic analysis into adjudication, jointly yielding a preference decision, an interpretable rationale, and mechanism, delivery, and failure tags rather than a black-box funniness score. Judgments are aggregated via an Adaptive Swiss tournament, with Bradley-Terry Maximum Likelihood Estimation (MLE) producing globally consistent humor generation capability rankings. Rankings are cross-judge stable: independent LLM judges (Llama 3.3 70B and Qwen 2.5 72B) yield Kendall tau = 0.889 on both benchmarks, and a human calibration study shows human-LLM agreement tracking human-human agreement on hard funny-versus-funny pairs. Our results demonstrate that HumorRank yields statistically grounded model stratifications, showing that humor quality is associated with mastery of comedic mechanisms such as incongruity, conciseness, escalation, and absurdity rather than model scale alone, with specialized fine-tuned models reaching parity with far larger systems. HumorRank thus provides a scalable, interpretable, and reproducible methodology for benchmarking and understanding LLM-generated humor.

Figures & tables

Figure 1: Overview of the HumorRank evaluation pipeline. Contestant models generate jokes on shared prompts, Adaptive Swiss pairing schedules pairwise duels, a GTVH-grounded LLM judge adjudicates each duel, and outcomes are aggregated into match history and converted to global Bradley–Terry ratings (primary) and Stable Elo scores (audit), producing a ranked leaderboard with interpretable humor-feature profiles.

Humor Transfer Bench (HTB): Full RR, Llama 3.3 70B judge, 14,400 judgments
Rank	Model	BT Rating	St. Elo	95% CI	Win %
1	GPT-5	1314.7	1300.4	$[1301.3,1329.0]$	84.1%
2	Kimi K2	1242.0	1239.1	$[1229.5,1255.6]$	77.2%
3	HumorGen-7B 1 1 1 Referred to as HumorGen SFT 7B in plots and figures.	1097.7	1122.8	$[1084.4,1109.9]$	60.6%
4	Claude 3.5 Haiku	1054.2	1058.3	$[1043.2,1068.5]$	55.1%
5	Gemini 2.5 Pro	1024.0	1009.0	$[1010.6,1038.2]$	51.3%

Table 1: Full round-robin HumorRank leaderboards on HTB (top) and SemEval (bottom), both judged by the primary Llama 3.3 70B judge. BT ratings are the primary metric, and Stable Elo (10 shuffle runs) is a sequence-robust audit. Anchor ranks are stable across benchmarks (GPT-5 #1, Llama 3.3 70B and Qwen 2.5 7B Instruct #8/#9). SemEval win-rate heatmap: Appendix Figure 6 . HTB extended budget ablations: Appendix D.5 .

Schedule	Pairs / prompt ( $K{=}9$ )	Budget	Avg. cross-LLM-judge $\tau$
Full RR	36	100%	0.889
Swiss 3RR	12	$\sim$ 33%	0.861
Swiss 2RR	8	$\sim$ 22%	0.806

Table 2: Adaptive Swiss Pairing budget ablation across Full RR, Swiss 3RR, and Swiss 2RR. SemEval Llama 3.3 70B

\leftrightarrow

Qwen 2.5 72B LLM-judge

\tau=0.889

under both Full RR and Swiss 3RR.

Comparison axis	Pairs
Cross-tier (rank span)	32
Within-tier, top quartile	10
Within-tier, lower ranks	18
Scale contrast	10
Rank-spanning frontier matchups	15
Alignment contrast	5

Table 3: Human evaluation set design (90 pairs, 75 unique headlines). Axes mirror tournament stratification, and all pairs are funny-versus-funny only.

Human-only dyads
Cohort	Krippendorff’s $\alpha$
H 1 + H 3	0.446
H 2 + H 3	0.436
H 1 + H 2	0.334
H 1 + H 2 + H 3	0.416
Human–Llama judge dyads

Table 4: Krippendorff’s

\alpha

(nominal winner-model labels) on the 90-pair blind evaluation set. H i : human annotator

i

, Llama / Qwen : production LLM judges. Human–Llama alignment on H 2 +H 3 is not significantly different from human–human agreement (Fisher exact

p=0.808

Feature	GTVH	Description
Incongruity	LM	Conflicting scripts or ideas.
Absurdity	SI	Breaks physical/social expectations.
Sarcasm	TA/LM	Irony with an explicit target.
Wordplay	LA	Puns and lexical/syntactic ambiguity.
Conciseness	LA	Efficient buildup, comedic timing.

Table 5: GTVH-grounded humor features tagged in structured LLM judge responses.

Figure 2: Per-model winning humor-mechanism distributions under the Llama judge (% of wins). Frontier models emphasize Incongruity , mid-tier specialists lead on Absurdity , and baselines over-index on Wordplay .

Figure 3: Per-model winning delivery-feature distributions under the Llama judge (% of wins). Frontier models emphasize Conciseness , while mid-tier specialists show elevated Escalation .

Figure 4: Per-model failure mode distributions (Llama judge). Cliché and Weak Punchline dominate most tiers. Mid-tier specialists show higher Overexplained (25.2%) and Buried Punchline (20.4%) rates.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Paradigm	Symptom	Response	n
Absolute 0–100 rubric	High-band clustering; mean within-HL spread 20.6	Abandoned	17k
Structured $4{\times}1$ – $5$ $\to$ 0–100	88.5% scores $=$ 70.0; 30.8% headlines all tie	Abandoned	12k
Scalar 1–20 funniness	82.7% in band 13–16; 22.2% spread $\leq 1$	Abandoned	12k
Early pairwise prompt (verbose CoT)	$\sim$ 62% tie rate in pilot	Revised prompt	pilot
GPT/Gemini judge (pairwise)	75–88% self-win on shared pool	Rejected	180–360
Structured pairwise + Llama/Qwen72	Cross-judge $\tau=0.889$	Production	10,800

Appendix

Table 6: LLM-as-a-Judge configurations tested on humor evaluation. Failed rows document why HumorRank does not use absolute scoring or proprietary self-preferring judges.

Rk	Score	Words	Joke (full text)
1	87	44	You know, “Experts weigh in” on body wipes. I guess that means they literally stepped on them with their lab coats on to see if they’d break. Good thing the wipes held up, otherwise we’d have a whole bunch of scientists doing the splits.
2 o	87	161	You know, body wipes… because sometimes you need a quick fix for that lingering funk, like when you’ve been sitting in traffic for three hours and your deodorant decided to take a vacation. And let me tell you, experts are super excited about this groundbreaking issue. They’re probably the same ‘experts’ who got their degree from the University of “I Can’t Shower Because I’m Too Important To Be Clean.” Listen, if these wipes really worked as well as advertised, we’d all just stop using showers altogether and the water crisis would be solved. We could rename them “body band-aids”—they don’t cure the smell, they just give you a little time before everyone starts backing away slowly. But hey, if you’re in a rush and can’t decide between a wipe and a shower, here’s a tip: If you think a wipe will cut it, you’re already too late. The cat’s out of the bag and it’s not the furry kind.
3 o	87	163	So, do body wipes actually work? Let’s ask the experts! I heard they can tell you not only if you smell good enough to eat (which is totally a thing now, apparently), but also if you weigh more than your cat (and who wouldn’t want to know that)? Just imagine it—an expert body wipe panel sitting around discussing your cleanliness level with cups of tea and judging your body odor on a scale from ‘newly mown hay’ to ‘forgotten gym socks’. And if you fail their sniff test, fear not, because there’s always the option of becoming a human body wipe for the rest of the team—think of all the scrubbing you could do! The best part is, even if body wipes don’t work, at least we know the experts will be busy and not questioning life’s bigger mysteries like why the remote never works when you need it most. Now that’s a silver lining I can get behind!
4	83	95	Have you ever noticed how we have experts for everything now? Like, “Experts weigh in on whether body wipes work.” I mean, hello! If you’re wiping yourself with a wet cloth, odds are you’re not getting dirtier. It’s like having a panel of experts to discuss whether a hair dryer makes your hair dry. You know what? I’m calling it: if you’re using a body wipe, you are already an expert on body wipes. You don’t need some other guy telling you if it works or not. You’re the boss of your own smell situation!
5 o	83	154	So I tried one of these wipes the other day, you know, just to test out the efficacy. I mean, I’m pretty sure it’s just a ploy to make me buy more deodorant. But what if the real secret is in the corners, like the little nooks where your bra straps dig in or behind your knees? I mean, who even thinks about those places? And now, every time I use a body wipe, I’m like, ‘Did it get the corner under my left armpit?’ And then I find myself spending the next 30 minutes just staring at that spot in the mirror, turning my arm around and around, going, ‘Did I miss it? Did I really miss it?!’ And I swear, if anyone catches me doing this, they’ll think I’m totally insane. But hey, I’d rather be the guy with the obsessive-compulsive corner-wiping routine than the guy who smells like old gym socks!
6 u	82	50	You know, they say “experts weigh in” on whether body wipes actually work, but I’m pretty sure these so-called experts were using the heavy-duty kind—the ones that add pounds just from the packaging. At least now we know why the debate on body wipes has gotten so… weighty!

Appendix

Table 7: Exp. 1 Failure Case ( en_0004 ). Fifteen same-headline candidates scored by Llama 3.3 70B Instruct. Metrics: Score = Llama-assigned 0–100 rating; Words = word count. Row Colors (score anchors): gold/peach 87–83, lilac 82, blue 77–74, mint 62. Error Markers: coral ( o ) = verbose build rated too high; emerald ( u ) = tight punchline rated too low. Takeaway: absolute scoring collapses distinct jokes onto six anchors and misjudges which responses land.

Panel of Figure 5 — Figure 5: Prompt template used for pairwise comparisons in all full round-robin runs. Template variables ( {headline} , {joke_a} , {joke_b} ) are instantiated per comparison. The three feature lists (humor mechanisms, delivery, and loser features) enforce structured and consistent JSON outputs across all evaluations.

Humor Mechanisms (Deep Structure / GTVH-Aligned)
Feature	Definition
incongruity	Violation of expectations or semantic mismatch.
wordplay	Puns, double meanings, or clever syntactic manipulation.
absurdity	Bizarre, surreal, or hilariously illogical situations.
surprise	Sudden misdirection or sharp pivot in the punchline.
irony	Contrast between expectation and reality, often subverting literal meaning.

Appendix

Table 8: Taxonomy of Humor Mechanisms, Delivery Features, and Failure Modes evaluated by the LLM-as-a-judge. These definitions align the evaluation protocol with General Theory of Verbal Humor (GTVH) constructs.

Comparison	Full RR	Swiss 2RR	Swiss 3RR
SemEval: Llama vs Qwen	0.889	0.667	0.889
HTB: Llama vs Qwen	0.889	1.000	0.889
SemEval vs HTB (Llama)	0.889	0.778	0.778
SemEval vs HTB (Qwen)	0.889	0.778	0.889
Average	0.889	0.806	0.861

Appendix

Table 9: Kendall

\tau

across budget modes (four cross-judge / cross-benchmark cells). Swiss 3RR restores SemEval Llama

\leftrightarrow

Qwen agreement to Full RR levels.

Model	Rank (Full RR / Swiss 3RR / Swiss 2RR)
GPT-5	1 / 1 / 1
Kimi K2	2 / 2 / 4
Gemini 2.5 Pro	3 / 3 / 2
HumorGen-7B	4 / 5 / 6
Claude 3.5 Haiku	5 / 7 / 5
GPT OSS 120B	6 / 4 / 3

Appendix

Table 10: SemEval Llama judge ranks under Full RR, Swiss 3RR, and Swiss 2RR. Ranks #1 (GPT-5), #8 (Llama 3.3 70B), and #9 (Qwen 2.5 7B Instruct) are invariant across all three schedules; mid-tier models reorder most under Swiss 2RR.

Model	Rank (Full RR / Swiss 3RR / Swiss 2RR)
GPT-5	1 / 1 / 1
Kimi K2	2 / 2 / 2
HumorGen-7B	3 / 3 / 6
Claude 3.5 Haiku	4 / 6 / 4
Gemini 2.5 Pro	5 / 5 / 5
GPT OSS 120B	6 / 4 / 3

Appendix

Table 11: HTB Llama judge ranks under Full RR, Swiss 3RR, and Swiss 2RR. Ranks #1 (GPT-5), #8 (Llama 3.3 70B), and #9 (Qwen 2.5 7B Instruct) are invariant; HumorGen-7B drops from #3 to #6 under Swiss 2RR.

Panel of Figure 6 — Figure 6: SemEval, Full RR, Llama judge (Table 1 ). Top: Bradley–Terry leaderboard with 95% confidence intervals (10,800 judgments). Bottom: Pairwise win-rate heatmap. Observation: clear frontier/mid/baseline separation; cross-judge agreement with Qwen Full RR is Kendall $\tau=0.889$ .

Panel of Figure 7 — Figure 7: SemEval, Swiss 3RR, Llama judge (Table 12 ). Top: BT leaderboard from 3,600 judged pairs. Bottom: Win-rate heatmap. Observation: Swiss 3RR at one-third of Full RR comparisons recovers the same cross-judge rank correlation ( $\tau=0.889$ ) and keeps anchor ranks #1/#8/#9 fixed.

Rank	Model	BT Rating	Stable Elo	95% CI	Win Rate
1	GPT-5	1268.7	1253.2	$[1243.0,1302.7]$	76.7%
2	Kimi-K2	1133.2	1132.7	$[1110.8,1160.9]$	63.9%
3	Gemini 2.5 Pro	1126.4	1108.6	$[1105.0,1151.3]$	52.8%
4	GPT OSS 120B	1077.3	1086.3	$[1048.5,1106.8]$	49.3%
5	HumorGen-7B	1042.0	1058.5	$[1012.9,1070.7]$	64.1%
6	Qwen 3 32B	990.7	997.8	$[965.5,1021.9]$	35.2%

Appendix

Table 12: SemEval, Swiss 3RR, Llama judge. Rank

\tau

vs. Full RR

=0.889

; cross-judge

\tau

(Llama vs. Qwen on SemEval)

=0.889

. Anchor ranks #1, #8, #9 unchanged from Table 10 .

Panel of Figure 8 — Figure 8: SemEval, Swiss 2RR, Llama judge (Table 13 ). Top: BT leaderboard from 2,400 judged pairs. Bottom: Win-rate heatmap. Observation: lowest-budget mode preserves top/bottom anchors but increases mid-tier rank volatility and reduces cross-judge agreement ( $\tau=0.667$ ).

Rank	Model	BT Rating	Stable Elo	95% CI	Win Rate
1	GPT-5	1358.4	1307.9	$[1304.3,1423.6]$	79.3%
2	Gemini 2.5 Pro	1190.1	1103.0	$[1135.1,1246.6]$	45.3%
3	GPT OSS 120B	1122.0	1037.6	$[1056.8,1207.7]$	35.7%
4	Kimi-K2	1044.1	1117.4	$[985.3,1111.5]$	68.8%
5	Claude 3.5 Haiku	1040.5	1028.4	$[993.9,1085.8]$	55.3%
6	HumorGen-7B	950.0	1000.7	$[898.4,1008.2]$	66.5%

Appendix

Table 13: SemEval, Swiss 2RR, Llama judge. Cross-judge Llama

\leftrightarrow

Qwen

\tau=0.667

; ranks #1, #8, and #9 remain stable (Table 10 ).

Rank	Model	BT Rating	Stable Elo	95% CI	Win Rate
1	GPT-5	1314.7	1300.4	$[1301.3,1329.0]$	84.1%
2	Kimi K2	1242.0	1239.1	$[1229.5,1255.6]$	77.2%
3	HumorGen-7B	1097.7	1122.8	$[1084.4,1109.9]$	60.6%
4	Claude 3.5 Haiku	1054.2	1058.3	$[1043.2,1068.5]$	55.1%
5	Gemini 2.5 Pro	1024.0	1009.0	$[1010.6,1038.2]$	51.3%
6	GPT OSS 120B	1009.6	1017.2	$[998.4,1021.6]$	49.5%

Appendix

Table 14: HTB, Full RR, Llama judge (14,400 judgments). Observation: ordering mirrors SemEval at the extremes (GPT-5 #1; Llama/Qwen 2.5 7B Instruct #8/#9); HumorGen-7B ranks 3rd on this held-out benchmark.

Panel of Figure 9 — Figure 9: HTB, Swiss 3RR, Llama judge (Table 15 ). Top: BT leaderboard from 4,800 judged pairs. Bottom: Win-rate heatmap. Observation: one-third budget retains HTB anchor ranks and cross-judge stability ( $\tau=0.889$ ).

Rank	Model	BT Rating	Stable Elo	95% CI	Win Rate
1	GPT-5	1259.2	1246.7	$[1236.8,1282.7]$	76.7%
2	Kimi K2	1210.5	1204.0	$[1188.5,1236.0]$	71.4%
3	HumorGen-7B	1067.9	1055.4	$[1040.1,1094.1]$	63.9%
4	GPT OSS 120B	1058.2	1062.8	$[1032.3,1088.9]$	50.9%
5	Gemini 2.5 Pro	1037.7	1056.0	$[1018.5,1061.0]$	42.2%
6	Claude 3.5 Haiku	1013.6	1025.1	$[986.8,1036.6]$	51.6%

Appendix

Table 15: HTB, Swiss 3RR, Llama judge. Ranks #1, #3 (HumorGen-7B), #8, and #9 match Full RR anchors; cross-judge HTB

\tau=0.889

Panel of Figure 10 — Figure 10: HTB, Swiss 2RR, Llama judge (Table 16 ). Top: BT leaderboard from 3,200 judged pairs. Bottom: Win-rate heatmap. Observation: anchor ranks hold, but specialist mid-tier placement becomes less stable (HumorGen-7B 3 $\rightarrow$ 6).

Rank	Model	BT Rating	Stable Elo	95% CI	Win Rate
1	GPT-5	1326.5	1303.3	$[1274.5,1386.9]$	81.4%
2	Kimi K2	1145.7	1180.1	$[1077.0,1208.1]$	77.3%
3	GPT OSS 120B	1097.1	1043.6	$[1032.3,1163.1]$	47.5%
4	Claude 3.5 Haiku	1087.9	1072.2	$[1046.1,1137.9]$	59.5%
5	Gemini 2.5 Pro	1078.1	1051.7	$[1024.0,1138.7]$	33.8%
6	HumorGen-7B	991.0	1025.6	$[936.6,1052.7]$	64.4%

Appendix

Table 16: HTB, Swiss 2RR, Llama judge. Anchor ranks #1/#8/#9 stable; HumorGen-7B drops 3

\rightarrow

6 vs. Full RR. Cross-judge HTB

\tau=1.000

at this budget (Table 9 ).

Rank	Model	BT Rating	Stable Elo	95% CI	Win Rate
1	GPT-5	1245.7	1278.6	$[1234.5,1258.7]$	78.8%
2	Kimi K2	1199.5	1205.3	$[1187.2,1211.7]$	73.8%
3	Claude 3.5 Haiku	1101.4	1094.1	$[1089.4,1114.0]$	61.8%
4	HumorGen-7B	1099.8	1099.2	$[1086.1,1115.0]$	61.6%
5	GPT OSS 120B	1024.3	1015.9	$[1013.4,1037.9]$	51.7%
6	Gemini 2.5 Pro	1001.3	990.4	$[989.4,1013.7]$	48.8%

Appendix

Table 17: HTB, Full RR, Qwen judge (14,400 judgments). Observation: cross-judge HTB

\tau=0.889

vs. Table 14 ; bottom-two ranks unchanged.

Panel of Figure 11 — Figure 11: HTB, Swiss 3RR, Qwen judge (Table 18 ). Top: BT leaderboard from 4,800 judged pairs. Bottom: Win-rate heatmap. Observation: GPT-5/Kimi remain top-two; Llama/Base Qwen stay #8/#9; HumorGen-7B remains top-tier (#4).

Rank	Model	BT Rating	Stable Elo	95% CI	Win Rate
1	GPT-5	1204.0	1207.1	$[1182.2,1228.7]$	72.1%
2	Kimi K2	1178.6	1182.5	$[1158.3,1199.0]$	69.9%
3	GPT OSS 120B	1071.1	1064.0	$[1045.8,1100.2]$	55.1%
4	HumorGen-7B	1071.1	1081.3	$[1043.8,1098.1]$	64.4%
5	Claude 3.5 Haiku	1052.0	1053.2	$[1034.1,1078.3]$	57.6%
6	Gemini 2.5 Pro	993.9	1004.3	$[975.0,1016.3]$	37.4%

Appendix

Table 18: HTB, Swiss 3RR, Qwen judge. Cross-judge HTB

\tau=0.889

; anchor ranks #1/#8/#9 stable vs. Qwen Full RR.

Panel of Figure 12 — Figure 12: HTB, Swiss 2RR, Qwen judge (Table 19 ). Top: BT leaderboard from 3,200 judged pairs. Bottom: Win-rate heatmap. Observation: minimum-budget HTB run preserves top/bottom anchors but increases mid-tier volatility, consistent with the Llama judge Swiss 2RR pattern.

Rank	Model	BT Rating	Stable Elo	95% CI	Win Rate
1	GPT-5	1262.7	1239.4	$[1221.6,1315.4]$	76.1%
2	Kimi K2	1140.0	1166.5	$[1077.3,1196.3]$	74.0%
3	GPT OSS 120B	1137.6	1098.8	$[1092.1,1196.4]$	55.5%
4	Claude 3.5 Haiku	1075.7	1057.5	$[1035.0,1113.9]$	62.7%
5	Gemini 2.5 Pro	1055.1	1045.2	$[1016.5,1106.4]$	32.0%
6	HumorGen-7B	1023.2	1057.8	$[972.7,1061.8]$	66.4%

Appendix

Table 19: HTB, Swiss 2RR, Qwen judge. Anchor ranks #1/#8/#9 stable; HumorGen-7B 4

\rightarrow

6 vs. Qwen Full RR. Cross-judge HTB

\tau=1.000

(Table 9 ).

$K$	$R$	Mode	Pairs/prompt	Budget %	$\rho$	$\tau$
9	3	exact	12	33.3	0.983	0.944
9	4	exact	18	50.0	0.983	0.944
100	3	exact	150	3.0	0.998	0.971
1,000	3	exact	1,500	0.30	0.987	0.904
1,000	10	exact	5,000	1.0	0.997	0.955
10,000	3	windowed	15,000	0.03	0.948	0.802

Appendix

Table 20: Synthetic ASP scaling (no jokes/LLM). Outcomes from a latent BT ladder; rankings recovered by BT-MLE (

K\leq 500

) or Elo (

K>500

). At

K{=}9

R{=}3

uses the paper Swiss 3RR budget of 12 pairs/prompt (

\sim

33% of Full RR). All listed graphs are connected.

R{=}3

\mathcal{O}(K)

;

R{=}\lceil\log_{2}K\rceil

\mathcal{O}(K\log K)

Panel of Figure 13 — Figure 13: SemEval, Full RR, Qwen judge (Table 21 ; Kendall $\tau=0.889$ vs. SemEval panel of Table 1 ). Top: Bradley–Terry leaderboard with 95% confidence intervals (10,800 judgments). Bottom: Pairwise win-rate heatmap. Observation: GPT-5 and Kimi K2 remain ranks 1–2; Llama 3.3 70B and Qwen 2.5 7B Instruct remain ranks 8–9; HumorGen-7B moves 4 $\rightarrow$ 3 and Gemini 2.5 Pro 3 $\rightarrow$ 5 vs. the Llama judge, consistent with modest mid-tier reordering at $\tau=0.889$ .

Rank	Model	BT Rating	Stable Elo	95% CI	Win Rate
1	GPT-5	1262.7	1267.7	$[1246.2,1279.8]$	80.7%
2	Kimi-K2	1126.2	1135.8	$[1113.8,1140.2]$	64.8%
3	HumorGen-7B 1 1 1 Referred to HumorGen-7B as HumorGen SFT 7B in plots and figures.	1100.9	1092.1	$[1088.5,1114.9]$	61.5%
4	Claude 3.5 Haiku	1062.1	1062.4	$[1049.4,1074.9]$	56.3%
5	Gemini 2.5 Pro	1058.5	1068.4	$[1044.0,1070.4]$	55.8%
6	GPT OSS 120B	1036.0	1036.2	$[1023.5,1054.1]$	52.9%

Appendix

Table 21: SemEval, Full RR, Qwen judge (10,800 judgments). BT ratings are primary; Stable Elo is a sequence-robust audit metric (Appendix L ). Observation: Kendall

\tau=0.889

vs. SemEval panel of Table 1 ; anchor ranks #1/#8/#9 match the Llama judge; HumorGen-7B ranks 3rd (vs. 4th under the Llama judge).

Parameter	Value
Temperature	0.7
Top- $p$	0.9
Max New Tokens	256
System Prompt	”You are a joke generator. Given a headline or topic, generate a funny joke. Output ONLY the joke text. No thinking tags, no reasoning, no explanation, no extra words.”

Appendix

Table 22: Hyperparameters for candidate humor generation across all local and API-based models.

Parameter	Value
Primary Judge	Llama 3.3 70B Instruct
Ablation Judge	Qwen 2.5 72B Instruct
Temperature	0.1
Max New Tokens	512
Max Retries	3
Backoff Base	2.0

Appendix

Table 23: Hyperparameters for LLM-as-a-Judge adjudication.

Parameter	Value
Initial Elo Rating	1000.0
$K$ -factor	32
Min Rounds per Model	2
Max Rounds per Model	3
Stable Elo Shuffles	10
BT Convergence ( $\epsilon$ )	$10^{-6}$

Appendix

Table 24: Tournament configuration and Bradley–Terry global MLE parameters.

Panel of Figure 15 — Figure 15: Qwen judge: per-model winning feature distributions. Top: Humor mechanisms (% of wins). Bottom: Delivery features (% of wins). Tags are co-emitted with duel outcomes in the structured LLM judge response. Rank patterns are consistent with the primary Llama judge (Figures 2 and 3 ).

Figure 16: Qwen judge: per-model failure mode distributions (% of losses). HumorGen-7B again shows markedly higher Overexplained (49.5%) rates than other models, consistent with the primary Llama judge (Figure 4 ).

Humor Explanation Generation

See all 8 papers

Mar 19, 2026

HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation

Humor generation poses a significant challenge for Large Language Models (LLMs), because their standard training objective (next-token prediction) inherently conflicts with the surprise and incongruity required for comedy. To bridge this gap, we introduce the Cognitive Synergy Framework, a methodology for generating highquality humor data inspired by psychological theories of humor. Utilizing a Mixtureof-Thought (MoT) approach, we deploy six cognitive personas (e.g., The Absurdist, The Cynic) to synthesize diverse comedic perspectives for a given prompt. This framework produces a theory-grounded dataset, which we use to fine-tune a 7B-parameter student model. We further evaluate two alignment strategies, Direct Preference Optimization (DPO) and an offline group-relative variant O-GRPO, finding that neither improves over SFT. However, our 7B HumorGen model variants significantly outperform larger instruction-tuned baselines and achieve top-tier open-weight performance while remaining competitive with frontier proprietary systems. These results suggest that cognitively driven data curation is more critical than alignment algorithms or model scale for humor generation.

Edward Ajayi, Prasenjit Mitra

Carnegie Mellon University Africa

Humor Explanation Generation Text Generation Language Model Post-Training Synthetic Data Generation for Language Models

Ask PDF

Apr 14, 2026

lmfaoooo at SemEval-2026 Task 1: Humor Is an Audience. Preference Modeling for Constrained Humor Generation

Humor generation remains difficult not only because producing fluent, novel jokes is hard, but because "funny" is audience-dependent and supervision is noisy -- preferences vary with audience, context, and culture, and annotator agreement is often low. In this paper, we describe our system for the SemEval-2026 Task-1 (MWAHAHA), which focuses on humor generation under explicit constraints. The task evaluates submitted systems via human preference judgments in 1-on-1 arena-style comparisons. We adopt a "generate-many -> select-best" strategy. First, we generate a diverse pool of candidates per instance using multi-step prompting, model ensembling, and diversity-oriented decoding. Second, we select outputs using a preference model that approximates a "reader" by learning from human comparisons rather than absolute funniness scores. To support this approach, we release 2.5K human pairwise judgments collected through the Humor Arena prototype. We further propose an interpretable pipeline that converts labeled comparisons into a preference model. Across three preference datasets, our models consistently outperform baselines and show stronger cross-domain transfer. Finally, we apply the learned preference model to rank candidates for the MWAHAHA setting and release intermediate artifacts (candidate pools and rankings) to facilitate follow-up work. Our system ranked 1st in the English and Chinese subtasks of MWAHAHA and 2nd in the Spanish subtask.

Alexey Tikhonov, Alexey Ivanov

Inworld.AI Berlin, Germany · OpenAI Mountain View, CA

Humor Explanation Generation Text Generation Human Preference Modeling Pairwise Preference Learning Language Model Generation Evaluation Controllable Text Generation

Ask PDF

Sep 14, 2026

IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English

Our team, VANGUARD, presents IROH (Insightful Ranking of Humor), a three-stage retrieval system for JOKER Task 1 English at CLEF 2026, achieving first place on the leaderboard with 0.6347 MAP. Our pipeline combines hybrid sparse-dense retrieval, cross-encoder reranking, and a LoRA-adapted Large Language Model judge ensemble. We employ Gemma 4 to generate query-aware rationales under two prompt strategies, generic and typed, and produce up to four types of structured hard negatives for training data construction. Through an ablation across three cross-encoder architectures, four dense embedders, and eight judge configurations, our key findings are threefold: (1) the rationale-distilled judge is the primary driver of ranking quality, whereas appending rationales to the first-stage index contributes negligibly; (2) structured hard negatives degrade generalisation in nearly all configurations despite inflating local validation scores; and (3) across the components we ablate, the lighter, better-calibrated model is competitive with or stronger than its larger counterpart, with the generic-rationale Qwen2.5-7B judge (0.6055 MAP) outperforming every Gemma-4-31B configuration, and the advantage of generic over typed rationales is concentrated almost entirely in the smaller model.

Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truica +1

National University of Science and Technology POLITEHNICA Bucharest, Splaiul Independenţei 313, Bucureşti 060042, Romania · Academy of Romanian Scientists, Ilfov 3, Bucharest, 050044, Romania

LLM-as-a-Judge Hybrid Retrieval

Ask PDF

Abstract

Humor Explanation Generation

Explore similar work

HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation

lmfaoooo at SemEval-2026 Task 1: Humor Is an Audience. Preference Modeling for Constrained Humor Generation

IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English