Comedic Fool's Gold: Reward Exploits and Countermeasures in Conversational Humor

Sam Larson

cs.AISep 18, 2026

Comedic Fool's Gold: Reward Exploits and Countermeasures in Conversational Humor

Authors: Sam Larson

Organizations: Pebble ML

Abstract

We investigate automated rewards for training language models in conversational humor, focusing on reward exploits and countermeasures. Two approaches aim to capture understandable surprise and predicted audience amusement. Controlled tests show that an embedding-based surprise reward accepts word-shuffled replies as readily as witty ones. A fluency filter detects the shuffles, but the combined reward also rejects some witty replies and fails further validation. An audience model's predicted laughter is instead vulnerable to laughter cues in either speaker's messages. Normalizing these cues across speakers blocks the covered attacks, although unmatched expressions remain exploitable. Three reinforcement-learning runs evaluate training with successive reward revisions. The final run improves the combined evaluation score by 0.0903 and reduces zero-score sessions by 40%, but its humor-specific improvement remains below our preregistered target. These findings illustrate a broader challenge for automated reward design: countermeasures must block exploitable shortcuts while preserving the behavior the reward was intended to encourage.

Figures & tables

Figure 1: Final reward path (instrument v2). Laughter is stripped from both speakers only in the audience model’s scoring copy; other checks use original policy replies. GRPO receives the mean additive turn reward. Evaluation uses the multiplicative session score

C

and reports taste

T

separately. The rejected surprise gate is excluded.

Eval	Run	$n$ paired	Seed offset	Temp. / prov. rate	Driver script
eval4	RL-C	500	10,000,000	1.0 / 0.5	eval4_ab.sh
eval5	RL-D	1,000	10,000,000	1.0 / 0.5	eval5_ab.sh
eval6	RL-E	1,000	12,000,000	1.0 / 0.5	eval6_ab.sh

Table 1: Evaluation protocol transcribed from the archived driver scripts in the last column. “Prov. rate” is the provocation rate.

Intended signal	Shortcut	Countermeasure and measured limit
Audience amusement	Append laughter to the policy reply; the audience predicts more laughter.	Strip laughter in the scoring input. Covered baits fall from $+5.3$ – $+7.5$ nats to near zero; unmatched tokens still work.
Amusement at the policy’s contribution	Partner laughter raises the policy’s taste score by $+0.102$ /turn.	Normalize every speaker’s text. Stripping only policy turns leaves the partner channel open.
An unexpected but understandable reply	Word-shuffled replies pass the surprise gate as often as witty replies (AUC 0.511).	A fluency score detects the shuffles (AUC 1.000 on this set). The combined reward is rejected after further checks; the screen also rejects some witty replies.

Table 2: Intended humor proxies, measured shortcuts, and countermeasures. The first two rows concern the audience reward; the third concerns a rejected candidate. A fix’s success applies to the tested attack set.

Figure 2: Suffix-induced changes in audience laughter log-probability before and after normalization. Points show means over 500 paired contexts; bars are archived 95% bootstrap intervals (10,000 resamples). All fourteen laughter-adjacent suffixes and three controls are included. Covered forms lose their score advantage; some other cues retain residual effects. The rule separates controls. Emoji labels: Joy (U+1F602), Rolling (U+1F923), Cat (U+1F639), Squinting (U+1F606), Smiling (U+1F604). These are controlled probes, not attacks discovered by the trained policy.

Figure 3: One RL-E checkpoint versus base on three disjoint evaluation-seed sets under instrument v2 (1,000 matched sessions each). Points show mean paired changes; bars are 95% paired

t

intervals. Dashed lines mark registered effect-size targets. The taste criterion also requires

t\geq 2

; all three taste estimates remain below

+0.015

. Panels use different scales. Both metrics are automated proxies, not human ratings or user retention.

Run	LoRA $r$	Retention ( $t$ )	bar	Taste ( $t$ )	bar
RL-C	32	$+0.0593$ $(t{=}2.87)$	$\geq{+0.03}$ @ $t{\geq}2$	(not primary)	–
RL-D	32	$+0.0257$ $(t{=}1.81)$	$\geq{+0.03}$ @ $t{\geq}2$	$+0.0032$ $(t{=}0.61)$	$\geq{+0.015}$ @ $t{\geq}2$
RL-E	128	$+0.0903$ $(t{=}6.89)$	$\geq{+0.02}$	$+0.0092$ $(t{=}1.99)$	$\geq{+0.015}$ @ $t{\geq}2$
RL-E, repl. 1	128	$+0.0951$ $(t{=}6.79)$	(fresh seeds, 14M)	$+0.0148$ $(t{=}3.14)$	$\geq{+0.015}$ @ $t{\geq}2$
RL-E, repl. 2	128	$+0.1092$ $(t{=}8.33)$	(fresh seeds, 16M)	$+0.0086$ $(t{=}1.84)$	$\geq{+0.015}$ @ $t{\geq}2$

Table 3: The RL-C

\to

RL-D

\to

RL-E program. “Retention” is the certified paired delta vs. base on the run’s primary evaluation (

n{=}500

for RL-C,

n{=}1{,}000

for RL-D/RL-E); “Taste” is the paired delta on the audience-scored taste component alone. Bars are the pre-registered success thresholds.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Id	Claim (short form)	Status
C1	RL-E config: LoRA r128/ $\alpha$ 256, attn-only, 200 GRPO steps	B
C1b	Policy: Qwen3-30B-A3B (MoE)	B
C3	eval6 protocol: $n{=}1000$ , seed offset 12,000,000	B
C4	Certified retention delta, eval6 (headline): $+0.0903$ , $t{=}6.89$	B
C5	Zero-certification sessions, eval6: $184\to 110$	B
C6	Guard readouts, eval6	B

Appendix

Table 4: Evidence status for the numbered claims. B: independently checked archived artifact. R: new measurement or live re-measurement.

Humor Explanation Generation

See all 8 papers

HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models
Mar 31, 2026Edward Ajayi, Prasenjit MitraHumor Explanation Generation LLM Evaluation
Carnegie Mellon University Africa Kigali, Rwanda
HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation
Mar 19, 2026Edward Ajayi, Prasenjit MitraHumor Explanation Generation Text Generation
Carnegie Mellon University Africa
Automatically Finding Reward Model Biases
Feb 16, 2026Atticus Wang, Iván Arcuschin, Arthur ConmyReward Modeling LLM Auditing
MATS · Massachusetts Institute of Technology · Independent

Mar 31, 2026J/K move · Enter open · S save

HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

Edward Ajayi, Prasenjit Mitra

Carnegie Mellon University Africa Kigali, Rwanda

Humor Explanation Generation LLM Evaluation Generative AI Evaluation Text Generation LLM-as-a-Judge

Open paper AskPDF arXiv

Evaluating humor in large language models (LLMs) is an open challenge because existing approaches yield isolated, incomparable metrics rather than unified model rankings, making it difficult to track progress across systems. We introduce HumorRank, a tournament-based evaluation framework and leaderboard for textual humor generation. On two public benchmarks (SemEval-2026 MWAHAHA and Humor Transfer Bench), we conduct extensive automated pairwise evaluation across nine models spanning proprietary, open-weight, and specialized systems. Pairwise judgments are produced by LLM judges grounded in the General Theory of Verbal Humor (GTVH): each judge integrates structured comedic analysis into adjudication, jointly yielding a preference decision, an interpretable rationale, and mechanism, delivery, and failure tags rather than a black-box funniness score. Judgments are aggregated via an Adaptive Swiss tournament, with Bradley-Terry Maximum Likelihood Estimation (MLE) producing globally consistent humor generation capability rankings. Rankings are cross-judge stable: independent LLM judges (Llama 3.3 70B and Qwen 2.5 72B) yield Kendall tau = 0.889 on both benchmarks, and a human calibration study shows human-LLM agreement tracking human-human agreement on hard funny-versus-funny pairs. Our results demonstrate that HumorRank yields statistically grounded model stratifications, showing that humor quality is associated with mastery of comedic mechanisms such as incongruity, conciseness, escalation, and absurdity rather than model scale alone, with specialized fine-tuned models reaching parity with far larger systems. HumorRank thus provides a scalable, interpretable, and reproducible methodology for benchmarking and understanding LLM-generated humor.

Abstract

Humor Explanation Generation

Explore similar work

HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation

Automatically Finding Reward Model Biases