ManzaiSet: A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedy

Kazuki Kawamura; Kengo Nakai; Jun Rekimoto

cs.CVOct 20, 2025

ManzaiSet: A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedy

Authors: Kazuki Kawamura, Kengo Nakai, Jun Rekimoto

Organizations: Sony CSL Kyoto · The University of Tokyo · Yoshimoto Kogyo Holdings Co., Ltd.

Abstract

We introduce ManzaiSet, a multimodal dataset of individual viewer responses to Japanese manzai comedy. The corpus contains 2,301 viewer-performance recordings from 241 native Japanese speakers, totaling 191.8 hours of facial video and microphone audio. Participants watched up to 10 shared professional performances in randomized order at home and rated each performance on a 0-100 scale. Exploratory analyses yielded three findings. First, viewers differed in both their average ratings and how much their ratings varied across performances. Viewers who rated one random half of the performances highly also tended to rate the other half highly within the same viewing sequence. Second, performances watched later in the sequence received higher ratings on average. Third, after multiple-comparison correction, no statistically significant differences in viewer responses were found between humor categories. ManzaiSet supports the study of individual differences by linking shared Japanese comedy performances to viewer ratings, viewing order, and audiovisual responses.

Figures & tables

Figure 1: Overview of ManzaiSet. Participants watch professional manzai performances at home while facial video and audio are synchronously recorded, followed by a subjective rating after each video. The resulting dataset contains 2,301 viewing sessions from 241 participants (191.8 hours) and enables analyses of individual viewer differences, temporal dynamics, and humor responses. The rating panel schematically illustrates the 0–100 overall evaluation.

Quantity	Value
Participants	241
Shared performances	10
Viewer–performance recordings	2,301
Recordings per participant (mean)	9.55
Summed recording duration	191.8 h
Facial video	$640\times 360$ , 25 fps

Table 1: Corpus units and analysis samples. Duration sums recordings across viewers exposed to the shared set of 10 performances.

$k$	Silhouette	Calinski–Harabasz
2	0.515	148.14
3	0.523	151.87
4	0.476	158.03
5	0.488	149.93

Table 2: Cluster-number diagnostics for standardized rating features. The three-cluster solution provides an interpretable summary of this sample.

	High ratings low variability	Variable ratings positive trend	Lower ratings negative trend
$n$	166	32	30
Share (%)	72.8	14.0	13.2
Mean	87.0	71.2	65.6
SD	4.84	17.9	13.9
CV	0.058	0.263	0.218
Range	15.6	57.8	43.7

Table 3: Rating profiles among 228 viewers. Columns represent high-rating, variable-rating, and lower-rating profiles. Mean, SD, CV, and range are averages of participant-level features. Trends indicate the direction of the average viewing-order slope.

Figure 2: Automated humor-tag assignments across the 10 performances. Each colored segment counts assignments of one tag. The bars sum to 131 assignments over 77 instances; multi-label instances contribute to multiple segments.

Category	$n$	Share	Response (%)
Wordplay	8	10.4%	46.7 (44.7–48.8)
Exaggeration	19	24.7%	56.1 (53.4–58.8)
Unexpected twist	35	45.5%	54.1 (51.8–56.4)
Self-deprecation	9	11.7%	57.5 (53.9–61.4)
Impersonation	5	6.5%	50.9 (48.4–52.7)
Repetition	28	36.4%	53.5 (51.6–55.5)

Table 4: Humor categories and audio-based response rates with 95% bootstrap confidence intervals. The analysis includes 77 instances and 207 viewers. Responses are measured within 3 seconds of instance onset. Multi-label instances contribute to every assigned category.

Figure 3: Candidate vocal response rates within 3 seconds of humor-instance onset, by automated humor tag, among 207 viewers. Points and error bars show category-level response percentages and 95% bootstrap confidence intervals.

MultiHuSE: A Multimodal Dataset for Humour Styles and Emotions
Sep 10, 2026Mary Ogbuka Kenneth, Foaad Khosmood, Abbas EdalatText Classification Multimodal Understanding
Algorithmic Human Development group, Department of Computing Imperial College London London, United Kingdom · Computer Engineering Department California Polytechnic State University San Luis Obispo, United States
Timing In stand-up Comedy: Text, Audio, Laughter, Kinesics (TIC-TALK): Pipeline and Database for the Multimodal Study of Comedic Timing
Mar 23, 2026Yaelle Zribi, Florian Cafiero, Vincent Lépinay +1
ENC · Centre Jean-Mabillon, École nationale des chartes, PSL, Paris, France · ENC, LRE +3
Video2Reaction: Training Foundation Video Models to Predict Audience Reaction
Sep 1, 2026Sidong Zhang, Trang Nguyen, Shiv Shankar +4VLM Adaptation Video-Language Models
UMass Amherst 130 Governors Drive, Amherst · Dolby Laboratories 1275 Market Street, San Francisco

Sep 10, 2026J/K move · Enter open · S save

MultiHuSE: A Multimodal Dataset for Humour Styles and Emotions

Mary Ogbuka Kenneth, Foaad Khosmood, Abbas Edalat

Algorithmic Human Development group, Department of Computing Imperial College London London, United Kingdom · Computer Engineering Department California Polytechnic State University San Luis Obispo, United States

Text Classification Multimodal Understanding

Open paper AskPDF arXiv

Computational recognition of verbal humour remains a challenging task, requiring an understanding of language, delivery style, emotions, and cultural context. Most existing approaches focus on binary classification and lack datasets that capture psychological dimensions of humour alongside variations in expression. We introduce MultiHuSE, a multimodal dataset comprising 2,407 high-definition videos of 50 demographically diverse actors performing 1,463 text samples across four psychological humour styles (affiliative, aggressive, self-enhancing, and self-deprecating), as well as neutral content. A subset is additionally annotated for underlying emotions. The dataset uniquely captures multiple actor interpretations of the same texts, enabling systematic analysis of expressive diversity. Baseline experiments show that multimodal fusion outperforms unimodal approaches (80.1% vs. 77.4% accuracy) in humour style classification, with particularly strong gains for affiliative humour (66% to 74%). While text provides the strongest individual signal, fusion models deliver meaningful improvements. We hope that MultiHuSE provides empirical support for psychological theories linking humour and emotion, while also opening new avenues for research in human communication, well-being, and AI-driven interaction. The dataset is available for academic use under an End-User Licence Agreement.

Abstract

Explore similar work

MultiHuSE: A Multimodal Dataset for Humour Styles and Emotions

Timing In stand-up Comedy: Text, Audio, Laughter, Kinesics (TIC-TALK): Pipeline and Database for the Multimodal Study of Comedic Timing

Video2Reaction: Training Foundation Video Models to Predict Audience Reaction