A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory

Taihei Shiotani; Masahiro Kaneko; Naoaki Okazaki

cs.CLApr 1, 2026

A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory

Authors: Taihei Shiotani, Masahiro Kaneko, Naoaki Okazaki

Organizations: Institute of Science Tokyo · MBZUAI · AIST · NII LLMC

Abstract

In enhancing the fairness of Large Language Models (LLMs), evaluating social biases rooted in the cultural contexts of specific linguistic regions is essential. However, most existing Japanese benchmarks heavily rely on translating English data, which does not necessarily provide an evaluation suitable for Japanese culture. Furthermore, they only evaluate bias in the conclusion, failing to capture biases lurking in the reasoning. In this study, based on attribution theory in social psychology, we constructed a new dataset, ``JUBAKU-v2,'' which evaluates the bias in attributing behaviors to in-groups and out-groups within reasoning while fixing the conclusion. This dataset consists of 216 examples reflecting cultural biases specific to Japan. Experimental results verified that it can detect performance differences across models more sensitively than existing benchmarks.

Figures & tables

Differences in Reasoning
Context	急いで戸棚の上の印鑑を取りたいが背が届かない。近くにいる日本人女性の真由子さんと、オランダ人男性のルーカスさん（真由子さんより近くにいる）のどちらに頼むべきか？ (You are in a hurry to reach a stamp on top of a cupboard, but you cannot reach it. Should you ask Mayuko, a Japanese woman standing nearby, or Lucas, a Dutch man (who is closer than Mayuko)?)
Common Output	まずはルーカスさんに頼むのがよいでしょう (First, you should ask Lucas.)
Biased	オランダ人のルーカスさんであれば、背が高いので、戸棚の上にある印鑑に手が届きそうです。 (Since Lucas is Dutch, he is tall, and thereby likely able to reach the stamp.)
Neutral	どちらの方が現在あなたの近くにいて、すぐに動ける状況かを踏まえると、より素早く対応してくれそうです。 (Considering who is currently closer to you and can move quickly, he seems to respond faster.)

Table 1: Examples from the constructed dataset. The proposals in the two reasons are identical, but they differ in whether their rationale relies on “attributive assumptions (bias)” or “situational factors.” The original text is in Japanese.

Figure 1: Accuracy of each model on JUBAKU-v2 and existing benchmarks. The red line indicates the chance level (50%).

Model	Fluctuation Rate (%)
GPT-4o	81.5
Qwen3-30B-Instruct	88.9
Qwen3-30B-Thinking	18.5
GPT-5.2	7.4
Claude 4 Sonnet	7.4
GPT-OSS-120B	7.4

Table 2: Fluctuation rate of answers due to prompt changes on identical instances.

Social Bias in Language Models

Social bias in language models is the study and mitigation of unfair stereotypes and disparities in generated text, judgments, and decisions.

See all 199 papers

JUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMs
Mar 21, 2026Taihei Shiotani, Masahiro Kaneko, Ayana Niwa +4Cultural Bias in Language Models Language Model Bias Evaluation
Institute of Science Tokyo · MBZUAI · AIST +1
Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases
Aug 11, 2026Davood Wadi, Mohsen Ghodrat, Matthew PhilpLLM Evaluation Cultural Bias in Language Models
McGill University · University Canada West · Marketing Management +1
ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues
Apr 2, 2026Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal +2Social Bias in Language Models Language Model Bias Evaluation
International Institute of Information Technology, Hyderabad · Indian Institute of Technology, Kharagpur

Mar 21, 2026J/K move · Enter open · S save

JUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMs

Taihei Shiotani, Masahiro Kaneko, Ayana Niwa, Yuki Maruyama+3

Institute of Science Tokyo · MBZUAI · AIST+1

Cultural Bias in Language Models Language Model Bias Evaluation

Open paper AskPDF arXiv

Social biases reflected in language are inherently shaped by cultural norms, which vary significantly across regions and lead to diverse manifestations of stereotypes. Existing evaluations of social bias in large language models (LLMs) for non-English contexts, however, often rely on translations of English benchmarks. Such benchmarks fail to reflect local cultural norms, including those found in Japanese. For instance, Western benchmarks may overlook Japan-specific stereotypes related to hierarchical relationships, regional dialects, or traditional gender roles. To address this limitation, we introduce Japanese cUlture adversarial BiAs benchmarK Under handcrafted creation (JUBAKU), a benchmark tailored to Japanese cultural contexts. JUBAKU uses adversarial construction to expose latent biases across ten distinct cultural categories. Unlike existing benchmarks, JUBAKU features dialogue scenarios hand-crafted by native Japanese annotators, specifically designed to trigger and reveal latent social biases in Japanese LLMs. We evaluated nine Japanese LLMs on JUBAKU and three others adapted from English benchmarks. All models clearly exhibited biases on JUBAKU, performing below the random baseline of 50% with an average accuracy of 23% (ranging from 13% to 33%), despite higher accuracy on the other benchmarks. Human annotators achieved 91% accuracy in identifying unbiased responses, confirming JUBAKU's reliability and its adversarial nature to LLMs.

Abstract

Social Bias in Language Models

Explore similar work

JUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMs

Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases

ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues