LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation

Himel Ghosh; Nick Elias Werner

cs.CLMar 12, 2026

LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation

Authors: Himel Ghosh, Nick Elias Werner

Organizations: Technical University of Munich, Germany · Sapienza University of Rome, Italy

Abstract

As large language models (LLMs) are deployed widely, detecting and understanding bias in their outputs is critical. We present LLM BiasScope, a web application for side-by-side comparison of LLM outputs with real-time bias analysis. The system supports multiple providers (Google Gemini, DeepSeek, MiniMax, Mistral, Meituan, Meta Llama) and enables researchers and practitioners to compare models on the same prompts while analyzing bias patterns. LLM BiasScope uses a two-stage bias detection pipeline: sentence-level bias detection followed by bias type classification for biased sentences. The analysis runs automatically on both user prompts and model responses, providing statistics, visualizations, and detailed breakdowns of bias types. The interface displays two models side-by-side with synchronized streaming responses, per-model bias summaries, and a comparison view highlighting differences in bias distributions. The system is built on Next.js with React, integrates Hugging Face inference endpoints for bias detection, and uses the Vercel AI SDK for multi-provider LLM access. Features include real-time streaming, export to JSON/PDF, and interactive visualizations (bar charts, radar charts) for bias analysis. LLM BiasScope is available as an open-source web application, providing a practical tool for bias evaluation and comparative analysis of LLM behaviour.

Figures & tables

Figure 1: LLM BiasScope Application Home Page.

Figure 2: System architecture. The system uses a three-layer client–server design: (1) a React/Next.js frontend with dual chat panels for parallel LLM comparison and bias visualizations; (2) Next.js API routes handling model inference and the two-stage bias analysis pipeline; and (3) external services, including multi-provider LLMs via the Vercel AI Gateway and Hugging Face endpoints for bias detection and classification. Arrows show the flow from user input to model outputs, bias analysis, and visual comparison.

Model	SS (%)	Avg Latency (s)
toxic-bert	69.30	0.73
roberta-hate-speech	57.29	0.98
bias-detector	49.73	1.22
toxic-comment-model	37.86	0.89

Table 1: Evaluation results on CrowS-Pairs dataset. SS = Stereotype Score.

Figure 3: Comparison of Metrics for several bias detection models. The accuracy and F1 Score of the bias-detector show better performance compared to others.

Model	Accuracy	Precision	Recall	F1
toxic-bert	55.9%	55.9%	100.0%	71.7%
da-roberta-babe-ft	82.3%	97.0%	70.5%	81.7%
bias-detector	85.2%	92.4%	80.1%	85.8%

Table 2: Evaluation results on BABE dataset. Best F1-score in bold.

Figure 4: Bias Classification: examples of some bias types as detected by the system.

Figure 5: Model Comparison: Bias-Types distribution from Model responses.

Test Case	Model A	Model B	Difference
	(Meituan)	(MiniMax)
Healthcare Advice	2.60%	0.00%	+2.60%
Career Advice	10.60%	0.00%	+10.60%
Educational Content	28.20%	0.00%	+28.20%

Table 3: Illustrative model comparison results on three test prompts (mean bias percentage across

N=3

trials).

Figure 6: System performance across four text-length categories. Top: latency vs. text length and latency distributions, showing near-linear scaling and consistent behavior across trials. Bottom: success rates (100% for all cases) and summary latency statistics, indicating predictable performance from short inputs (0.14 s for 6 words) to longer texts (6.35 s for 83 words).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Bias analysis card for the input and model responses.

Figure 10: Confusion matrix for bias detection model predictions.

Figure 11: Evaluation results across all bias detection models.

Language Model Bias Evaluation

Language model bias evaluation tests how models’ outputs, decisions, and representations vary across social groups, cultures, languages, and contexts.

See all 492 papers

Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregation
Apr 24, 2026Matteo Pelossi, Rita Sevastjanova, Thilo Spinner +1LLM Auditing Social Bias in Language Models
ETH Zurich
Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
Apr 25, 2026Sadman Kabir SoumikDebiased ML LLM-as-a-Judge
Independent Researcher
Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models
Mar 16, 2026Nandini Arimanda, Achyuth Mukund, Sakthi Balan Muthiah +1Intersectional Bias in Language Models Social Bias in Language Models
Department of Computer Science and Engineering, Shiv Nadar University Chennai, Tamil Nadu, India · School of AI and CS, Plaksha University, Mohali, Punjab, India

Apr 24, 2026J/K move · Enter open · S save

Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregation

Matteo Pelossi, Rita Sevastjanova, Thilo Spinner, Mennatallah El-Assady

ETH Zurich

LLM Auditing Social Bias in Language Models Counterfactual Reasoning in Language Models Language Model Bias Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) exhibit representational and syntactic biases that are difficult to evaluate due to the stochastic nature of text generation. Standard auditing methods rely on a single output inspection or static automated metrics. These approaches obscure the underlying probability distributions and fail to capture biases hidden in lower-probability generation branches. This paper introduces TreeTracer, a visual analytics tool designed to evaluate LLM bias through aggregated comparison. Using a systematic perturbation analysis pipeline, the tool replaces ontology-defined terms in each input prompt, aggregates hundreds of stochastic generations into a syntax-aligned hierarchical structure, and then performs classification-aware node merging with an auxiliary language model. The resulting structure is visualized through a custom Sankey diagram. By juxtaposing two ontology-driven trees, the workspace enables direct comparison between semantic contexts and supports systematic bias detection. Because any visualization reflects only a subset of the model's learned behavior, the system further applies contrastive inference to compute and directly display counterfactual token probabilities across contexts, reducing the risk of misinterpreting the presence of bias. We validate the workspace through case studies comparing an unaligned baseline model GPT-2 XL against the constitutionally aligned Apertus models. The visual aggregation successfully exposes hidden representational harms, such as counterfactual pronoun suppression and conversational marginalization of individuals. A preliminary user study confirms that the aggregated comparative interface reduces cognitive load and effectively supports analysts in detecting systemic biases.

Abstract

Language Model Bias Evaluation

Explore similar work

Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregation

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines

Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models