cs.CLSep 30, 2025

Feedback Forensics: A Toolkit to Measure AI Personality

Authors: Arduin Findeis, Timo Kaufmann, Eyke Hüllermeier, Robert Mullins

Organizations: University of Cambridge · LMU Munich, MCML Munich · LMU Munich, MCML Munich, DFKI

Abstract

Some traits making a "good" AI model are hard to describe upfront. For example, should responses be more polite or more casual? Such traits are sometimes summarized as model character or personality. Without a clear objective, conventional benchmarks based on automatic checks struggle to measure such traits. Evaluation methods using human feedback such as Chatbot Arena have emerged as a popular alternative. These methods infer "better" personality and other desirable traits implicitly by ranking multiple model responses relative to each other. Recent issues with model releases highlight limitations of these existing opaque evaluation approaches: goblin-obsessed, sycophantic and generally over-the-top personalities have resulted in model rollbacks and extensive media coverage. Despite these known issues, limited public tooling exists to explicitly evaluate model personality. We introduce Feedback Forensics: an open-source toolkit to track AI personality and related behavioural traits, both those encouraged by human (or AI) feedback, and those exhibited across AI models trained and evaluated on such feedback. Feedback Forensics is adaptive and can automatically find new traits appearing in models. We demonstrate the toolkit's usefulness in two steps: (A) first we analyse the personality traits encouraged in popular human feedback datasets including MultiPref, PRISM and data from Chatbot Arena; and (B) then use our toolkit to analyse how much popular models exhibit such traits. We release (1) our Feedback Forensics toolkit alongside (2) a web app tracking AI personality in popular models and feedback datasets and (3) the underlying annotation data, see https://github.com/rdnfn/feedback-forensics.

Figures & tables

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Persona Cartography: Charting Language Model Personality Traits in Weight Space

    Jul 8, 2026Luke Baines, Anton Gonzalvez Hawthorne, Mariia Koroliuk +4Personality Modeling in Language Models

  2. Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalities

    Mar 20, 2026Avinash Krishna, Kalyana Chadalavada, Unso Eun Seo JoPersonality Modeling in Language ModelsBehavioral Consistency of LLM Agents

  3. AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models

    Nov 10, 2026Zihan Chen, Lei Nico Zheng, Di ZhuLLM Evaluation