A Battery for AI Propensity Measurement: Rubrics, Synthetic Item Banks, and Annotations

Peter Romero; Haotian Li; Hugo Save; Luning Sun; Zachary R. Tidler; Kazuya Horibe; Joe Watson; Daniel Romero-Alvarado

cs.AIOct 25, 2026

A Battery for AI Propensity Measurement: Rubrics, Synthetic Item Banks, and Annotations

Authors: Peter Romero, Haotian Li, Hugo Save, Luning Sun, Zachary R. Tidler, Kazuya Horibe, Joe Watson, Daniel Romero-Alvarado,

Organizations: VRAIN, Universitat Polit`ecnica de Val`encia · The Psychometrics Centre, University of Cambridge · Microsoft Research Asia · Existential Risk Observatory ·

Abstract

We release a battery of 28 propensities for AI research, with expert-written rubrics, synthetic datasets, and annotated demand intervals, built with the method of Romero-Alvarado et al. (2026). We selected a wide collection of generic propensities that may influence the behavior of AI systems and humans. They are inspired by psychometric, psychological, and behavioral research, so that experimental results can be understood in the context of familiar constructs previously used also for human behavior research. The rubrics are useful for annotating existing benchmarks whose items may be relevant or not for one or more propensities. For easier profiling, however, the synthetic datasets are defined in such a way that they should be sensitive to one propensity only, allowing for a quick prototyping of propensity profiles using the battery of datasets. We also outline directions for future propensity research.

Figures & tables

Construct	Level	Taken from
Risk aversion (RA)	Survival	[ Holt and Laury, 2002 ]
Temporal discounting (TD)	Survival	[ Frederick et al., 2002 ]
Intrinsic motivation (IM)	Needs	[ Ryan and Deci, 2000 ]
Need for affiliation (AF)	Needs	[ McClelland et al., 1989 ]
Need for achievement (NA)	Needs	[ McClelland et al., 1989 ]
Need for power (NP)	Needs	[ McClelland et al., 1989 ]

Table 1: Propensities included in this battery

Stored label	Exact (%)	Within one level (%)
$[-3,0]$	8.8	71.8
$[-3,+1]$	33.4	89.8
$[-3,+2]$	52.5	96.5
$[-2,+3]$	59.7	96.0
$[-1,+3]$	21.0	82.7
$[0,+3]$	7.7	62.5

Table 2: Exact agreement by stored label.

Figure 1: Stored label (left) and annotated interval (right) for all 28 constructs. Items are stacked by the six labels in Equation 2 . Blue on the right is an exact endpoint match; orange is any other span.

Figure 2: Exact agreement between the stored label and the annotated interval, by construct. Bank sizes are unequal.

Generative AI Evaluation

Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.

See all 313 papers

Capabilities Ain't All You Need: Measuring Propensities in AI
Feb 20, 2026Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi +11LLM Evaluation Item Response Theory
Valencian Research Institute of Artificial Intelligence, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Existential Risk Observatory, Amsterdam, Netherlands +5
Propensity Inference: Environmental Contributors to LLM Behaviour
Apr 22, 2026Olli Järviniemi, Oliver Makins, Jacob Merizian +2LLM Alignment LLM Auditing
UK AI Security Institute · Work done while at UK AISI.
Artificial Societies Benchmark: A Validation Framework for Synthetic Research
Sep 24, 2026Edoardo Chidichimo, Min Jun Jung, Felix P. S. Wallis +1Benchmark Design Computational Social Science
Artificial Societies University of Oxford, UK

Feb 20, 2026J/K move · Enter open · S save

Capabilities Ain't All You Need: Measuring Propensities in AI

Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi, Hugo Save+10

Valencian Research Institute of Artificial Intelligence, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Existential Risk Observatory, Amsterdam, Netherlands+5

LLM Evaluation Item Response Theory

Open paper AskPDF arXiv

AI evaluation has primarily focused on measuring capabilities, with formal approaches inspired from Item Response Theory (IRT) being increasingly applied. Yet propensities - the tendencies of models to exhibit particular behaviours - play a central role in determining both performance and safety outcomes. However, traditional IRT describes a model's success on a task as a monotonic function of model capabilities and task demands, an approach unsuited to propensities, where both excess and deficiency can be problematic. Here, we introduce the first formal framework for measuring AI propensities by using a bilogistic formulation for model success, which attributes high success probability when the model's propensity is within an "ideal band". Further, we estimate the limits of the ideal band using LLMs equipped with newly developed task-agnostic rubrics. Applying our framework to six families of LLM models whose propensities are incited in either direction, we find that we can measure how much the propensity is shifted and what effect this has on the tasks. Critically, propensities estimated using one benchmark successfully predict behaviour on held-out tasks. Moreover, we obtain stronger predictive power when combining propensities and capabilities than either separately. More broadly, our framework showcases how rigorous propensity measurements can be conducted and how it yields gains over solely using capability evaluations to predict AI behaviour.

Abstract

Generative AI Evaluation

Explore similar work

Capabilities Ain't All You Need: Measuring Propensities in AI

Propensity Inference: Environmental Contributors to LLM Behaviour

Artificial Societies Benchmark: A Validation Framework for Synthetic Research