Concept frustration: Aligning human concepts and machine representations

Enrico Parisini; Christopher J. Soelistyo; Ahab Isaac; Alessandro Barp; Christopher R. S. Banerji

cs.LGMar 31, 2026

Concept frustration: Aligning human concepts and machine representations

Authors: Enrico Parisini, Christopher J. Soelistyo, Ahab Isaac, Alessandro Barp, Christopher R. S. Banerji

Organizations: Comprehensive Cancer Centre, Guy’s Hospital, King’s College London, Great Maze Pond, London, SE1 9RT, UK · PharosAI, King’s College London, London, SE1 9RT, UK · The Francis Crick Institute, 1 Midland Rd, London, NW1 1AT, UK · Department of Statistical Science, University College London, WC1E 7HB, UK ·

Abstract

Aligning human-interpretable concepts with the internal representations learned by modern machine learning systems remains a central challenge for interpretable AI. We introduce a geometric framework for comparing supervised human concepts with unsupervised intermediate representations extracted from foundation model embeddings. Motivated by the role of conceptual leaps in scientific discovery, we formalise the notion of concept frustration: a contradiction that arises when an unobserved concept induces relationships between known concepts that cannot be made consistent within an existing ontology. We develop task-aligned similarity measures that detect concept frustration between supervised concept-based models and unsupervised representations derived from foundation models, and show that the phenomenon is detectable in task-aligned geometry while conventional Euclidean comparisons fail. Under a linear-Gaussian generative model we derive a closed-form expression for Bayes-optimal concept-based classifier accuracy, decomposing predictive signal into known-known, known-unknown and unknown-unknown contributions and identifying analytically where frustration affects performance. Experiments on synthetic data and real language and vision tasks demonstrate that frustration can be detected in foundation model representations and that incorporating a frustrating concept into an interpretable model reorganises the geometry of learned concept representations, to better align human and machine reasoning. These results suggest a principled framework for diagnosing incomplete concept ontologies and aligning human and machine conceptual reasoning, with implications for the development and validation of safe interpretable AI for high-risk applications.

Figures & tables

Figure 1: The importance of concept frustration. Consider a flatworlder training a model to predict a position on Earth from a complex signal (such as from a satellite), in an interpretable manner. They suggest two known concepts as $C_{1}$ =distance form the North Pole and $C_{2}$ =distance from the South pole. They expect these to be anti-correlated. Their trained model is highly accurate of position, but reasons using an unknown concept $C_{3}$ , which frustrates their known inter-concept semantics. Increasing $C_{3}$ forces $C_{1}$ and $C_{2}$ to be positively correlated. Once $C_{3}$ is understood as distance to the core of a curved Earth, the frustration makes sense and known understanding of the system is improved.

Figure 2: Overview of approach. (A) Raw data is typically processed by a feature extractor (such as a foundation model) to produce an activation vector. (B) Activations can be used to predict an outcome reasoned on a set of intermediate concepts some which may be known prior to predicting the task. This task can be approached using (C) a black box model, (D) a sparse autoencoder (E) a concept bottleneck model.

Figure 3: The treasure hunter task and frustration. A. Consider that we receive a complex signal encoding the position of a treasure, from this signal we wish to predict the binary task of identifying whether a treasure is in a fixed search radius of our position. We employ an interpretable model where known concepts $C_{1}$ and $C_{2}$ are the distance to the North and South polar lines parallel to the Earth’s surface, and we consider the unknown concept $C_{3}$ to denote depth below the surface. We consider two geometries: a flat Earth in which there is no frustration between known and unknown concepts, and a round Earth in which there is frustration. B. Box and violin plots display $\gamma_{F_{A}}$ and $\gamma_{E}$ calculated from 50 simulations of the flat and round Earth geometry treasure hunter tasks, alongside paired Wilcoxon $p$ -values and 95% confidence intervals of the median difference between round and flat Earth metric values. We see that $\gamma_{F_{A}}$ detects frustration while $\gamma_{E}$ does not.

Figure 4: CBM accuracy and interpretability are degraded under frustration. Box and violin plots display (A) accuracy of black box classifiers (B) accuracy of CBM classifiers (C) concept mean-square error (MSE) for CBM classifiers and (D) concept semantic fidelity metric $\beta$ for $\alpha\in\{-1,0,1\}$ , calculated from 600 simulations of our synthetic data generator across a range of parameter regimes, alongside paired Wilcoxon $p$ -values and 95% confidence intervals of the median difference between $\alpha$ values. We see that frustration significantly degrades accuracy and interpretability of CBMs without making the task more difficult for a non-interpretable model

Figure 5: $\gamma_{F_{A}}$ detects frustration in inter-concept covariance and concept task weights between known and unknown concepts Box and violin plots display $\gamma_{F_{A}}$ and $\gamma_{E}$ calculated from 600 simulations of our synthetic data model across 60 parameter regimes separated by (A) $\alpha$ and (B) the sign of $T_{2}$ the theoretical cross alignment term between known and unknown signal. Paired Wilcoxon $p$ -values and 95% confidence intervals of the median difference between metric values across various parameter groups are displayed. We see that $\gamma_{F_{A}}$ is elevated when concept covariance between known and unknown concepts is frustrated, while $\gamma_{E}$ is not. Similarly we see that $\gamma_{F_{A}}$ is consistently elevated when the cross alignment term $T_{2}$ is non-zero, while $\gamma_{E}$ is not.

Figure 6: The value of frustration in real world language and vision tasks. Box and violin plots display $\textrm{Frust}^{F_{A}}(1,2)$ and $\textrm{Frust}^{E}(1,2)$ for CBM1 and CBM2, calculated from 10 fold cross-validation of models trained to predict (A) a language task: sarcasm detection and (B) a vision task: discriminating pictures of gulls from terns. Paired Wilcoxon $p$ -values and 95% confidence intervals of the median difference between metric values between CBM1 and CBM2 are displayed. We see that $\textrm{Frust}^{F_{A}}_{CBM1}(1,2)>\textrm{Frust}^{F_{A}}_{CBM2}(1,2)$ in both settings while $\textrm{Frust}^{E}_{CBM1}(1,2)=\textrm{Frust}^{E}_{CBM2}(1,2)$ .

Figure 7: Effect of decision-boundary window size on frustration detection. Left: Bar plot shows $-\log_{10}(p)$ for the corresponding paired Wilcoxon tests comparing $\gamma^{F_{A}}$ between spherical and cylindrical geometries. Right: Violin and box plots show the paired differences $\Delta\gamma^{F_{A}}=\gamma^{F_{A}}_{\mathrm{sphere}}-\gamma^{F_{A}}_{\mathrm{cylinder}}$ across 50 simulations for different decision-boundary windows $(p_{\mathrm{low}},p_{\mathrm{high}})$ used to compute the Fisher metric. Across all boundary windows, $\gamma^{F_{A}}$ is consistently higher in the spherical geometry than in the cylindrical geometry, indicating robust detection of concept frustration. The effect is quantitatively strongest for the intermediate window $(p_{\mathrm{low}},p_{\mathrm{high}})=(0.2,0.8)$ .

Metric	Cylinder	Sphere	Hodges–Lehmann median paired difference (sphere $-$ cylinder)	$p$
BB acc.	$0.876$ $[0.874,\,0.878]$	$0.894$ $[0.891,\,0.896]$	$0.0180$ $[0.0152,\,0.0207]$	$\mathbf{<10^{-4}}$
CBM acc.	$0.811$ $[0.759,\,0.862]$	$0.824$ $[0.782,\,0.866]$	$0.0132$ $[0.00805,\,0.0177]$	$\mathbf{3.9\times 10^{-3}}$
Concept MSE	$2.81\mathrm{e}{-2}$ $[2.58,\,3.05]\mathrm{e}{-2}$	$2.99\mathrm{e}{-2}$ $[2.70,\,3.29]\mathrm{e}{-2}$	$1.10\mathrm{e}{-3}$ $[-6.34,\,3.15]\mathrm{e}{-3}$	$0.284$
$\mathbf{\gamma^{F_{A}}}$	$0.126$ $[0.0896,\,0.163]$	$0.198$ $[0.163,\,0.233]$	$7.25\mathrm{e}{-2}$ $[0.0213,\,0.127]$	$\mathbf{5.94\times 10^{-3}}$
$\gamma^{E}$	$4.85\mathrm{e}{-2}$ $[4.30,\,5.39]\mathrm{e}{-2}$	$4.47\mathrm{e}{-2}$ $[3.88,\,5.05]\mathrm{e}{-2}$	$-3.52\mathrm{e}{-3}$ $[-8.25,\,9.15]\mathrm{e}{-4}$	$0.118$
$G$	$0.927$ $[0.922,\,0.933]$	$0.926$ $[0.919,\,0.932]$	$-1.87\mathrm{e}{-3}$ $[-6.44,\,2.99]\mathrm{e}{-3}$	$0.469$

Table 1: Supplementary Table S1: Comparison of performance and geometric metrics between cylindrical (flatworld) and spherical (round-world) geometries. Values are means (top) with 95% confidence intervals (bottom). Paired Wilcoxon signed-rank tests compare matched repetitions (sphere minus cylinder).

Metric	$\alpha=0$ Independent	$\alpha=-1$ Consistent	$\alpha=1$ Frustrated	Median diff (+1 minus $-1$ ) (95% CI)	$p$ Paired
BB acc.	$0.935$ $[0.932,\,0.938]$	$0.936$ $[0.930,\,0.943]$	$0.936$ $[0.929,\,0.944]$	$2.03\mathrm{e}{-3}$ $[-5.02\mathrm{e}{-4},\,4.77\mathrm{e}{-3}]$	$0.132$
CBM acc.	$0.783$ $[0.769,\,0.797]$	$0.718$ $[0.703,\,0.734]$	$0.699$ $[0.684,\,0.714]$	$-1.63\mathrm{e}{-2}$ $[-2.25,\,-1.03]\mathrm{e}{-2}$	$\mathbf{<10^{-4}}$
Concept MSE	$2.10$ $[2.05,\,2.15]$	$40.9$ $[26.0,\,55.8]$	$63.7$ $[43.4,\,84.1]$	$6.23$ $[4.42,\,12.5]$	$\mathbf{<10^{-4}}$
$\beta$	$26.0$ $[23.8,\,28.3]$	$258$ $[168,\,347]$	$373$ $[264,\,483]$	$39.8$ $[15.0,\,74.0]$	$\mathbf{<10^{-4}}$
$G$	$0.995$ $[0.994,\,0.996]$	$0.991$ $[0.987,\,0.995]$	$0.985$ $[0.974,\,0.996]$	$-1.76\mathrm{e}{-4}$ $[-7.08\mathrm{e}{-4},\,3.66\mathrm{e}{-4}]$	$0.513$
$\mathbf{\gamma^{F_{A}}}$	$1.60\mathrm{e}{-2}$ $[1.48,\,1.71]\mathrm{e}{-2}$	$2.76\mathrm{e}{-2}$ $[2.51,\,3.02]\mathrm{e}{-2}$	$3.67\mathrm{e}{-2}$ $[3.31,\,4.04]\mathrm{e}{-2}$	$7.76\mathrm{e}{-3}$ $[5.47\mathrm{e}{-3},\,1.00\mathrm{e}{-2}]$	$\mathbf{<10^{-4}}$

Table 2: Supplementary Table S2: Summary statistics across concept regimes defined by

\alpha

. The upper block reports regime means with 95% confidence intervals together with the paired comparison between the frustrated (

\alpha=1

) and consistent (

\alpha=-1

) settings. The lower block reports paired Hodges–Lehmann median shift estimates with 95% confidence intervals and paired Wilcoxon signed-rank

p

-values for comparisons against the independent baseline (

\alpha=0

Metric	CBM1 Mean (95% CI)	CBM2 Mean (95% CI)	Median diff (CBM1 $-$ CBM2)	$p$
CBM acc.	$0.728$ $[0.657,\,0.798]$	$0.805$ $[0.777,\,0.834]$	$-7.20\mathrm{e}{-2}$ $[-1.77\mathrm{e}{-1},\,1.04\mathrm{e}{-3}]$	$0.058$
$\gamma^{E}$	$1.01\mathrm{e}{-3}$ $[6.59\mathrm{e}{-4},\,1.36\mathrm{e}{-3}]$	$1.31\mathrm{e}{-3}$ $[8.00\mathrm{e}{-4},\,1.82\mathrm{e}{-3}]$	$-2.75\mathrm{e}{-4}$ $[-7.82\mathrm{e}{-4},\,1.31\mathrm{e}{-4}]$	$0.492$
$\gamma^{F_{A}}$	$4.47\mathrm{e}{-2}$ $[-1.71\mathrm{e}{-2},\,1.06\mathrm{e}{-1}]$	$5.30\mathrm{e}{-3}$ $[-1.83\mathrm{e}{-3},\,1.24\mathrm{e}{-2}]$	$3.13\mathrm{e}{-3}$ $[3.21\mathrm{e}{-4},\,1.12\mathrm{e}{-1}]$	$\mathbf{0.0488}$
$G$	$0.989$ $[0.987,\,0.991]$	$0.987$ $[0.987,\,0.988]$	$1.50\mathrm{e}{-3}$ $[-4.46\mathrm{e}{-4},\,3.55\mathrm{e}{-3}]$	$0.131$

Table 3: Supplementray Table S3: Comparison of CBM1 and CBM2 performance metrics in the language task. Values are mean (top) with 95% confidence intervals (bottom). Median differences are Hodges–Lehmann estimates from paired Wilcoxon signed-rank tests (CBM1

-

CBM2).

Metric	CBM1 Mean (95% CI)	CBM2 Mean (95% CI)	Median diff (CBM1 $-$ CBM2)	$p$
CBM acc.	$0.639$ $[0.547,\,0.731]$	$0.755$ $[0.680,\,0.829]$	$-1.27\mathrm{e}{-1}$ $[-2.62\mathrm{e}{-1},\,2.25\mathrm{e}{-2}]$	$0.0972$
$\gamma^{E}$	$7.22\mathrm{e}{-3}$ $[6.48\mathrm{e}{-3},\,7.95\mathrm{e}{-3}]$	$6.98\mathrm{e}{-3}$ $[5.87\mathrm{e}{-3},\,8.09\mathrm{e}{-3}]$	$4.29\mathrm{e}{-4}$ $[-9.77\mathrm{e}{-4},\,1.46\mathrm{e}{-3}]$	$0.557$
$\gamma^{F_{A}}$	$6.79\mathrm{e}{-2}$ $[5.67\mathrm{e}{-3},\,1.30\mathrm{e}{-1}]$	$6.49\mathrm{e}{-3}$ $[-3.83\mathrm{e}{-3},\,1.68\mathrm{e}{-2}]$	$5.71\mathrm{e}{-2}$ $[4.65\mathrm{e}{-5},\,1.22\mathrm{e}{-1}]$	$\mathbf{0.0488}$
$G$	$0.995$ $[0.993,\,0.997]$	$0.993$ $[0.991,\,0.995]$	$1.69\mathrm{e}{-3}$ $[-1.60\mathrm{e}{-3},\,5.22\mathrm{e}{-3}]$	$0.232$

Table 4: Supplementrary Table S4: Comparison of CBM1 and CBM2 on the CUB gull–tern classification task using CLIP ViT-B/16 embeddings. Values are mean (top) with 95% confidence intervals (bottom). Median differences are Hodges–Lehmann estimates from paired Wilcoxon signed-rank tests (CBM1

-

CBM2).

Interpretable ML

Interpretable machine learning develops models and explanations that make predictions, evidence, and decision processes understandable to people.

See all 306 papers

May 20, 2026

Investigating Concept Alignment Using Implausible Category Members

Developing AI systems with a human-like understanding of everyday concepts is a key step towards developing safe, reliable systems whose behavior makes sense to humans. When probing concept understanding, asking questions about plausible category members (e.g., "Is a car a vehicle?") is likely to recall patterns in the model's vast training data. We pursue an alternative strategy, characterizing the boundaries of conceptual categories by asking about implausible category members (e.g., "Is an olive a vehicle?") to probe the kind of concept-level knowledge we take for granted in fellow humans. We characterize concept boundaries for a set of fundamental concepts by studying AI systems' assignments of objects to superordinate categories from a classic psychological study by Rosch and Mervis, as well as their assignments of the same objects to mismatched superordinate categories. We compare these assignments to those made by human participants on the full range of within-category and cross-category assignment tasks. Our results reveal a range of concepts for which which models differ in meaningful and surprising ways from humans, including treating "words" as belonging to categories like "vehicles" and "clothing," identifying several "vegetable" category members as "fruit," and assigning exemplars from non-weapon categories to the "weapons" category. We also demonstrate how these instances of concept misalignment translate into problematic downstream behavior with implications for AI safety.

Sunayana Rane, Brenden M. Lake, Thomas L. Griffiths

Department of Computer Science · Princeton University · Princeton, NJ 08540 +1

AI Alignment Concept Alignment

Ask PDF

Jun 29, 2026

On the Faithfulness of Post-Hoc Concept Bottleneck Models

Human decision-making interprets the world through high-level concepts, such as recognizing a bird by its belly color. To bridge the gap between opaque deep learning representations and human understanding, Post-Hoc Concept Bottleneck Models (post-hoc CBMs) project latent features onto interpretable concept spaces using auxiliary datasets or vision-language models. However, relying on target task accuracy as the primary measure of post-hoc CBM success obscures whether the learned concepts are semantically meaningful or merely predictive artifacts. For example, random concept projections can achieve competitive accuracy despite being semantically meaningless. In this work, we analyze the learned projections directly and identify two failure cases: First, for concept projections learned from auxiliary data, covariate shifts can lead to unfaithful concept representations for the target task. In particular, we provide an upper bound on the error introduced by this shift. Second, systematic label noise in surrogate concept labels generated by vision-language models leads to unfaithful projections. After formalizing these failure modes, we introduce novel metrics that decouple concept faithfulness from predictive accuracy. Our empirical results across real-world and synthetic benchmarks confirm that these metrics identify unfaithful behaviors that standard accuracy-based evaluation fails to detect.

Laines Schmalwasser, Jan Blunk, Niklas Penzel +2

Institute of Data Science, German Aerospace Center, Jena, Germany · Computer Vision Group Jena, Friedrich Schiller University Jena, Germany · GEOMAR Helmholtz Centre for Ocean Research Kiel, Germany

Concept Bottleneck Models Explainability Evaluation

Ask PDF

Apr 17, 2026

Prototype-Grounded Concept Models for Verifiable Concept Alignment

Concept Bottleneck Models (CBMs) aim to improve interpretability in Deep Learning by structuring predictions through human-understandable concepts, but they provide no way to verify whether learned concepts align with the human's intended meaning, hurting interpretability. We introduce Prototype-Grounded Concept Models (PGCMs), which ground concepts in learned visual prototypes: image parts that serve as explicit evidence for the concepts. This grounding enables direct inspection of concept semantics and supports targeted human intervention at the prototype level to correct misalignments. Empirically, PGCMs achieve similar predictive performance as state-of-the-art CBMs while substantially improving transparency, interpretability, and intervenability.

Stefano Colamonaco, David Debot, Pietro Barbiero +1

Department of Computer Science, KU Leuven · IBM Research, Zurich

Concept Bottleneck Models Concept Alignment Interpretable ML Explainable Image Classification Prototype-Based Explanation

Ask PDF

Abstract

Interpretable ML

Explore similar work

Investigating Concept Alignment Using Implausible Category Members

On the Faithfulness of Post-Hoc Concept Bottleneck Models

Prototype-Grounded Concept Models for Verifiable Concept Alignment