The shape of individual differences in judgment: shared axes, personal weights, and when their spread crosses the consensus
Organizations: Faculty of Engineering Science, Kansai University, Suita, Osaka, Japan.
Abstract
People differ in how they judge faces, artworks, photographs and jokes. Previous work measured the size of these differences; we ask what shape they take. We wrote three candidate shapes as nested generative models (same-sign reweighting of shared dimensions, a linear transform that allows reversal, and stimulus-specific offsets) and compared how well they predict people unseen in training, using ten public datasets (about 20,000 raters). The models recovered a person's judgment space from 10-20 judgments and predicted judgments of unseen stimuli. For faces, reweighting was sufficient, and weights opposing the population were not statistically detected. For artworks, weights spread continuously along the shared abstract-figurative direction, and 28 percent of raters fell on the opposite side from the population. This tail was much smaller when a different rater sample judged the same artworks. Individual differences thus differ in shape as well as size, and the shape depends on domain and raters.
Figures & tables
| Dataset | Task and stimuli | Raters | Per rater | Stimulus features (unseen stimuli) |
|---|---|---|---|---|
| Xu et al. Experiment 1 [ 33 ] | Multi-arrangement of 20 of 100 White faces (about 60 trials); the retest task used 40 faces, overlapping with the person’s 20 faces by 8 faces on average | 1,751 | 20 faces | Learned embedding (retest task) |
| Xu et al. Experiment 2 [ 33 ] | Multi-arrangement of 20 of 96 White, Black and Asian faces (about 50 trials); the retest was within the person’s 20 faces | 1,682 | 20 faces | Learned embedding (none) |
| London Set [ 36 ] | Attractiveness ratings of 102 faces (1–7, every rater rated every face) | 2,501 (2,513) | 102 faces | CLIP image features (yes) |
| AVFS [ 26 ] | Triplet odd-one-out judgments of face similarity. Main set of 4,921 faces (FFHQ [ 37 ] ), novel set of 56 faces, 1,000 common triplets 24 judgments | main 1,283, novel 504 (1,645) | 60–260 triplets | CLIP (yes: 56 novel faces) |
| FaceSim3D [ 38 ] | Triplets of 100 faces (CFD [ 39 ] ), 2D static and 3D dynamic | 1,219/1,053 (2,354) | median 183 | None |
| SCUT-FBP5500 [ 40 ] | Attractiveness ratings of 5,500 faces (4,000 Asian, 1,500 White) (1–5, every rater rated every face) | 60 | 5,500 faces | CLIP (yes) |
| Analysis | Raters | Image mean | D0 | D1 | D2 | D1 D0 | D2 D1 |
|---|---|---|---|---|---|---|---|
| LAPIS, full | 551 | 0.436 | 0.432 | 0.554 | 0.575 | ||
| LAPIS, subsampled (129 raters, all ratings) | 129 2 | 0.313 | 0.413 | 0.529 | 0.548 | ||
| LAPIS, subsampled (129 raters 227 ratings) | 129 2 | — | 0.438 | 0.530 | 0.552 | ||
| XPASS-Vis art (own model) | 129 | 0.478 | 0.473 | 0.501 | 0.508 | ||
| Transfer: XPASS-Vis raters LAPIS model | 129 | 0.477 | 0.466 | 0.508 | 0.516 | ||
| Transfer: LAPIS raters (same artworks) XPASS-Vis model | about 507 | — | — | — | — |
| Experiment 1 (100 White faces, ) | Experiment 2 (96 multiracial faces, ) | |||
| Held-out | Retest | Held-out | Retest | |
| Population RDM (training folds) | 0.190 | 0.217 | 0.295 | 0.329 |
| Shrinkage blend of own + population | 0.247 | 0.217 | 0.351 | 0.329 |
| D0 shared only | 0.201 | 0.226 | 0.299 | 0.334 |
| D1 same-sign reweighting | 0.202 | 0.227 | 0.338 | 0.367 |
| D2 linear transform | 0.220 | 0.235 | 0.341 | 0.369 |
| Data (measure, number used for estimation) | Raters | Population D1 | Gain D1 D0 | Shape D2 D1 | Loss under the D1 constraint | Tied-pair accuracy |
|---|---|---|---|---|---|---|
| Xu Experiment 2 ( , all) | 1,682 | 0.295 0.338 | n.s. | — | — | |
| Xu Experiment 1 ( , all) | 1,751 | 0.190 0.202 a | n.s. a | — | — | |
| London ( , 20 faces) | 2,501 | 0.590 0.622 | n.s. | (1.6%) | 0.51/0.57/0.59 | |
| AVFS novel (accuracy, 20) | 504 | 0.588 0.650 | — | — | ||
| AVFS main (accuracy, 20/all) | 1,283 | 0.603 0.622/0.630 | / | / | — | — |
| FaceSim3D 2D/3D (accuracy, 20) | 1,219/1,053 | 0.590 0.597/0.599 0.609 | / | / | — | — |
| a Data (measure, estimation size) | Difference | 95% CI | |||||
| Xu Experiment 2 ( , all) | 1,682 | [ , ] | |||||
| Xu Experiment 1 ( , all) | 1,751 | [ , ] | 0.59 | 0.58 | |||
| London ( , 20 faces) | 2,501 | [ , ] | |||||
| AVFS novel (accuracy, 20) | 504 | [ , ] | |||||
| AVFS main (accuracy, 20) | 1,283 | [ , ] | |||||
| AVFS main (accuracy, all) | 1,283 | [ , ] |
| Study | Domain | Shape | New individuals | Judgments per person | Hypothesis comparison | Unseen stimuli |
|---|---|---|---|---|---|---|
| INDSCAL [ 19 ] | Similarity | Weights | No (all individuals fitted jointly) | Complete matrix | — | No |
| IDIOSCAL, three-mode factor analysis [ 20 , 21 ] | Similarity | Transform | No | Complete matrix | Compared by fit [ 22 ] | No |
| Generalised context model [ 23 ] | Categorisation | Weights | Fitted per individual | Hundreds of trials | — | No |
| PsiZ [ 24 , 25 ] | Similarity (triplets) | Weights (per group) | Assigned to a group | Many, as a group | — | No |
| SPoSE, VICE [ 27 , 28 , 29 ] | Similarity (triplets) | None | — | Few (aggregated over the population) | — | No |
| A View From Somewhere [ 26 ] | Face similarity (triplets) | Weights | No (conditioned on annotator ID) | Median 260 | — | Yes, via image features |
Human Preference Modeling
Human preference modeling studies how to infer, represent, and predict people's judgments to evaluate and guide AI systems.
See all 70 papers