Personalized game generation requires inferring a player's abilities and behavioral style from how they play. Large language models have made this inference more attainable than ever: an LLM can read a raw gameplay transcript and produce a fluent, plausible profile of the player. Plausible, however, is not verified, and verification is precisely what the field lacks: latent traits are unobservable; questionnaires provide noisy proxies and become circular when self-reports are used to validate behavior-based inference; and behavior itself is ambiguous without context -- a player who never collects an item may not want it, or may never have had the chance. We address both problems. First, we construct a synthetic player population whose traits are ground truth by construction: each trait is an explicit bot parameter, accepted only after controlled manipulation produces consistent, trait-specific behavioral change. Unlike prior parameter-recovery work that inverts a known decision model, our benchmark evaluates policy-agnostic inference from behavioral transcripts alone. Second, we introduce an opportunity-aware decision-moment representation that disentangles preference from the chance to express it; ablating it selectively degrades opportunity-dependent traits. On this benchmark, few-shot LLM inference outperforms embedding- and rule-based baselines on most traits, though feature-based supervised regressors remain stronger overall. Finally, we close the loop: inferred profiles drive difficulty adaptation, evaluated against ground-truth references and mismatched-profile controls, and an exploratory human study examines whether these findings transfer to real players.
Figures & tables
Figure 1: Overview. (A) Synthetic players with parameterized traits produce opportunity-aware records; parameters enter the benchmark only after passing the admission test, and trait recovery is scored against them. (B) The same record schema drives the deployed pipeline: calibration play, LLM reading, and profile-conditioned generation.
Figure 2: Trait-parameter monotonicity validation: each parameter produces the expected monotone trend in its held-out behavioral probe (mean ± std over 5 seeds per level, 6 levels each).
Method
Dodge
Coll.
Aggr.
Skill
Macro
Rule
.67
.50
.45
.89
.63
FeatReg
.77
.56
.83
.91
.77
EmbReg
.65
−.04
.23
.67
.38
BPM 1-NN
.53
.32
.58
.84
.56
GMM (soft, K=8 )
.36
.38
.62
.76
.53
GMM (hard)
.32
.38
.56
.75
.50
Table 1: Recovery on the frozen test partition (Spearman ρ ). Best in bold , second best underlined . Supervised baselines are trained on 210 labeled sessions drawn from the same distribution as the test set; the LLM rows are our few-shot protocol instantiated with the indicated backbone.
Table 3: Collector recovery on the diagnostic probe level (Spearman ρ , n=30 per condition).
Bot
Truth
Inferred
Mismatch
Fixed
Easing
Novice
1.04 †
1.21 †
1.17 †
1.25 †
1.07 †
Low-mid
0
0
1.15 †
1.19 †
1.16 †
Mid
0
0.33
1.06 †
1.02 †
0
High
0
0
1.13 †
0
0
Expert
0
0.19
1.19 †
0.21
0.28
Survived
4/5
4/5
0/5
2/5
3/5
Table 4: Flow-matching score per agent level and arm (lower is better; 0 = inside the flow band). † marks a run that ended in death, which scores above 1 by construction. One run per cell.
Figure 3: Pilot study ( n=12 ). Per-participant LLM ability readings under three level-generation conditions; grey lines connect the same participant.
Understanding how large language models (LLMs) evaluate gameplay experience is important for using their feedback to guide game design. However, overall ratings alone provide limited insight into which aspects of experience are associated with these judgments and how they can inform concrete game changes. We introduce PREFER, a framework that uses a shared set of experience dimensions adapted from established psychometric questionnaires to benchmark how models evaluate gameplay and to guide persona-specific game refinement. Through these dimensions, we characterize the perspectives models offer as virtual playtesters by examining how their overall assessments relate to specific aspects of experience and to the gameplay they produce. To examine how the same dimensions can guide refinement for players with different preferences, we use personas to simulate distinct player profiles. For each persona, PREFER uses experience-preference relationships to select refinement objectives, revises game rules and their implementations, and playtests candidate games to assess experience changes and predicted preference gains. Experiments with five personas show that PREFER achieves 9.2% higher mean post-play preference ratings than the strongest evaluated baseline.
We examine whether human psychometric questionnaires can serve as reliable tools for characterizing and predicting LLM behavior in everyday user interactions. We analyze eight open-source LLMs by comparing their value and personality profiles derived from two different methods: Likert self-reports on established questionnaires (PVQ-40/21 and BFI-44/10) and generation probabilities over value-laden responses to everyday user queries. The two profiles diverge substantially. Within-construct item consistency, often cited as evidence of stable LLM dispositions, disappears in generation probabilities. We find that established questionnaire items contain explicit lexical cues that allow models to recognize the target construct and respond in alignment-consistent, socially desirable ways, whereas realistic user queries contain far less recognizable cues. In addition, demographic persona prompts shift models' responses to human questionnaires in ways consistent with real human patterns, but no such shifts appear in the generation setting, highlighting that human questionnaires overestimate LLMs' ability to faithfully reproduce expected psychological traits when role-playing demographic personas. Overall, our study indicates that questionnaire scores alone should not be treated as evidence of LLMs' response tendencies in realistic user interactions, and supports generation-probability profiling with ecologically valid items as a complementary behavioral measure.
Woojung Song, Dongmin Choi, Yoonah Park +3
1Graduate School of Data Science, Seoul National University · Department of Communication, Interdisciplinary Program in Artificial Intelligence, Seoul National University
Large language models (LLMs) are increasingly deployed as autonomous decision-makers in strategic settings, yet we have limited tools for understanding their high-level behavioral traits. We use activation steering methods in game-theoretic settings, constructing persona vectors for altruism, forgiveness, and expectations of others by contrastive activation addition. Evaluating on canonical games, we find that activation steering systematically shifts both quantitative strategic choices and natural-language justifications. However, we also observe that rhetoric and strategy can diverge under steering. In addition, vectors for self-behavior and expectations of others are partially distinct. Our results suggest that persona vectors offer a promising mechanistic handle on high-level traits in strategic environments.