Cross-Protocol Personality Assessment with LLMs: Aligning Behavioral Evidence with Human Ratings

Wei Shan; Tianyi Zhang; Yuan Zong; Wenming Zheng

cs.AIOct 12, 2026

Cross-Protocol Personality Assessment with LLMs: Aligning Behavioral Evidence with Human Ratings

Authors: Wei Shan, Tianyi Zhang, Yuan Zong, Wenming Zheng

Organizations: Key Laboratory of Child Development and Learning Science (Ministry of Education), School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, China.

Abstract

Personality assessment from asynchronous video interviews (AVIs) plays a crucial role in recruitment by enabling standardized and behaviorally grounded evaluation for job applicants. However, most of the previous works are confined to a single interview protocol. Independently collected protocols differ in question sets and question-trait mapping. Thus, the human-rated evidence from one protocol cannot be used directly under another. To address this limitation, we propose the Cross-protocol Evidence-grounded Personality Assessment (CEPA) framework, which aligns behavioral representations across interview protocols. CEPA addresses protocol heterogeneity by projecting source behavioral records into the target protocol's construct-oriented representation space. It retrieves behaviorally relevant and personality-complementary human-rated exemplars to ground an ensemble of LLM raters, which enables behavior-to-personality associations without target-specific supervision. In cross-protocol experiments, CEPA reduces average mean squared error (MSE) by 43.8% (Forward) and 17.0% (Reverse) relative to the strongest baseline. Our empirical experiments show that human-rated evidence from one protocol can support personality assessment under another without target-specific supervision or retraining.

Figures & tables

Fig. 1: Overview of the proposed CEPA framework. Multimodal interview videos are first converted into textual behavioral representations. Source evidence is aligned to the target protocol and encoded into a shared construct-oriented retrieval space. For a target subject, semantic retrieval forms a candidate pool, PMES selects complementary human-rated exemplars, and independent LLM raters produce continuous personality scores.

Direction	Source	Target	Evaluation space	$n$
Forward	OPVA	AVI-6	H/X/A/C	644
Reverse	AVI-6	OPVA	X/C (facet-mean)	677

TABLE I: Bidirectional cross-protocol transfer settings. Facet-mean denotes facet-level targets aggregated to the factor level.

Mean Squared Error (MSE)
Dataset		AVI-6					OPVA
Category	Approach	H	X	A	C	Avg	X	C	Avg
Textualized LLM ZS	Qwen3-14B	0.5282	0.3919	0.5603	0.5817	0.5155	0.2267	0.1788	0.2027
	Llama-3.1-8B	0.4315	0.3652	0.5210	0.4934	0.4528	0.3360	0.2612	0.2986
	Gemma-3-12B	0.5273	0.3156	0.5773	0.6215	0.5104	0.2170	0.1984	0.2077
	Ours w/o evidence	0.4380	0.2971	0.3991	0.4867	0.4052	0.2217	0.1764	0.1990

TABLE II: External comparison of the Forward and Reverse transfer settings (MSE, lower is better). All baselines are zero-shot (ZS) and receive the same behavioral descriptions without retrieved evidence. Bold denotes the best (lowest) result within each column.

Dir.	Baseline	MSE	$\Delta$ MSE	95% CI
Fwd.	Ours w/o evidence	0.4052	$-0.1773$	$[-0.1990,\,-0.1563]$
Fwd.	Qwen3.5-Plus	0.4550	$-0.2271$	$[-0.2508,\,-0.2036]$
Rev.	Ours w/o evidence	0.1990	$-0.0339$	$[-0.0481,\,-0.0201]$
Rev.	Qwen3.5-Plus	0.2216	$-0.0565$	$[-0.0734,\,-0.0386]$

TABLE III: Paired subject-level bootstrap comparison against the evidence-free scorer (Ours w/o evidence) and the strongest zero-shot textual baseline outside the Cont3 rater set (Qwen3.5-Plus).

Mean Squared Error (MSE)
Dataset		AVI-6					OPVA
Category	Configuration	H	X	A	C	Avg	X	C	Avg
Evidence retrieval	Ours w/o evidence	0.4380	0.2971	0.3991	0.4867	0.4052	0.2217	0.1764	0.1990
Evidence retrieval	Ours w/o PMES	0.2084	0.2728	0.1952	0.4586	0.2838	0.1766	0.1734	0.1750
Representation Alignment	Ours w/o Alignment	0.2517	0.3980	0.2847	0.5199	0.3636	0.3465	0.5005	0.4235
Full model	Ours (full)	0.1998	0.2249	0.2032	0.2837	0.2279	0.1726	0.1576	0.1651

TABLE IV: Component ablation results across Forward and Reverse transfer (MSE, lower is better). All retrieval-based configurations use Cont3 with

(K_{\mathrm{pre}},K)=(50,5)

. Bold denotes the best (lowest) result within each column.

CI: $[-.0839,-.0588]$ ; $[-.1783,-.1401]$
Retrieval	Predictor	Fwd.	Rev.
Sim- $5$	Cont3	0.2838	0.1750
	Neighbor	0.3550	0.3341
	$\Delta$ MSE	$-0.0713$	$-0.1591$
PMES	Cont3 (Ours)	0.2279	0.1651
	Neighbor	0.2905	0.3792

TABLE V: Evidence-grounded Cont3 vs. neighbor-label mean under matched retrieval sets (

K{=}5

). Negative

\Delta

MSE favors Cont3.

(a) Candidate-pool size (fixed $K{=}5$ )
$K_{\mathrm{pre}}$	20	50	100
Minimum $d_{\sigma}$	2.08	2.54	2.85
(b) Evidence budget (fixed $K_{\mathrm{pre}}{=}50$ )
$K$	3	5	10
Minimum $d_{\sigma}$	3.19	2.54	1.82
Context (tokens)	$\sim$ 8.4k	$\sim$ 9.6k	$\sim$ 12.6k

TABLE VI: Forward offline retrieval-budget diagnostics (

n{=}644

Personality Modeling in Language Models

Personality modeling in language models studies how traits are measured, represented, simulated, and controlled in model behavior.

See all 172 papers

LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion
Jun 29, 2026Tianyi Zhang, Wei Shan, Yuan Zong +2Multimodal Fusion Multimodal Learning
Key Laboratory of Child Development and Learning Science (Ministry of Education), School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, China
InterviewSim: A Scalable Framework for Interview-Grounded Personality Simulation
Feb 23, 2026Yu Li, Pranav Narayanan Venkit, Yada Pruksachatkun +1Personality Modeling in Language Models Human Behavior Simulation
Salesforce Research
Frozen Multimodal Embeddings for AI-Assisted Interview Assessment of Personality and Cognitive Ability
Jun 10, 2026Kuo-En Hung, Hung-Yue Suen, Shih-Ching Yeh +1Multimodal Fusion Multimodal Understanding
Technology Application and Human Resource Development, National Taiwan Normal University, Taiwan and HRDA.pro, Taiwan · Technology Application and Human Resource Development, National Taiwan University, Taiwan · Computer Science and Information Engineering, National Central University, Taiwan +1

Jun 29, 2026J/K move · Enter open · S save

LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion

Tianyi Zhang, Wei Shan, Yuan Zong, Tianhua Qi+1

Key Laboratory of Child Development and Learning Science (Ministry of Education), School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, China

Multimodal Fusion Multimodal Learning

Open paper AskPDF arXiv

Personality recognition in asynchronous video interviews (AVIs) has become increasingly important due to their widespread adoption in modern recruitment. Existing approaches often rely on large language models (LLMs) to analyze textual responses of interviewees in AVI. However, unimodel methods often suffer from information loss (e.g., ignore facial cues). In contrast, multimodal methods that employ full-face images or sparsely sampled frames can discard fine-grained temporal dynamics critical for accurate personality assessment. To overcome these limitations, we propose an LLM-based framework that semantically fuse facial action units (AUs) with textual responses of AVI. AU sequences are first converted into interpretable textual descriptions, which are then fused with participants' textual responses through an LLM. A lightweight regression head transforms the resulting embeddings into continuous personality scores without disrupting the underlying semantic space. Experiments on the AVI-6 benchmark demonstrate consistent improvements over most baselines, with lower prediction errors and stronger correlations with human-rated scores across multiple traits. Further analysis reveals that AU-derived semantic representations offer complementary non-verbal cues to textual responses. Decoupling semantic understanding from regression prediction within the LLM also leads to greater training stability and clearer interpretability. Overall, these findings demonstrate that AU-text fusion provides a psychologically grounded and computationally efficient framework for personality recognition in AVIs.

Abstract

Personality Modeling in Language Models

Explore similar work

LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion

InterviewSim: A Scalable Framework for Interview-Grounded Personality Simulation

Frozen Multimodal Embeddings for AI-Assisted Interview Assessment of Personality and Cognitive Ability