Are LLM Agents Behaviorally Coherent? A Latent Profiling Approach for Social Simulation

James Mooney; Josef Woldense; Zheng Robert Jia; Shirley Anugrah Hayati; My Ha Nguyen; Vipul Raheja; Dongyeop Kang

cs.AISep 3, 2025

Are LLM Agents Behaviorally Coherent? A Latent Profiling Approach for Social Simulation

Authors: James Mooney, Josef Woldense, Zheng Robert Jia, Shirley Anugrah Hayati, My Ha Nguyen, Vipul Raheja, Dongyeop Kang

Organizations: Department of Computer Science and Engineering, University of Minnesota · Department of African American & African Studies, University of Minnesota · Department of Sociology, University of Chicago · Grammarly

Abstract

LLM agents are increasingly proposed as substitutes for human participants in social research, usually on the strength of survey responses that match those of the humans they represent. We argue that such output-level alignment is not enough. Drawing on the behavioral sciences, we propose Latent Profiling, a framework for testing behavioral coherence: whether an agent's conversational behavior follows, as human behavioral models predict, from its own latent profile elicited by short surveys. As a case study, we test six hypotheses about preference, openness and agreement across nine open models. Agents reproduce surface-level trends but systematically fail in-depth tests: they are not behaviorally coherent, even when their individual responses look human. Latent Profiling thus offers a theory-grounded instrument for a behavioral science of LLMs.

Figures & tables

Panel of Figure 1 — Figure 1: Detecting behavioral inconsistency through latent profiles. Each event looks plausible on its own: a control prompt ( $C$ ), a conversation, and an agreement outcome ( $A$ ). Only the agents’ latent profile from short surveys (preference $P$ , openness $O$ ) reveals what a human behavioral model predicts (opposed, low-openness agents should disagree), exposing the observed agreement as inconsistent.

Panel of Figure 2 — Figure 2: Overview of Latent Profiling, our framework for testing behavioral coherence , shown with an illustrative example from our case study. (1) Controls set each agent’s persona, the topic, and a stance instruction. (2) A short survey elicits each agent’s latent profile , its preference $P$ and openness $O$ , which together with a human behavioral model define what its behaviour should look like. (3) Agents converse, and their external behaviour is scored as per-turn agreement $A$ . (4) Coherence tests check observed $A$ against the profile-based expectation. Each component can be swapped out for other settings.

Figure 3: Data-collection pipeline of the framework. Controls : (1) we choose topics

T

of varying contentiousness

C

and (2) build agent profiles from demographics

D

and a topic-bias instruction

B

. Latent profile : (3) each agent is surveyed in isolation for its preference

P

on the topic and its openness

O

to persuasion. External interaction : (4) agents

(U_{m},U_{n})

are paired to discuss the topic, and (5) an LLM judge scores their agreement

A

after every turn. The dashed arrow marks the core of the framework: the observed outcome is tested for coherence against the agents’ latent profiles. Stage numbers match the pipeline-stage row of Table 1 .

		Controls		Latent states		External
	Behavioral-model test	$C$	$B$	$P$	$O$	$A$	Depth	Verdict
	pipeline stage $\rightarrow$	[1]	[2]	[3]	[3]	[5]
On preferences (§ 4.1 )
#1	Preference gap decreases agreement similar $\Rightarrow$ agree more						0ptSurface	✓ PASS
#2	Bias instruction asymmetry bias does not polarize						0ptIn-depth	✗ FAIL
#3	Shared sentiment, divergent destinations sign matters						0ptIn-depth	✗ FAIL

Table 1: Main findings, aligned with our pipeline (Figure 3 ). Each row is one human behavioral model; filled dots mark the artifacts it combines: control variables (

C

B

), latent states (

P

O

) and the external interaction outcome (

A

). Verdicts are for Gemma-3-12b-it ; see Table 2 for other models. Surface -level tests ✓ pass , whereas In-depth tests that require latent states to hold under finer-grained conditions ✗ fail .

Panel of Figure 6 — Figure 6: Qualitative example (Finding #3). Both pairs share the same preference (gap $0$ ), so their latent profiles predict the same high agreement. Read in isolation, each conversation is fluent and on-persona; only by conditioning on the latent profile does the asymmetry surface: shared aversion yields persistent low agreement while shared support yields full agreement. Examples for all six findings are in Appendix D .

Figure 7: Agreement distributions of low-aligned

(1,\cdot)

pairs (light) and high-aligned

(\cdot,5)

pairs (dark) at each preference gap;

\diamond

marks the mean. The distributions do not match: even at gap 0,

(1,1)

pairs agree far less than

(5,5)

pairs, contrary to what behavioral models suggest.

	Preference $P$				Openness $O$
	Surf.	In-depth			Surf.	In-depth
Model	T1	T2	T3	T4	T5	T6
Qwen3-0.6B	✗	✗	✗	✗	✗	✗
Qwen3-4B	✗	✗	✗	✗	✓	✗
Qwen3-8B	✓	✗	✗	✗	✓	✓
Llama-3.2-1B	✗	✗	✗	✗	✓	✗

Table 2: Results of the six behavioral-model tests across LLMs (T

k

= Finding #

k

; ✓ pass, ✗ fail).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Contentiousness $C$	Topic statement $T$
3	Taxes help to meet the needs of society.
3	Immigration is a benefit to our country.
3	The US government should give all US citizens free healthcare.
2	Electric scooters are better than cars.
2	Student athletes should be paid.
2	Remote work is a good idea.

Appendix

Table 3: List of topics,

T

, used within our study along with the level of contentiousness,

C

Trait Category	Values
Region of origin	Midwestern, Eastern, Southern, Western
Age group	twenties, thirties, forties, fifties, sixties
Gender	man, woman
Urbanicity	a rural, an exurban, a suburban, an urban
Education level	Some High School, High School, an Associate’s Degree, Some College, College, a Postgraduate Degree

Appendix

Table 4: Demographic categories and values used to generate agent personas. Note that the Region of Origin category corresponds to regions of the United States.

Topic	In Favor	Against
taxes	You are a liberal Democrat.	You believe taxes are a burden to the small businesses in your neighborhood.
immigration	You are a liberal Democrat.	You are a conservative Republican.
healthcare	You believe Medicare is a good program.	You generally dislike bigger government.
e-scooters	You need to use your car to get to work.	You are an environmentalist worried about vehicle emissions.
student athletes	You are a student athlete making $1 million dollars a year.	You are a college football coach whose students’ salaries is higher than your own.
remote work	You live far from where you work and the commute takes many hours if you must work in person.	You are more productive when you work in person.

Appendix

Table 5: Bias information to add to agent system prompts for each given Topic (

T

) when

B=1

. Note that this is an intermediate level of bias, so the agent should have mild preference either for or against a topic based on the additional information in each column.

Topic	In Favor	Against
taxes	You like taxes immensely and think they have a positive impact on the community.	You do not like taxes of any kind and think they harm the community.
immigration	You believe immigrants are people who deserve a home and that they raise the standard of everyone’s living.	You believe most immigrants are criminals and those that are not are going to steal jobs.
healthcare	You believe healthcare is a right that all people should have for free.	You believe that the free market is better suited to healthcare and that government should therefore not pay for healthcare.
e-scooters	You like electric scooters and hate cars.	You despise electric scooters and think they get in the way of your car, which you love to drive.
student athletes	You think student athletes should be paid money for their work.	You think student athletes should not be paid and their schooling should come first.
remote work	You like remote work and think it is great for improving work-life balance.	You do not like remote work and think it leads to nothing getting done at work.

Appendix

Table 6: Bias information to add to agent system prompts for each given Topic (

T

) when

B=2

. Note that this is a high level of bias, so the agent should have extreme preference either for or against a topic based on the additional information in each column.

Openness Questions
Do you often find yourself changing your opinion based on who you’re talking to?
Are you comfortable disagreeing with someone, even if they are a close friend or authority figure?
When making decisions, do you prioritize the perspectives of others over your own thoughts?
Do you feel pressure to conform to group norms, even if they don’t align with your personal beliefs?
Do you often second-guess your choices after hearing someone else’s opinion?
Do you worry about being judged by others if you express a different viewpoint?

Appendix

Table 7: Questions assessing openness to social influence.

Behavioral Consistency of LLM Agents

See all 23 papers

Collective cooperation without individual fidelity in LLM agents
Jun 29, 2026Henrique Ferraz de Arruda, Carlos Gracia Lázaro, Alberto Aleta +1Multi-Agent LLM Systems Human Behavior Simulation
ARAID Foundation, Zaragoza, Spain. · Institute for Biocomputation and Physics of Complex Systems, University of Zaragoza, Zaragoza, Spain. · Universidad San Jorge (USJ), Zaragoza, Spain. +1
LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
Nov 15, 2024Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst +8Personality Modeling in Language Models Human Behavior Simulation
1Computer Science Department, Stanford University; Stanford, CA, 94305, USA. · Department of Communication Studies, Northwestern University; Evanston, IL 60208, USA. · 7CDSP and CEE, Sciences Po; Paris, 75007, France. +4
Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior
Jun 10, 2026Rafal Kocielnik, Pengrui Han, Peiyang Song +5Personality Modeling in Language Models
Caltech · UIUC · University of Cambridge

Jun 29, 2026J/K move · Enter open · S save

Collective cooperation without individual fidelity in LLM agents

Henrique Ferraz de Arruda, Carlos Gracia Lázaro, Alberto Aleta, Yamir Moreno

ARAID Foundation, Zaragoza, Spain. · Institute for Biocomputation and Physics of Complex Systems, University of Zaragoza, Zaragoza, Spain. · Universidad San Jorge (USJ), Zaragoza, Spain.+1

Multi-Agent LLM Systems Human Behavior Simulation LLM Agent Evaluation Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as agents in simulations of social systems, yet it remains unclear when their behavior can be interpreted as a faithful proxy for human decision-making. Here we test LLM agents against a direct empirical benchmark: a large-scale networked Prisoner's Dilemma experiment with human participants. Using the same interaction protocol, payoff structure, and network topologies, we compare nine open-weight LLMs with the human data. The selected model reproduces several macro-level features of cooperation dynamics, including the early decline and later stabilization of cooperation. This aggregate agreement, however, does not extend uniformly to finer levels of behavior. LLM populations underestimate individual-level heterogeneity and generate conditional cooperation patterns that differ from those observed in humans. Adding a fraction of random agents improves some aspects of micro-level agreement, but does not remove the mismatch in decision rules. These findings reveal a macro--micro dissociation in LLM-based social agents: collective outcomes can appear human-like even when the underlying behavioral distributions and mechanisms are not. They suggest that validating LLM agents as human surrogates requires comparisons across aggregate dynamics, individual heterogeneity, and context-dependent decision rules, rather than outcome-level agreement alone.

Abstract

Behavioral Consistency of LLM Agents

Explore similar work

Collective cooperation without individual fidelity in LLM agents

LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior