Simulating Organized Group Behavior: New Framework, Benchmark, and Analysis

Xinkai Zou; Yiming Huang; Zhuohang Wu; Jian Sha; Nan Huang; Longfei Yun; Jingbo Shang; Letian Peng

cs.CLApr 10, 2026

Simulating Organized Group Behavior: New Framework, Benchmark, and Analysis

Authors: Xinkai Zou, Yiming Huang, Zhuohang Wu, Jian Sha, Nan Huang, Longfei Yun, Jingbo Shang, Letian Peng

Organizations: University of California, San Diego · University of California, Irvine · Meta

Abstract

Simulating how organized groups (e.g., corporations) make decisions (e.g., responding to a competitor's move) is essential for understanding real-world dynamics and could benefit relevant applications (e.g., market prediction). In this paper, we formalize this problem as a concrete research platform for group behavior understanding, providing: (1) a task formalization with benchmark and evaluation criteria, (2) a structured and adaptive analytical framework, and (3) detailed temporal and cross-group analysis. Specifically, we propose Organized Group Behavior Simulation, a task that models organized groups as collective entities from a practical perspective: given a group facing a particular situation (e.g., AI Boom), predict the decision it would take. To support this task, we present GROVE (GRoup Organizational BehaVior Evaluation), a benchmark covering 44 organized groups with 8,052 real-world context-decision pairs collected from Wikipedia and TechCrunch across 9 domains, with an end-to-end evaluation protocol assessing consistency, initiative, scope, magnitude, and horizon. Beyond straightforward prompting pipelines, we propose a structured analytical framework that converts collective decision-making events into an interpretable, adaptive, and traceable behavioral model, achieving stronger performance than summarization- and retrieval-based baselines. It further introduces an adapter mechanism for time-aware evolution and group-aware transfer, and traceable evidence nodes grounding each decision rule in originating historical events. Our analysis reveals temporal behavioral drift within individual groups, which the time-aware adapter effectively captures for stronger prediction, and structured cross-group similarity that enables knowledge transfer for data-scarce organizations.

Figures & tables

Figure 1: Framework for Simulating Organized Group Behavior.

Figure 2: Curation and Evaluation of GROVE Benchmark

Figure 4: Adaptation settings and results for groups with significant drift (

*

), non-significant drift (n.s.), and cross-source transfer from Wikipedia to TechCrunch.

Figure 6: Cases for temporal behavior drift (left), group similarity and knowledge transfer (right) with traceable evidence node for validation

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Dom.	Group	Tr	Te	Dom.	Group	Tr	Te
Aero.	Boeing	111	48	Spo.*	CR Vasco	148	64
	Brit. Caledonian	195	84		Dallas Cowboys	142	61
	Cubana	99	43		Bayern Munich	72	32
	United Airlines	103	45		Fluminense	296	127
Auto.	Alfa Romeo	133	58		Grêmio	129	56
	Chrysler	105	46		McLaren	180	78

Appendix

Table 3: Experiment dataset statistics.

Panel of Figure 7 — Figure 7: Group behavior filtering prompt with three criteria: attribution, successor, and significance tests.

Source	Question	Agree (%)	$\kappa$	Final Human–GPT (%)
Wikipedia	Q1: Group Behavior	89.0	0.76	92.0
	Q2: Rewrite Fidelity	96.0	0.91	97.0
	Q3: Question Quality	93.0	0.82	94.0
TechCrunch	Q4: Summary Accuracy	92.0	0.78	90.0
TechCrunch	Q5: No Leakage	94.0	0.84	96.0

Appendix

Table 4: Benchmark quality validation. “Agree” is agreement between the two primary annotators. “Final Human–GPT” is agreement between the adjudicated human label and GPT-4.1 over all sampled items.

Comparison	Agree (%)	$\kappa$	Spearman $\rho$
H–H (primary annotators)	78.0	0.67	0.74
Final H–GPT-4.1	73.0	0.60	0.71

Appendix

Table 5: Consistency-judge validation. H–H compares the two primary annotators; Final H–GPT compares the adjudicated human label against GPT-4.1 over all 100 pairs. The lower block gives per-class Final H–GPT agreement.

Figure 8: CDT Visualization for Nintendo

Figure 9: A running case for one data sample from Nintendo

Symbol	Description	Value
$D_{\max}$	Maximum tree depth	3
$R$	Clustering rounds (guide suffixes)	4
$m$	Observations sampled per centroid	8
$k$	Hypotheses generated per cluster	3
$\tau_{\mathrm{accept}}$ / $\tau_{\mathrm{keep}}$	Precision threshold to accept / keep a statement	0.65
$\tau_{\mathrm{reject}}$ / $\tau_{\text{delete}}$	Precision threshold to reject / delete a statement	0.35

Appendix

Table 6: Hyperparameters for CDT construction (§ A.1 ) and adaptation (§ 3.4 ). Thresholds

\tau_{\mathrm{accept}}

and

\tau_{\mathrm{keep}}

share the same value, as do

\tau_{\mathrm{reject}}

and

\tau_{\text{delete}}

Panel of Figure 10 — Figure 10: Example of a human-written Wikipedia profile used as background knowledge (Boeing).

Prediction Model	Vanilla	Human Profile	Summarization	RAG	Ours (CDT)
Qwen3-4B-Instruct-2507	57.9 / 79.8	59.4 / 83.0	64.6 / 86.2	56.8 / 83.8	68.2 / 86.6
Qwen3.5-4B	55.0 / 82.9	53.8 / 81.7	58.7 / 85.7	58.1 / 81.3	63.1 / 88.3
gpt-5-nano-2025-08-07	76.1 / 88.8	78.1 / 88.8	77.7 / 90.5	76.2 / 88.5	83.1 / 91.6
gpt-5-mini-2025-08-07	80.4 / 89.8	80.2 / 90.1	81.6 / 92.1	77.9 / 86.0	84.4 / 93.0
gpt-5-2025-08-07	83.0 / 89.7	82.8 / 89.9	84.2 / 90.3	81.5 / 85.2	87.2 / 92.0
gemini-3-flash-preview	64.0 / 85.9	67.4 / 86.2	78.0 / 88.0	74.1 / 75.9	79.2 / 89.4

Appendix

Table 7: Consistency results across prediction models. Each entry reports Wikipedia / TechCrunch average.

Prediction Model	Fixed CDT	Retrained CDT	Adapted CDT
Qwen3-4B-Instruct-2507	66.2	66.6	68.5
Qwen3.5-4B	61.1	61.4	65.1
gpt-5-nano-2025-08-07	80.1	82.0	86.0
gpt-5-mini-2025-08-07	83.6	83.4	85.3
gpt-5-2025-08-07	85.5	86.1	88.3
gemini-3-flash-preview	79.8	79.7	80.1

Appendix

Table 8: Temporal adaptation consistency results across prediction models.

Config.	$\boldsymbol{\tau_{\mathrm{accept}}}$	$\boldsymbol{\tau_{\mathrm{reject}}}$	gpt-5-nano	gpt-5-mini	Qwen3-4B-Inst.	Average
C0	0.65	0.35	80.2	84.2	64.6	76.33
C1	0.55	0.35	82.3	85.5	63.5	77.10
C2	0.75	0.35	80.4	84.6	63.1	76.03
C3	0.85	0.35	83.6	83.2	60.5	75.77
C4	0.65	0.25	85.1	82.4	65.7	77.73
C5	0.65	0.45	81.2	80.9	63.9	75.33

Appendix

Table 9: Threshold sensitivity on five representative groups. C0 is the default configuration.

	Wikipedia										TC
Method	Aero.	Auto.	Broad.	Edu.	Ener.	Game	Ret.	Spo.	Tech.	Avg	Avg
Vanilla	80.0	80.7	79.8	78.7	86.5	74.2	72.2	77.2	82.7	79.1	89.9
Human Profile	81.0	78.8	78.7	75.3	80.2	75.8	66.7	77.9	81.5	77.3	88.0
Summary-based	80.2	78.9	79.6	79.0	87.5	74.2	69.4	77.9	80.5	78.6	88.7
RAG	77.8	74.5	80.9	80.1	85.4	77.5	69.4	75.0	75.6	77.4	85.9
Ours (CDT)	82.8	82.8	85.8	81.0	92.7	79.2	68.1	82.7	78.9	81.6	90.2

Appendix

Table 10: Consistency results on GROVE benchmark (gpt-4o-mini).

Domain	Source	Target	Vanilla	Tgt CDT	Transfer
Aero.	Brit. Cal.	Airbus	84.1	79.5	79.5
Auto.	Volkswagen	Ford	81.0	81.0	88.1
Spo.	Flumin.	Derry City	80.0	80.0	83.3
Spo.	SE_Palmeiras	GELP	87.5	70.8	85.4
Edu.	Cornell	Princeton	100.0	96.2	96.2
	Cornell	UPenn	66.7	76.7	80.0

Appendix

Table 11: CDT Transfer Experiment (gpt-4o-mini, Consistency Score)

Domain	Source	Target	Vanilla	Tgt CDT	Transfer
Aero.	Brit. Cal.	Airbus	71.6	62.5	76.1
Auto.	Volkswagen	Ford	79.2	75.0	82.1
Spo.	Flumin.	Derry City	74.2	68.3	71.2
Spo.	SE_Palmeiras	GELP	79.2	60.9	72.9
Edu.	Cornell	Princeton	74.0	76.9	80.8
	Cornell	UPenn	81.7	77.5	83.3

Appendix

Table 12: CDT Transfer Experiment (Qwen2.5-7B-Instruct, 4-Dim Score)

Figure 11: Event-level Analysis for Group Similarity on Technology, Aerospace, Education, Sports and Automotive

Figure 12: Event-level Analysis for Group Similarity on Broadcast, Energy, Game, Retail

	Wikipedia
Method	Aero.	Auto.	Broad.	Edu.	Ener.	Game	Ret.	Spo.	Tech.	Avg
Ours (CDT)	74.1	77.1	74.4	74.8	76.6	75.6	67.2	74.4	78.0	74.7
Ours (CDT retrained)	72.9	75.9	74.5	71.6	78.1	75.6	61.5	74.3	79.9	73.8
Ours (CDT + Adaptation)	74.1	76.7	73.2	78.0	76.6	80.0	62.5	76.3	81.0	75.4

Appendix

Table 13: Temporal Adaptation Study — 4-Dim (Qwen2.5-7B-Instruct).

Panel of Figure 13 — Figure 13: Hypothesis generation prompt (Step 2 of CDT construction). Given clustered scene–action pairs and existing statements, the LLM proposes $k$ behavioral statements and $k$ corresponding gate questions via chain-of-thought reasoning.

Panel of Figure 14 — Figure 14: Hypothesis summarization and deduplication prompt (Step 3). Merges overlapping hypotheses from multiple clusters into a compact, non-redundant set.

Panel of Figure 15 — Figure 15: Ungated validation prompt (Step 4, Stage 1). The discriminator LLM judges whether a single action is consistent with a candidate behavioral statement without any gate conditioning.

Panel of Figure 16 — Figure 16: Gate evaluation prompt (Step 4, Stage 2). Determines whether a scene satisfies a candidate gate question. Used during both construction and inference-time tree traversal.

Panel of Figure 17 — Figure 17: Multi-candidate selection prompt (Step 5). Each of $R_{\mathrm{sel}}$ voting rounds presents the candidates in a random order; the candidate with the most votes is selected.

Panel of Figure 18 — Figure 18: Batch NLI classification prompt (Phase 1). Classifies the relationship between one action and all statements at a node in a single call, enabling efficient incremental update of the relation matrix $\mathbf{M}_{n}$ .

Panel of Figure 19 — Figure 19: Gate hypothesis generation prompt (Phase 2, demotion). When a statement’s precision falls below $\tau_{\mathrm{keep}}$ , this prompt generates candidate gate questions to separate supporting from contradicting observations.

Panel of Figure 20 — Figure 20: Gate–statement semantic compatibility check (Phase 2, demotion). Verifies that a candidate gate question is semantically coherent with the statement being demoted to a child subtree.

Panel of Figure 21 — Figure 21: New statement generation prompt (Phase 3). Proposes behavioral statements for observations not covered by any existing statement at the current node, conditioned on the gate path from root.

Panel of Figure 22 — Figure 22: CDT inference prompt.

Panel of Figure 23 — Figure 23: NLI-based evaluation prompt for consistency-level evaluation.

Panel of Figure 24 — Figure 24: Evaluation prompt for initiative dimension.

Panel of Figure 25 — Figure 25: Evaluation prompt for scope dimension.

Panel of Figure 26 — Figure 26: Evaluation prompt for magnitude dimension.

Panel of Figure 27 — Figure 27: Evaluation prompt for horizon dimension.

Panel of Figure 28 — Figure 28: Vanilla inference prompt (no background knowledge).

Panel of Figure 29 — Figure 29: Human-profile inference prompt.

Panel of Figure 30 — Figure 30: Summarization-based profile extraction prompt.

Panel of Figure 31 — Figure 31: Summarization-based profile aggregation prompt.

Panel of Figure 32 — Figure 32: RAG inference prompt.

Human Behavior Simulation

Human behavior simulation models how individuals and groups decide, interact, move, and adapt in social, economic, educational, and physical settings.

See all 239 papers

Learning to Simulate Individuals from Macro Social Signals
Oct 5, 2026Yining Zhao, Bushi Liu, Haofei Yu +5Human Behavior Simulation RL for Language Model Reasoning
University of Illinois Urbana-Champaign · Independent Researcher
Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulation
Apr 16, 2026Ziyang Chen, Renbing Chen, Daowei Li +4Human Behavior Simulation User Simulation
LongCat Interaction Team, Meituan Inc. Beijing, China · Independent Researcher Changsha, China
BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces
Jun 1, 2026Liangwei Yang, Jielin Qiu, Zixiang Chen +9Forecasting Benchmarks User Modeling
Salesforce AI Research

Oct 5, 2026J/K move · Enter open · S save

Learning to Simulate Individuals from Macro Social Signals

Yining Zhao, Bushi Liu, Haofei Yu, Zhengyang Qi+4

University of Illinois Urbana-Champaign · Independent Researcher

Human Behavior Simulation RL for Language Model Reasoning Belief Updating in LLMs Large Language Model-Based Social Simulation Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Large language models are increasingly used to simulate how individuals respond to new situations, yet the behavioral reasoning behind these responses is either inherited from pretraining or learned from individual-level annotations, which offer limited behavioral diversity and little supervision of the reasoning itself. We propose to learn behavioral reasoning from prediction markets, whose price trajectories record how populations respond to real-world events at scale. We introduce macro2mind, which trains a language model with GRPO using market signals. A social behavioral decomposition makes behavioral reasoning an explicit step of forecasting: the model infers representative groups of market participants, predicts how each interprets the news and updates its beliefs, reasons about their interactions, and aggregates these responses into a price. A hindsight-regret curriculum with difficulty-aware sampling focuses training on transitions where hindsight-identified groups substantially improve the forecast while prioritizing examples that remain learnable for the current policy. The learned reasoning applies to user simulation without further training. On SWM-Bench, macro2mind achieves state-of-the-art directional accuracy and correlation on Polymarket. Trained on market data, it transfers zero-shot to four user-simulation benchmarks (Humanual, OvertonBench, PRISM, and CAD) and has competitive performance among zero-shot methods. Used as a data generator, macro2mind also raises a downstream simulator's accuracy on unseen users by 15.5 points, outperforming data generated by its backbone by 13.2 points.

Abstract

Human Behavior Simulation

Explore similar work

Learning to Simulate Individuals from Macro Social Signals

Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulation

BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces