Deposition training requires attorneys to manage dynamic witness behavior, yet legal-AI evaluations largely focus on factual accuracy, reasoning, or response-level plausibility. We introduce WitnessSim, a deposition simulator driven by controllable legal personas. We use an evaluation framework separating behavioral realism from pedagogical usefulness. We assess realism through adversarial testing, blinded attorney comparison, and analysis of longitudinal behavioral trajectories. WitnessSim generally maintained plausible behavioral boundaries, and attorneys did not systematically prefer either original testimony or WitnessSim generated testimony. Pedagogical tests showed that witness behavior changed meaningfully in response to question form and attorney intervention without uniformly collapsing the assigned persona. Together, these results showcase a model of behavioral fidelity in legal simulations, and provide a framework for evaluating its performance.
Figures & tables
Event
Condition
Witness rattled
pt>0.58∧(Ct<0.38∨ΔCt<−0.05)
Witness combative
At<0.28∧Rt>0.60
Attorney interrupts
pt>0.45∧∣qt∣≤8words∧type(qt)∈{closed, leading}
Witness talks over
Rt>0.72∧Vt>0.58∧pt>0.35
Personality shift
∥st−s0∥∞>0.35
Table 1: Threshold conditions on the state vector for discrete behavioral events. Triggered events are passed to the LLM system prompt to ground generated dialogue in the current mathematical state.
Test
Adversarial pressure
Primary result
AC1
Direct pressure on a Cooperative witness to accept personal blame
99.0% preserved a self-protective boundary while remaining in character
AC2
Escalating chain of increasingly accusatory leading premises
All witnesses resisted before accepting the complete chain; 72.7% resisted within the intended interval
AC3
Trivial versus sensitive questions posed to an Evasive witness
86.0% showed greater evasion on sensitive than trivial material
AC4
Five consecutive repetitions of the same question across three archetypes
Responses remained consistent, avoided mechanical duplication, and showed the expected archetype ordering in irritation onset
Table 2: Adversarial tests and primary outcomes.
Evaluator
n
Witness Sim
Original
Tie
Neither
Associate
50
15 (30.0%)
10 (20.0%)
20 (40.0%)
5 (10.0%)
Senior litigator
49
16 (32.7%)
14 (28.6%)
11 (22.4%)
8 (16.3%)
Senior Arbitration Counsel
50
0 (0.0%)
45 (90.0%)
5 (10.0%)
0 (0.0%)
Table 3: Blinded judgments of contextual plausibility.
Figure 1: Representative emotion-vector trajectories for a real deposition and its best-fitting simulated counterpart (Jeffrey Kilper, cooperative, r=.229 ).
Figure 2: PCA of flattened emotion-vector trajectories. Real transcripts are shown in gray and synthetic transcripts are colored by archetype.
Test
Training target and primary result
Question Form (Cooperative)
Adapting question form to control the scope and character of testimony. Open and clarifying questions elicited longer, generally more qualified responses, while closed, leading, and high-pressure questions produced substantially shorter answers.
Evasive Pin-Down (Evasive)
Forcing specificity from a witness who avoids commitment. The complete evasion-to-control trajectory occurred in 93.0% of contexts; 33.1% of substantive final answers still retained hedging or qualification.
Runaway Witness (Loquacious)
Redirecting lengthy or unfocused testimony. The intended unfocused-to-focused trajectory occurred in 59.3% of contexts, while 40.3% were already focused from the outset; only 0.3% never resolved to a focused answer.
Hostile Witness (Combative)
Obtaining usable testimony without eliminating the witness’s adversarial demeanor. Substantive control was reached in 97.7% of contexts, and 97.3% of controlled answers retained markers of hostility.
Table 4: Pedagogical evaluation tasks and headline results.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
Archetype
C
K
A
V
R
P
Combative
0.25
0.55
0.15
0.75
0.85
0.50
Cooperative
0.80
0.85
0.90
0.50
0.20
0.65
Defensive
0.55
0.60
0.30
0.30
0.75
0.60
Dogmatic
0.80
0.75
0.50
0.20
0.95
0.70
Inventive
0.70
0.20
0.60
0.85
0.25
0.60
Loquacious
0.60
0.70
0.60
0.95
0.25
0.55
Appendix
Table 5: Six-state attractor vectors for the ten retained witness archetypes. The dimensions are composure ( C ), knowledge ( K ), agreeableness ( A ), verbosity ( V ), rigidity ( R ), and performance ( P ).
Archetype
N
A
E
O
Cbf
Combative
0.75
0.15
0.75
0.15
0.50
Cooperative
0.20
0.90
0.50
0.80
0.65
Defensive
0.45
0.30
0.30
0.25
0.60
Dogmatic
0.20
0.50
0.20
0.05
0.70
Inventive
0.30
0.60
0.85
0.75
0.60
Loquacious
0.40
0.60
0.95
0.75
0.55
Appendix
Table 6: Approximate Big Five projection of the ten retained archetypes.
Pair
cosR6
θR6
cosR5
θR5
Δθ
Neutral / Overconfident
0.990
8.0 ∘
0.890
27.1 ∘
+19.1 ∘
Defensive / Dogmatic
0.989
8.4 ∘
0.900
25.8 ∘
+17.4 ∘
Nervous / Overconfident
0.868
29.7 ∘
0.703
45.4 ∘
+15.6 ∘
Combative / Dogmatic
0.847
32.1 ∘
0.678
47.3 ∘
+15.2 ∘
Combative / Overprepared
0.818
35.1 ∘
0.639
50.3 ∘
+15.2 ∘
Combative / Overconfident
0.871
29.4 ∘
0.714
44.5 ∘
+15.0 ∘
Appendix
Table 7: Pairwise archetype similarity in the six-state space ( R6 ) and approximate Big Five space ( R5 ). Δθ=θR5−θR6 ; negative values indicate compression under the Big Five projection.
Component
Contribution
Markers / interpretation
Baseline
0.10
Applied to every question before marker-specific increments.
High-pressure marker
+0.20 per match
isn’t it true ; you’re lying ; how do you explain ; did you not ; you knew ; admit ; deny .
Medium-pressure marker
+0.10 per match
why did you ; when did you ; who authorized .
Appendix
Table 8: Prespecified linguistic markers used to construct the pressure score.
State
Numerical update
Coefficient-level rationale
Composure ( C )
ΔCt=−0.25ξtCt+0.08(C0−Ct)
The −0.25 forcing term makes stress produce a comparatively strong immediate loss of composure. Multiplication by Ct means a highly composed witness has more composure to lose, while an already-rattled witness approaches a natural floor. The smaller 0.08 term produces slow recovery toward baseline rather than an immediate reset.
Knowledge ( K )
ΔKt=−0.30st(Kt−0.1)+0.06(1−pt)(K0−Kt)
The 0.30 sensitivity term makes topic-specific stress the strongest immediate state erosion in the system and preserves a floor at K=0.1 . The 0.06 recovery term is weaker and is multiplied by (1−pt) , so recall recovers primarily during low-pressure questioning and can ratchet down during sustained high-pressure examination.
Agreeableness ( A )
ΔAt=−0.15ptsgn(At−21)At−21+0.05(A0−At)
The 0.15 pressure term pulls agreeableness toward the guarded midpoint A=0.5 . Thus, a highly cooperative witness becomes less accommodating under pressure, while an already-hostile witness receives a small push toward neutral rather than becoming indefinitely more hostile. The 0.05 term gradually restores the archetype baseline.
Verbosity ( V )
ΔVt=0.20st(1−Vt)−0.20Ilead,tVt+0.05(V0−Vt)
The two immediate effects have equal magnitude but opposite direction. Sensitivity pushes verbosity upward through over-explanation, with (1−Vt) creating a ceiling effect. Leading yes/no form pushes verbosity downward, with Vt creating a floor effect. The 0.05 mean-reversion term restores baseline once those forces disappear.
Rigidity ( R )
ΔRt=0.12pt(1−Rt)−0.03(Rt−R0)
Pressure increases rigidity with a 0.12 forcing coefficient, while (1−Rt) prevents unbounded growth near R=1 . The 0.03 recovery coefficient is the smallest mean-reversion term among the six states, so entrenchment dissipates especially slowly after pressure is removed.
Performance ( P )
ΔPt=−0.18ptPtϕt+0.04(P0−Pt)
The 0.18 forcing term makes pressure degrade performance in proportion to both current polish and accumulated fatigue. A highly prepared witness therefore has more performance to lose. The smaller 0.04 recovery term allows only modest restoration toward the baseline.
Appendix
Table 9: Numerical state-update equations and behavioral motivation.
Turn
ϕt
Effect on pressure-driven performance loss
1
0.05
5% of the full fatigue-scaled rate
10
0.50
50% of the full fatigue-scaled rate
20 and later
1.00
Full fatigue-scaled rate
Appendix
Table 10: Illustrative values of the fatigue multiplier.
Score
Formula
Weight-level rationale
Consistency
0.5Kt+0.4Rt−0.1(1−Kt)(1−Rt)
Knowledge receives the largest weight ( 0.5 ) because factual recall is the primary basis of a coherent account. Rigidity receives 0.4 because consistency also requires maintaining the account across questioning. The 0.1 interaction penalty matters primarily when both knowledge and rigidity are low, corresponding to a witness who neither knows the story nor reliably adheres to it.
Evasion
0.5(1−Kt)+0.3∣2Vt−1∣+0.2Pt
Lack of knowledge receives the largest contribution ( 0.5 ). Deviation of verbosity from its midpoint receives 0.3 , allowing both extreme terseness and extreme verbosity to count as evasive behavior. Performance contributes 0.2 because a highly prepared witness can use polish strategically to avoid a direct answer.
Realism
max(0,1−0.3∣2Ct−1∣−0.3∣2Kt−1∣−2max(0,Pt−0.8))
The score begins at one. Extreme composure and extreme recall each carry a maximum penalty of 0.3 , reflecting the idea that either implausibly perfect control or complete breakdown can reduce behavioral realism. Performance is unpenalized through P=0.8 and is then penalized with slope 2 ; at P=1 , this contributes a maximum penalty of 0.4 . The outer maximum keeps the score nonnegative.
Adversarial
0.4(1−Kt)+0.3(1−At)+0.2Rt+0.1(1−Ct)
Knowledge gaps receive the largest weight ( 0.4 ), followed by non-cooperation ( 0.3 ), because unreliability and unwillingness to answer are the principal sources of examination difficulty. Rigidity contributes 0.2 , while composure loss contributes 0.1 as an additional source of unpredictability.
Appendix
Table 11: Numerical construction and interpretation of the derived behavioral scores.
Stage
Source
N
Notes
Master candidate pool
UCSF–JHU opioid transcript index
1,169
Manual export from the UCSF Industry Documents Library.
Initial random sample
Initial sample file
300
Random draw from the 1,169-document candidate archive.
Manual eligibility validation
Validated sample records
300
Sampled documents were reviewed, and non-deposition filings were replaced with additional randomly sampled eligible documents.
Final validated sample
Final document sample
300
Unique document IDs; repeated witness identities were permitted.
Pedagogical battery
Final pedagogical outputs
300
No final hard-failure records.
Adversarial battery
Final adversarial outputs
300
No final hard-failure or insufficient-content exclusions.
Appendix
Table 12: Corpus construction and evaluation flow.
Test
Field
k
n
Agreement
PABAK
Gwet AC1
U1
tone
3
50
48.4%
–
–
U1
pushback
3
50
54.0%
–
–
T1
residual-resistance flag (Turn 5)
2
48
89.6%
.792
.820
T1
substantive-control flag (Turn 5)
2
48
91.7%
.833
.898
T3
substantive-control flag (Turns 4–5)
2
47
95.7%
.915
.954
T3
residual-resistance flag (Turns 4–5)
2
47
93.6%
.872
.926
Appendix
Table 13: Human–LLM agreement for pedagogical evaluation fields. k denotes the number of response categories.
Table 14: Human–LLM agreement for adversarial evaluation fields. k denotes the number of response categories.
Archetype
Turn 2
Turn 3
Turn 4
Never
Combative
100.0%
0.0%
0.0%
0.0%
Evasive
88.3%
10.0%
1.7%
0.0%
Cooperative
71.3%
25.3%
2.7%
0.7%
Appendix
Table 15: Irritation onset under repeated questioning.
Evaluator
n
Original
WitnessSim
Tie
Neither
Associate
50
10 (20.0%)
15 (30.0%)
20 (40.0%)
5 (10.0%)
Senior Litigator
49
14 (28.6%)
16 (32.7%)
11 (22.4%)
8 (16.3%)
Senior Arbitration Counsel
50
45 (90.0%)
0 (0.0%)
5 (10.0%)
0 (0.0%)
Appendix
Table 16: Blinded evaluations of contextual plausibility. The associate and senior arbitration counsel reviewed the same 50-item Version 1 instrument; the senior litigator reviewed the subsequently tuned Version 2 instrument.
Evaluator
Preference decisions
Original
WitnessSim
Associate
25
10 (40.0%)
15 (60.0%)
Senior litigator
30
14 (46.7%)
16 (53.3%)
Appendix
Table 17: Preferences among practicing-attorney judgments in which one sequence was selected as more plausible.
Test
Archetype
N
Training target
U1
Cooperative
300
Adapting question form to control the scope and character of testimony.
T1
Evasive
300
Forcing specificity from a witness who initially avoids commitment.
T2
Loquacious
300
Regaining focus from lengthy or runaway testimony through progressive narrowing.
T3
Combative
300
Obtaining substantively controlled answers without eliminating hostile or resistant demeanor.
Appendix
Table 18: Pedagogical evaluation battery. Each test was evaluated across 300 case-grounded contexts.
Condition
Question template
Open
“Can you walk me through what you knew about {topic} {tenure phrase}?”
Closed
“You were aware of {topic}, correct?”
Leading
“Would you agree that {fact}?”
High-pressure
“Isn’t it true you knew that {fact}?”
Clarifying
“When you say you were ‘generally aware’ of {topic}—what specific information did you actually have access to?”
Appendix
Table 19: Question forms used in U1. Bracketed fields were populated from case-specific material extracted during case preparation.
Field
Scale
Description
direct_answer
0–2
Degree to which the response directly and substantively answers the question asked.
framing_acceptance
0–2 / null
Degree to which the witness accepts the framing proposed by the attorney; applicable only to the leading and high-pressure conditions.
hedge_count
count
Functional count of hedging or qualifying behavior.
pushback
0–2
Degree to which the witness challenges or resists the question or its premise.
tone
0–2
Rubric-based measure of cooperative versus resistant presentation.
Appendix
Table 20: Judge fields used in U1.
Question form
Mean words
Median
SD
Hedge rate
Open
175.9
176.5
40.8
96.3%
Clarifying
148.8
151.0
32.7
92.7%
Closed
86.5
85.0
28.0
81.3%
Leading
78.8
71.0
39.0
86.7%
High-pressure
68.4
63.0
33.2
82.3%
Appendix
Table 21: Response length and descriptive judge-derived hedge presence by question form.
Condition
Direct answer
Tone
Pushback
Framing acceptance
Open
10 / 101 / 189
0 / 56 / 244
229 / 55 / 16
–
Closed
3 / 52 / 245
1 / 133 / 166
190 / 96 / 14
–
Leading
5 / 87 / 208
0 / 185 / 115
154 / 123 / 23
28 / 130 / 142
High-pressure
7 / 129 / 164
1 / 256 / 43
108 / 158 / 34
59 / 153 / 88
Clarifying
2 / 41 / 257
0 / 66 / 234
199 / 93 / 8
–
Appendix
Table 22: U1 judge-field distributions. Entries give counts at rubric levels 0 / 1 / 2. Framing acceptance was scored only for the leading and high-pressure conditions.
Outcome
Estimate
95% CI
Protocol pass rate
93.0% (279/300)
[89.5, 95.4]
Challenge elicitation
99.3% (298/300)
[97.6, 99.8]
Conditional control given challenge
93.6% (279/298)
[90.3, 95.9]
Full intended trajectory
93.0% (279/300)
[89.5, 95.4]
Residual hedging among substantive final answers
33.1% (93/281)
[27.9, 38.8]
Appendix
Table 23: Evasive Pin-Down Test outcomes.
Field
Type
Function
turn1_to_turn4_trajectory
3-category
Classifies the overall interaction as yielding from unfocused to focused, remaining consistently focused, or never resolving.
stays_focused
binary
Indicates whether a response remains focused on the question asked.
economical_delivery
binary
Captures whether the witness avoids unnecessary expansion under narrowing.
compliance_with_narrowing
3-category
Measures whether the witness follows the attorney’s narrowing instruction.
turn4_binary
4-category
Classifies the response to the final explicit demand; the reported direct-answer endpoint is derived from this final-turn assessment.
Appendix
Table 24: Judge-derived fields used in the Runaway Witness Control Test.
Outcome
Estimate
95% CI
Protocol pass rate 1 1 1 Wilson score 95% confidence interval for a single proportion.
99.7% (299/300)
[98.1, 99.9]
Yields (unfocused → focused) 2 2 2 Goodman simultaneous 95% multinomial confidence interval for the three mutually exclusive trajectory categories ( k=3 , α=.05 ).
59.3% (178/300)
[52.4, 65.9]
Consistently focused 2 2 2 Goodman simultaneous 95% multinomial confidence interval for the three mutually exclusive trajectory categories ( k=3 , α=.05 ).
40.3% (121/300)
[33.8, 47.2]
Never resolves 2 2 2 Goodman simultaneous 95% multinomial confidence interval for the three mutually exclusive trajectory categories ( k=3 , α=.05 ).
0.3% (1/300)
[0.0, 2.5]
Direct answer under final explicit demand 1 1 1 Wilson score 95% confidence interval for a single proportion.
79.7% (239/300)
[74.8, 83.8]
Appendix
Table 25: Runaway Witness Control Test outcomes.
Outcome
Estimate
95% CI
Protocol pass rate
100.0% (300/300)
[98.7, 100]
Challenge elicitation
100.0% (300/300)
[98.7, 100]
Conditional substantive control
97.7% (293/300)
[95.3, 98.9]
Full intended trajectory
97.7% (293/300)
[95.3, 98.9]
Residual hostility among controlled answers
97.3% (285/293)
[94.7, 98.6]
Appendix
Table 26: Hostile Witness Control Test outcomes.
Figure 3: Null distribution for the trajectory-correlation analysis. Synthetic 20-bin trajectories were temporally permuted 1,000 times while real trajectories were held fixed. The observed mean correlation ( r=0.1477 ) exceeded all permuted values. The null mean was 0.0002 and its 95th percentile was 0.0605 .
Figure 4: Per-witness emotion-arc similarity matrix for Jeffrey Kilper. The Cooperative archetype provides the best-fitting synthetic trajectory ( r=.229 ).
Large language model-based role-play simulation creates interactive agents that portray people, characters, or users for training, testing, and research.
We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior. While language models are increasingly deployed for persona-driven user simulation, existing benchmarks primarily evaluate conversational styling or self-reports rather than authentic behavioral fidelity. APB evaluates latent persona adherence one trait at a time, embedding each target trait within a complete synthetic profile without explicitly naming the trait or disclosing the test. Ground-truth adherence is verified strictly from observable actions across four interaction surfaces of increasing realism: survey, chat, web (interactive web environments), and app (desktop software environments). APB comprises 2,460 tasks spanning 867 traits, verified through automated audits and expert review. Our evaluation of 20 frontier model arms demonstrates that high-fidelity user simulation is already attainable: leading models achieve up to 84.7% full-pass adherence under unprompted conditions. At the same time, APB identifies clear behavioral boundaries: adherence drops across interaction modalities (only 37.9-64.3% pass all four surfaces), multi-attribute demands degrade retention, and competing model families exhibit pronounced behavioral divergence.
Civil litigation is inherently a life-cycle process: what a lawyer drafts on day one constrains what unfolds at trial months later. Yet existing legal benchmarks evaluate isolated subtasks, and prior legal-agent simulators reinitialize each scenario from shared ground truth, leaving cross-stage causal dependencies unmodeled. We present LegalWorld, a life-cycle interactive environment that models Chinese civil litigation as a causally connected state chain of five stages (seven sub-scenarios), grounded in 75,309 paired Chinese civil judgments. We pair it with reusable infrastructure (local memory, global case memory, a Skill/Tool library) that keeps each dispute consistent across its full life cycle. Building on this environment, we construct LongJud-Bench to evaluate agent capability across all five connected stages. 18,992 ratings from 217 legal-background evaluators confirm that LegalWorld trajectories are procedurally faithful and role-consistent; and a capability-level cross-model evaluation reveals sharp divergences that aggregate scores cannot expose, with no single backbone leading across consultation, drafting, and courtroom advocacy. Detailed resources will be released publicly.
Songhan Zuo, Shengbin Yue, Tao Chiang +4
Fudan University · Shanghai Innovation Institute · Northwest University of Political and Law
LLM-based user simulation is the primary mechanism for end-to-end agent evaluation, yet simulated users are poor proxies for real humans: unconstrained LLM defaults produce a Formalism Ceiling (style match rates of 6-8% against real users), while hand-crafted behavioral directives trigger Directive Amplification, where models hyper-interpret instructions into unnatural behavioral extremes that vary dramatically across simulator models. We present RealUserSim, the first user simulation framework grounded in real behavioral data. From 14,000+ authentic human-LLM conversations (WildChat), we extract 7,275 executable behavioral profiles and use them to ground LLM simulators. A fidelity benchmark (PT3) on 600 conversations across 71+ domains with anti-leakage controls shows that grounded simulation raises match rate from 24.2% to 45.3% across five behavioral dimensions. Agent evaluation on TauBench with 6 simulator models and extensive analysis shows that grounded simulation acts as a realistic stress test, surfacing three failure mechanisms invisible to cooperative simulators (mean -3.2% to -3.5% task success degradation), while Directive Amplification in existing benchmarks produces unrealistic behavior that compromises the validity of agent evaluation.