In oral arguments, judges probe attorneys with questions about the factual record, legal claims, and the strength of their arguments. To prepare for this questioning, both law schools and practicing attorneys rely on moot courts: practice simulations of appellate hearings. Leveraging a dataset of U.S. Supreme Court oral argument transcripts, we examine whether AI models can effectively simulate justice-specific questioning for moot court-style training. Evaluating oral argument simulation is challenging because there is no single correct question for any given turn. Instead, effective questioning should reflect a combination of desirable qualities, such as anticipating substantive legal issues, detecting logical weaknesses, and maintaining an appropriately adversarial tone. We introduce a two-layer evaluation framework that assesses both the realism and pedagogical usefulness of simulated questions using complementary proxy metrics. We construct and evaluate both prompt-based and agentic oral argument simulators. We find that simulated questions are often perceived as realistic by human annotators and achieve high recall of ground truth substantive legal issues. However, models still face substantial shortcomings, including low diversity in question types and sycophancy. Importantly, these shortcomings would remain undetected under naive evaluation approaches.
Figures & tables
Figure 1. Oral Argument Simulation Pipeline. A single task sample takes a) the facts of the case, b) the legal question, c) the context of the last n−1 turns in the oral argument, and d) name of the justice j that speaks next in the conversation. The simulator predicts a given justice’s nth turn. We implement two types of moot-court simulators: (1) Prompt-based — we apply 3 prompt variants on both open and closed base models, (2) Agentic — give larger reasoning models (Gemini and GPT variants) access to tools including closed search over case docket files and historical voting trends of justices. Finally, we evaluate each of our oral argument simulators using a two-layered evaluation framework, which assesses quality based on realism of simulation and pedagogical usefulness.
Figure 2. Overview of our Evaluation Framework. We evaluate oral argument simulators using two complementary layers. Realism assesses baseline plausibility through a) adversarial tests to check whether simulated justices respond appropriately to overtly provocative advocate behaviors, and b) human preference judgments. Pedagogical usefulness evaluates whether the simulator exhibits properties important for its use in moot court-style training settings, including a) coverage of substantive legal issues, (b) diversity of question types, c) detection of logical fallacies, and d) an appropriately adversarial tone of questioning.
XX First Place
XX Second Place
XX Third Place
Table 1. Model rankings across all metrics in our evaluation framework. Ranks on adversarial tests for realism are calculated based on average performance across the three test types: Decorum , Switching-Sides , and Rage-Bait . Human evaluation ranks are computed from Win-Rate on preference judgments. Legal issue coverage ranks are based on the Issue-Broad metric. Ranks for diversity of question types are calculated based on average Jensen-Shannon Divergence across the three classification schemes: Metacog , Legalbench , and Stetson . Fallacy detection ranks are calculated as average performance across the ten logical fallacy types. Overall ranks are computed as the average rank across all metrics. Gemini-based simulators perform best overall, though significant variation can be seen across different metrics. Despite being a smaller model, Llama-3.3-70B does remarkably well on most metrics.
Figure 3. The models we test push back against egregious decorum violations less than 40% of the time on the set of direct decorum violations and less than 10% of the time on the politicized rage bait and complete concession sets.
Model
Wins
Losses
Ties (eff.)
Win Rate (weighted) ↑
Bad Rate (%) ↓
Gemini-2.5-Pro (AGENT)
72
31
25
55.6
15.8
Llama-3.3-70B-Instruct (PROMPT)
66
37
34
54.6
9.9
GPT-4o (PROMPT)
62
46
31
51.0
8.6
Gemini-2.5-Pro (PROMPT)
62
41
26
49.3
15.1
Ground Truth
46
55
33
41.1
11.8
GPT-4o (AGENT)
45
52
32
40.1
15.1
Table 2. Results from human preference ratings. Models above the dashed line achieve higher win rates than ground truth justice turns in our evaluation set. The weighted win rate is computed as (wins+0.5×ties)÷total_matches . The bad rate is the proportion of matches in which both the model’s response and the comparison response were judged to be bad. See Appendix Table 3 for additional details.
Figure 4. How well different models address issues using our broad topical comparison, aggregated across all 30 sampled sections.
Figure 5. Distribution of generated turns across the LEGALBENCH, METACOG and STETSON categories for both prompt-based and agentic simulators, with the ground truth distribution highlighted in green. For all three classifications schemes, models are less diverse than the ground truth turns (as indicated by the relatively flatter ground truth distribution).
Figure 6. The amount of logical flaws caught by different models with the MOOT_COURT and agentic simulators (see Appendix B.5 for distributions across simulators). Gemini-2.5-Pro variant appears to perform the best across models, though all models struggle with sampling and number fallacies. The percentage of fallacies caught overlaps for certain fallacies, in which case we only display one model.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. View 1/4 of annotation interface used to collect human preference judgments for realism. Landing page of the interface with annotation instructions. Annotator picks their annotator ID and a specific conversation context to annotate. When they click on "Start New Context", they are taken to the next page shown in Figure 8 .
Figure 8. View 2/4 of annotation interface used to collect human preference judgments for realism. Based on the context selected by the annotator in the previous view, the facts, legal question and conversation history so far is populated. The conversation history contains at least the opening statement of the advocate, but is scrollable element. Figure 9 shows the same context but scrolled down to the last advocate turn.
Figure 9. View 3/4 of annotation interface used to collect human preference judgments for realism. Based on the context selected by the annotator on the landing page in Figure 7 , the facts, legal question and conversation history so far is populated. The conversation history is scrollable element, this view shows the conversation scrolled down till the last advocate turn. Once the annotator has read through the conversation, they begin their annotations of potential justice responses in response to the conversation so far, shown in Figure 10 .
Figure 10. View 4/4 of annotation interface used to collect human preference judgments for realism. Once the annotator has read the facts, legal question and conversation history so far (shown in Figures 8 and 9 ), they begin annotating pairs of possible justice responses for a given justice in response to the conversation so far. They can either vote in favor of one of the two responses, mark the pair as a ’tie’ or mark both as ’bad’. They can also fill out an optional open feedback box. Once they submit the annotation for one pair, they are shown the next pair of responses for this context. Once they are done annotating all 36 possible pairs of responses for this context, they are redirected to the landing page shown in Figure 7 to select the next context to annotate.
Model
Wins
Losses
Ties (raw)
Disagree
Ties (eff.)
Bads
Win Rate (weighted)
Win Rate (strict)
Bad Rate (%)
Rank (weighted)
Rank (strict)
Rank (bad rate)
gemini-2.5-pro_AGENT
72
31
18
7
25
24
55.592
47.368
15.789
1
1
6
Llama-3.3-70B-Instruct_SCOTUS_DEFAULT
66
37
19
15
34
15
54.605
43.421
9.868
2
2
2
gpt4o_SCOTUS_DEFAULT
62
46
23
8
31
13
50.987
40.789
8.553
3
3.5
1
gemini-2.5-pro_SCOTUS_DEFAULT
62
41
13
13
26
23
49.342
40.789
15.132
4
3.5
4.5
actual_text
46
55
11
22
33
18
41.118
30.263
11.842
5
5
3
gpt4o_AGENT
45
52
28
4
32
23
40.132
29.605
15.132
6
6
4.5
Appendix
Table 3. Human evaluation counts, win rates, bad rates, and corresponding rankings. Note bad rate rankings are computed in ascending order (so model with lowest bad rate is of rank 1.
Figure 11. To assess whether our simulated questions topically cover the set of legal issues raised by SCOTUS justices, we extract issues from a subsample of sections from the actual transcripts and check whether or not issues raised are covered by AI generations in our simulation.
Figure 12. Issue coverage (broad) across all 30 tested transcript sections – only the best performing model is plotted. Gemini-2.5-Pro, Qwen3-32B, and gpt-oss-120b are liberally represented.
Figure 13. How well different models address issues using our specific topical comparison, aggregated across all 30 sampled sections. .
Figure 14. We extract questions from all transcripts from 2021-2023 and filter out all incoherent questions. Then, we perform purpose labeling using Llama 70B. We subsequently perform two rounds of clustering to arrive at the final set of 7 MetaCog question purpose classes.
MetaCog Question Purpose
DESCRIPTION
statutory_interpretation
Related to the interpretation and application of statutes
precedent_and_doctrine
Related to the examination and application of precedents and doctrines
case_facts_and_context
Related to the examination of case facts and context
judicial_role_and_review
Related to the examination of the judicial role and review
argumentation_and_clarification
Related to the examination of argumentation and clarification
constitutional_issues
Related to the examination of constitutional issues
Appendix
Table 4. Final set of question purpose classes identified through the Metacog purpose discovery and clustering process outlined in Figure 14 .
Stetson Question Purpose
DESCRIPTION
elicit_information
A judge may ask about the facts in the evidence or record, or about authorities, parties, or background.
authority_applicability_legal_reach
A judge may ask about how precedent applies to a client’s facts or how far legal principles extend. A judge might also ask how authorities are similar to or distinguishable from the case at hand.
hypothetical
A judge may ask hypothetical questions to see just how far a legal position will reasonably reach. These can be phrased as examples.
opposing_counsel_args
A judge may ask questions about the opposing counsel’s argument. These questions will be designed to probe the weaknesses of the current advocate’s argument.
policy
A judge may ask policy questions to know how their decisions will impact society.
seek_concessions
A judge may ask the current advocate to concede a point because judges like to narrow the legal and factual questions in the case and to determine just how far the consequences of a decision will reach.
Appendix
Table 5. Question type classification scheme used for the Stetson metric. We add “unclear” and “other” for more comprehensive coverage.
Figure 15. Distribution of generated turns across LEGALBENCH categories for each model variant, with the ground truth distribution highlighted in green. Models lean heavily towards Criticism, Implications, and sometimes clarification, as opposed to the actual text which is a lot more widely distributed.
Figure 16. Distribution of generated turns across STETSON categories for each model variant, with the ground truth distribution highlighted in green. In general, models are much less diverse than the ground truth turns. The two agentic simulators (purple) exhibit a slightly increased diversity compared to the other variants.
Figure 17. Distribution of generated turns across METACOG categories for each model variant, with the ground truth distribution highlighted in green. Models are much less diverse than actual generations, almost always leaning towards the ’statutory_interpretation’ category.
Model
Legalbench
Stetson
Metacog
Llama-3.3-70B-Instruct_SCOTUS_DEFAULT
0.122
0.134
0.080
Qwen3-32B_SCOTUS_DEFAULT
0.131
0.153
0.149
gemini-2.5-pro_AGENT
0.072
0.095
0.059
gemini-2.5-pro_SCOTUS_DEFAULT
0.087
0.109
0.072
gpt-oss-120b_AGENT
0.036
0.189
0.129
gpt-oss-120b_SCOTUS_DEFAULT
0.061
0.171
0.180
Appendix
Table 6. Jensen Shannon Divergence for each of the Question-Type Diversity classification schemes. Divergence is computed between the distribution of question types of simulated turns versus the distribution of ground truth transcript turns.
Figure 18. The amount of logical flaws caught by different models across simulators. The Moot Court simulator appears to have the best performance, likely because it is instructed to be the most discerning of logical fallacies. The percentage of fallacies caught overlaps for certain fallacies, in which case we only display one model.
Figure 19. Ablations across agentic simulators for adversarial fallacy detection tests.
Figure 20. Distribution of Valence across all generated turns, with the ground truth distribution (i.e. turns in actual transcripts) highlighted in green. Moot court prompting (purple highlight) makes all models more competitive.
Figure 21. Average Per-turn Accuracy of Valence Classification for each model variant. Most models exhibit 70% accuracy in valence, with MOOT_COURT prompting doing slightly better than the rest.
Figure 22. Valence distributions by justice for selected simulators (Gemini-2.5-Pro AGENT and GPT-4o AGENT) compared to the distribution observed in the actual transcript data. Each bar shows the proportion of turns classified into valence categories for a given justice. GPT-4o exhibits inter-judge variation in valence that more closely mirrors patterns observed in the transcript data (e.g., higher neutrality for the Chief Justice). At the same time, Gemini-2.5-Pro manages to generate highly competitive questions, similar to the transcript data, while GPT-4o produces a larger proportion of "Slightly Cooperative" and "Cooperative" questions.
Figure 23. We run human-model accuracy on a small manually annotated set with 4 potential judge models. Qwen3-32B performs the best across our categories.
Large language model-based role-play simulation creates interactive agents that portray people, characters, or users for training, testing, and research.
Most existing dialogue systems are user-driven, primarily designed to fulfill user requests. However, in many critical real-world scenarios, a conversational agent must proactively extract information to achieve its own objectives rather than merely respond. To address this gap, we introduce Inquisitive Conversational Agents (ICAs) and develop an ICA specifically tailored to U.S. Supreme Court oral arguments. We propose a Dual Hierarchical Reinforcement Learning framework featuring two cooperating RL agents, each with its own policy, to coordinate strategic dialogue management and fine-grained utterance generation. By learning when and how to ask probing questions, the agent emulates judicial questioning patterns and systematically uncovers crucial information to fulfill its legal objectives. Evaluations on a U.S. Supreme Court dataset show that our method outperforms various baselines across multiple metrics. It represents an important first step toward broader high-stakes, domain-specific applications.
Xubo Lin, Zezhi Deng, Shihao Wang +2
Georgetown University · Singapore Management University
Deposition training requires attorneys to manage dynamic witness behavior, yet legal-AI evaluations largely focus on factual accuracy, reasoning, or response-level plausibility. We introduce WitnessSim, a deposition simulator driven by controllable legal personas. We use an evaluation framework separating behavioral realism from pedagogical usefulness. We assess realism through adversarial testing, blinded attorney comparison, and analysis of longitudinal behavioral trajectories. WitnessSim generally maintained plausible behavioral boundaries, and attorneys did not systematically prefer either original testimony or WitnessSim generated testimony. Pedagogical tests showed that witness behavior changed meaningfully in response to question form and attorney intervention without uniformly collapsing the assigned persona. Together, these results showcase a model of behavioral fidelity in legal simulations, and provide a framework for evaluating its performance.
This work explores the role of prompt design and judge selection in LLM-as-a-Judge evaluations of free text legal question answering. We examine whether automatic task prompt optimization improves over human-centered design, whether optimization effectiveness varies by judge feedback style, and whether optimized prompts transfer across judges. We systematically address these questions on the LEXam benchmark by optimizing task prompts using the ProTeGi method with feedback from two judges (Qwen3-32B, DeepSeek-V3) across four task models, and then testing cross-judge transfer. Automatic optimization consistently outperforms the baseline, with lenient judge feedback yielding higher and more consistent gains than strict judge feedback. Prompts optimized with lenient feedback transfer better to strict judges than the reverse direction. Analysis reveals that lenient judges provide permissive feedback, yielding prompts with broader applicability, whereas strict judges produce restrictive feedback, leading to judge-specific overfitting. Our findings demonstrate algorithmically optimizing prompts on training data can outperform human-centered prompt design and that judges' dispositions during optimization shape prompt generalizability.
Mohamed Hesham Elganayni, Runsheng Chen, Sebastian Nagl +1