RCBSF: A Multi-Agent Framework for Automated Contract Revision via Stackelberg Game
Authors: Shijia Xu, Yu Wang, Xiaolong Jia, Zhou Wu, Kai Liu, April Xiaowen Dong
Organizations: Chongqing University, China · Queen Mary University of London, UK · Chongqing Key Laboratory of Big Data Intelligence and Privacy Computing, China · Fangda Partners, China
Despite the widespread adoption of Large Language Models (LLMs) in Legal AI, their utility for automated contract revision remains impeded by hallucinated safety and a lack of rigorous behavioral constraints. To address these limitations, we propose the Risk-Constrained Bilevel Stackelberg Framework (RCBSF), which formulates revision as a non-cooperative Stackelberg game. RCBSF establishes a hierarchical Leader Follower structure where a Global Prescriptive Agent (GPA) imposes risk budgets upon a follower system constituted by a Constrained Revision Agent (CRA) and a Local Verification Agent (LVA) to iteratively optimize output. We provide theoretical guarantees that this bilevel formulation converges to an equilibrium yielding strictly superior utility over unguided configurations. Empirical validation on a unified benchmark demonstrates that RCBSF achieves state-of-the-art performance, surpassing iterative baselines with an average Risk Resolution Rate (RRR) of 84.21% while enhancing token efficiency. Our code is available at https://github.com/xjiacs/RCBSF .
Figures & tables
Figure 1: Comparison of legal contract generation workflows between the Baseline (Standard LLM) and the Risk-Constrained Bilevel Stackelberg Framework (RCBSF).
Figure 2: Illustration of the comparison between traditional contract revision paradigms and our proposed framework. (a) Baseline Methods rely on static template assembly or simple unidirectional generation based on user requirements. These approaches are typically static and prone to errors, often ignoring hidden risks due to the lack of iterative feedback mechanisms; (b) In contrast, our proposed Risk-Constrained Bilevel Stackelberg Framework (RCBSF) models the revision process as a hierarchical bilevel optimization game. It employs a Global Prescriptive Agent (GPA) as the leader to construct strict risk-budget objectives, driving a follower system consisting of a Revision Agent and an Auditor Agent. Through a Multi-Round Revision-Audit Interaction, these agents iteratively converge towards a solution that dynamically minimizes residual risk and consumption costs while preserving semantic quality.
Model
Method
PrivacyQA
ContractNLI
MAUD
CUAD
ALL (Avg)
CQ
RRR
TES
CQ
RRR
TES
CQ
RRR
TES
CQ
RRR
TES
CQ
RRR
TES
Qwen2.5-7B-Base
Standard
71.45
67.34
72.89
70.12
65.45
70.56
69.88
72.34
71.45
70.23
65.67
70.12
70.42
67.70
71.26
CoT
73.89
69.56
74.34
72.45
67.89
72.67
72.12
74.56
73.89
72.56
68.34
72.89
72.76
70.09
73.45
RAG
77.23
73.89
80.56
77.12
72.45
78.34
76.56
78.45
78.12
76.45
71.23
77.56
76.84
74.01
78.65
Iteration
78.56
75.45
81.78
75.89
74.56
80.23
77.45
80.89
79.56
75.23
73.45
79.12
76.78
76.09
80.17
RCBSF
84.67
79.45
85.67
83.89
77.92
83.56
83.45
87.12
84.23
82.67
78.34
83.45
83.67
80.71
84.23
Table 1: Main Results on Legal Contract Revision. We compare different LLM backbones across four datasets. CQ denotes Contract Quality as the average score scaled to 0-100 that evaluates clarity, rigor, balance, and professionalism. RRR indicates Risk Resolution Rate (%). TES represents Token Efficiency Score (%). The dark blue and light blue cells indicate the best and second-best performance within each model group, respectively.
Figure 3: Fine-grained Quality Metrics Breakdown. We present a diagonal block heatmap illustrating the performance of the Qwen2.5-7B-Chat model across four fine-grained quality metrics (i.e., Clarity, Rigor, Balance, and Professionalism) for each individual dataset.
Model Variant
RRR (%)
Δ
TES (%)
Δ
BLEU-4
Win Rate
Full RCBSF (Ours)
84.21
–
87.29
–
85.65
–
w/o 5-Dim Constraints
73.15
-11.06
76.25
-11.04
82.15
15.2%
w/o Budget Penalty ( λ=0 )
84.85
+0.64
75.15
-12.14
83.45
35.8%
w/o Iterative ( K=1 )
76.45
-7.76
90.45
+3.16
84.25
28.4%
w/o Q-Score Weighting
79.35
-4.86
81.15
-6.14
84.85
41.5%
Table 2: Ablation Study on the Qwen2.5-7B-Chat Backbone. We investigate the contribution of each component in the RCBSF framework. RRR: Risk Resolution Rate (%). TES: Token Efficiency Score (Risks tokens, %). Win Rate: Pairwise human preference against the Full model. The Δ columns show the relative drop compared to the full model.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Num of Samples
Num of Categories
PrivacyQA
7
2
ContractNLI
92
7
MAUD
150
7
CUAD
462
25
Total
711
41
Appendix
Table 3: Statistics of the processed Unified Legal Benchmark. The dataset aggregates 711 samples from diverse legal sources, covering 41 distinct categories to ensure broad domain coverage.
Figure 5: Distribution of Contract Categories. The dataset features a long-tail distribution across 41 legal domains, with significant representation in major categories like Merger and Disclosure.
Figure 6: The automated classification logic used in Stage 0 to standardize noisy raw file headers into canonical legal categories.
Figure 7: The instruction set for LLM used to generate standardized, anonymized legal templates from raw source text.
Figure 8: The adversarial prompt for LLM, designed to generate the standard truth risk dataset used for calculating the Risk Resolution Rate (RRR).
Figure 9: The architectural overview of the Risk-Constrained Bilevel Stackelberg Framework (RCBSF). The system is organized into two strategic levels: (1) The Outer Stackelberg Leader Loop , where the Leader Agent defines the strategic environment by mapping risk category weights to prompt guidance and optimizing the global utility functional. (2) The Inner Follower Optimization Loop, where the Revision Agent and Audit Agent engage in a multi-round adversarial game. The cycle continues until the generation stabilizes, ensuring the final contract x∗ satisfies the Leader’s constraints via the minimization objective.
Figure 10: Effectiveness vs. Efficiency ( K ). The left axis (blue) shows the Risk Resolution Rate, while the right axis (red) shows the normalized Token Cost. K=3 represents the optimal trade-off point.
Figure 11: Impact of Temperature ( τ ). The curve demonstrates that a balanced temperature ( τ=1.0 ) significantly outperforms both conservative ( τ=0.5 ) and high-entropy ( τ=2.0 ) settings.
Figure 12: The strict verification prompt used to calculate RRR. It requires evidence of substantive changes rather than mere stylistic edits.
Dimension
High Quality (80–100) 90–100: Excellent; 80–89: Good
Precise & Unambiguous. Key terms consistent; rights/obligations expressed in enforceable terms. Defect: Minor ambiguity not impairing execution.
Workable but Vague. Main obligations understandable; reliance on vague terms (e.g., “reasonable”) without standards; occasional inconsistency. Defect: Requires supplementary interpretation.
Loose Structure. Usable but contains logical gaps; mixed organization; frequent incorrect references. Defect: Logical jumps exploitable in disputes.
Disorganized. “Pile of clauses” with overlapping provisions; missing references; no systematic organization. Defect: Cannot support dispute resolution.
Balance
Symmetrical. Fair risk allocation; remedies apply equally; no significant unfair benefits. Defect: Minor non-core tilts.
Surface Symmetry. Appears balanced but hides practical tilts; force majeure or procedural clauses favor one party. Defect: Mostly fair core rights, but biased details.
One-Sided. Extreme asymmetry; unilateral termination/caps without reciprocity; effectively a single-party tool. Defect: Clear lack of reciprocal remedies.
Professionalism
Formal & Standard. Professional revision; standard formatting; consistent terminology; error-free. Defect: Occasional small typos.
Mixed Quality. Noticeable grammar/format issues; occasional colloquialisms or ambiguous everyday expressions. Defect: Reads like a draft or informal memo.
Table 4: Condensed scoring rubric for Contract Quality. The Judge Agent selects a band (High/Borderline/Deficient) based on the qualitative descriptors and fine-tunes the score within the specific 10-point ranges (e.g., 90–100 vs. 80–89).
Model
Method
PrivacyQA
ContractNLI
MAUD
CUAD
ALL (Avg)
Cla
Rig
Bal
Pro
Avg
Cla
Rig
Bal
Pro
Avg
Cla
Rig
Bal
Pro
Avg
Cla
Rig
Bal
Pro
Avg
Cla
Rig
Bal
Pro
Avg
Qwen2.5-7B-Base
Standard
73.12
68.45
70.33
73.89
71.45
72.11
66.89
69.23
72.25
70.12
71.34
67.56
69.12
71.51
69.88
71.67
67.45
69.34
72.45
70.23
72.06
67.59
69.51
72.53
70.42
CoT
75.34
71.12
72.45
76.65
73.89
74.23
69.56
71.34
74.67
72.45
73.56
69.89
71.23
73.79
72.12
74.12
69.78
71.56
74.77
72.56
74.31
70.09
71.65
74.97
72.76
RAG
78.56
74.23
76.12
79.99
77.23
78.34
74.12
76.56
79.45
77.12
77.89
74.12
75.89
78.33
76.56
78.12
73.89
75.45
78.34
76.45
78.23
74.09
76.01
79.03
76.84
Iteration
79.88
75.67
77.45
81.23
78.56
77.45
72.89
74.11
79.12
75.89
78.56
75.34
76.78
79.13
77.45
76.89
72.56
73.45
78.02
75.23
78.19
74.12
75.45
79.37
76.78
RCBSF
81.01
82.99
87.43
87.25
84.67
80.41
82.46
86.21
86.48
83.89
80.01
82.69
83.99
87.11
83.45
80.15
82.51
80.31
87.71
82.67
80.40
82.66
84.49
87.14
83.67
Appendix
Table 5: Fine-grained Quality Metrics Breakdown. We report Clarity ( Cla ), Rigor ( Rig ), Balance ( Bal ), and Professionalism ( Pro ) for each dataset and their average ( ALL ). The dark blue and light blue cells indicate the best and second-best performance within each model group.
Figure 13: Heatmap of Fine-Grained Quality Metrics. Multi-model-group variants’ performance across multi-tasks. Dark magenta is the best in group, dark green is the worst in group. Metric values range from 57–92.
Configuration
HR ( ↓ )
FFR ( ↓ )
RRR ( ↑ )
Full RCBSF
4.23%
7.35%
84.21%
w/o Evidence ( ek )
18.68%
11.24%
76.25%
w/o Suggestion ( sk )
5.19%
19.86%
75.14%
w/o Location ( lk )
12.47%
9.54%
78.19%
Appendix
Table 6: Component-wise Ablation. Removing specific dimensions from the Leader’s instruction leads to distinct failure modes.
Figure 14: A complete revision flow for a Liability Clause. The Leader’s specific suggestion (highlighted in yellow) serves as a hard constraint that the Follower eventually satisfies in the final output (green).
Figure 15: Detailed visualization of the multi-risk resolution in a commercial software context. The GPA successfully identifies three disparate risks (Fairness, IP, Financial). The Inner Loop demonstrates the system’s self-correction capability: while Iteration #1 fails to fully transfer IP rights, the LVA’s audit feedback forces the CRA to execute a legally binding assignment in Iteration #3, satisfying all Leader constraints simultaneously.
Figure 16: The structured prompt used by the Leader Agent to extract the 5-dimensional risk tuple. This structure forces the model to ground its suggestions in specific evidence.
Figure 17: The LVA Prompt. By enforcing a structured JSON output containing severity weights and gradient feedback, the Auditor transforms qualitative legal analysis into quantitative signals for the next optimization round.
Figure 18: The composite prompt mechanism for the CRA. It translates the abstract Stackelberg theoretical constraints into executable instructions, combining global strategy (Leader), resource budget (Fusion), and specific risk gradients (Auditor).
Figure 19: The Force Rewrite injection mechanism. This acts as a perturbation vector to push the CRA out of local optima (i.e., lazy revisions) — triggered when the system detects zero effective edits despite unresolved risks.
Figure 20: The definition of the Q-Score metrics. Each dimension is graded on a 3-point scale to compute the final risk weight.
Role-based LLM multi-agent systems need adaptive role pools, yet adapting such systems is not merely a matter of prompt optimization: roles often carry structural obligations, including capability coverage, message compatibility, validation, final-answer aggregation, and parser-compatible output protocols. Existing systems either fix the role inventory and lose adaptivity, or allow unconstrained generation to induce role drift, removing structurally necessary roles and breaking answer contracts. We formulate this as contract-preserving role evolution, requiring every committed edit to preserve five structural contracts (capability, communication, validation, aggregation, output protocol). We instantiate this formulation in SERO, a Self-Evolving Role Orchestration framework that evolves a typed role-card pool through credit-guided retrieval, a credit-ranked communication DAG with a protected terminal aggregator and conditional validator repair, and a contextual-bandit controller whose LLM-proposed edits are committed only when they preserve the contracts and improve task score. Experiments on real-world reasoning benchmarks across three LLM backbones confirm the value of contract-preserving role evolution.
Ling-Yue Ge, Lan-Zhe Guo
1National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China
Legal work, with its heavy reliance on processing large amounts of text, is often considered one of the domains most exposed to the use of LLMs. Contract ``scrubbing,'' the final review of transactional agreements for errors and inconsistencies, is a particularly suitable task for automation, because it is routine, painstaking work requiring detailed attention to long documents. Scrubbing also seems to align naturally with the general capabilities expected of frontier LLMs around long-context reasoning, consistency checking, and named entity recognition (NER). Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted. We introduce ContractScrub, the first benchmark designed to evaluate contract scrubbing capabilities, comprising contracts hand-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect references, and inconsistent language. Frontier models perform surprisingly poorly with only one model reaching 0.75 macro average recall despite strong performance on seemingly related general benchmarks, demonstrating the practical limits of current models and the importance of narrowly targeted, domain-specific benchmarks for measuring real-world impact.
Yejin Bang, Kirsty Fielding, Brandan Oliver +3
Thomson Reuters Foundational Research, London, UK · Imperial College London, UK
Legal reasoning requires not only high accuracy but also the ability to justify decisions through verifiable and contestable arguments. However, existing Large Language Model (LLM) approaches, such as Chain-of-Thought (CoT) and Retrieval-Augmented Generation (RAG), often produce unstructured explanations that lack a formal mechanism for verification or user intervention. To address this limitation, we propose Adaptive Collaboration of Argumentative LLMs (ACAL), a neuro-symbolic framework that integrates adaptive multi-agent collaboration with an Arena-based Quantitative Bipolar Argumentation Framework (A-QBAF). ACAL dynamically deploys expert agent teams to construct arguments, employs a clash resolution mechanism to adjudicate conflicting claims, and utilizes uncertainty-aware escalation for borderline cases. Crucially, our framework supports a Human-in-the-Loop (HITL) contestability workflow, enabling users to directly audit and modify the underlying reasoning graph to influence the final judgment. Empirical evaluations on the LegalBench benchmark demonstrate that ACAL outperforms strong baselines across Gemini-2.5-Flash-Lite and Gemini-2.5-Flash architectures, effectively balancing efficient predictive performance with structured transparency and contestability. Our implementation is available at: https://github.com/loc110504/ACAL.