Organizations: National Key Laboratory for Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · DiDichuxing Co. Ltd, Beijing, China
Liability adjudication tasks (e.g., legal charge determination) require analyzing event descriptions against domain-specific skills to determine whether a party should bear liability. While Large Language Models (LLMs) have shown promise in this area by leveraging executable skills for interpretable reasoning, existing approaches face three critical limitations: skill libraries become stale as real-world scenarios dynamically evolve, skill descriptions often suffer from a semantic execution gap where LLMs deviate from intended expert logic, and growing skill collections exceed context budgets causing critical skill omission. To address these challenges, we propose Knowledge-Data Dual-Driven Automated Skill Optimization (KDASO), which continuously evolves a skill library through neighborhood-conditioned induction from recent adjudication data, execution-aligned refinement based on LLM trajectory analysis, and context-budgeted routing for scalable skill selection. Experiments across ride-hailing adjudication, legal liability determination, and logical reasoning judgment demonstrate that KDASO significantly outperforms current general-purpose and domain-specific baselines. KDASO further achieves 8.96% improvement when transferring skills to concept-similar scenarios and maintains consistent performance gains through continuous optimization, underscoring the high transferability and stability of generated skills.
Figures & tables
Figure 1
Figure 3: The overall illustration of KDASO framework.
Model †
CA
CR
FD
O-Precision
O-Recall
B-Precision
B-Recall
O-Precision
O-Recall
B-Precision
B-Recall
O-Precision
O-Recall
B-Precision
B-Recall
NL2FOL
68.21% ± 0.54%
62.13% ± 1.87%
41.84% ± 0.32%
23.56% ± 0.45%
62.46% ± 0.73%
50.66% ± 0.68%
73.96% ± 0.91%
30.31% ± 0.29%
77.17% ± 0.64%
60.85% ± 0.77%
9.72% ± 0.19%
43.75% ± 0.58%
NeoBERT
57.98% ± 1.62%
11.02% ± 0.21%
52.94% ± 0.48%
36.21% ± 0.56%
63.95% ± 0.71%
24.13% ± 0.33%
56.30% ± 0.59%
35.66% ± 0.47%
71.13% ± 0.83%
9.61% ± 0.18%
14.43% ± 0.26%
17.50% ± 0.51%
ModernBERT
63.44% ± 0.75%
18.85% ± 0.28%
47.31% ± 0.48%
46.57% ± 0.69%
59.02% ± 0.66%
16.73% ± 0.04%
50.95% ± 0.52%
24.24% ± 0.35%
70.30% ± 0.79%
9.89% ± 0.20%
13.86% ± 0.23%
17.50% ± 1.31%
SIM-RAG
71.31% ± 0.88%
59.16% ± 0.74%
57.78% ± 0.65%
29.89% ± 0.41%
58.43% ± 0.92%
15.48% ± 0.42%
51.24% ± 0.45%
23.75% ± 0.34%
74.27% ± 0.85%
37.46% ± 0.51%
12.36% ± 0.21%
23.75% ± 0.34%
NodeRAG
72.24% ± 2.89%
61.70% ± 0.76%
56.82% ± 0.63%
28.74% ± 0.39%
61.66% ± 0.90%
31.21% ± 0.53%
50.02% ± 1.33%
24.63% ± 0.36%
74.91% ± 0.86%
45.23% ± 1.89%
14.22% ± 0.25%
25.49% ± 0.37%
Table 1: Score comparison of KDASO against liability adjudication general-purpose algorithms and open-source general LLMs across three tasks from ride-hailing adjudication. The best is in bold and the second best is underlined among non-proprietary models.
Table 4Figure 5
Model Variants
O-Precision
O-Recall
B-Precision
B-Recall
QwQ-32B
72.49% ± 0.89%
62.30% ± 0.97%
57.78% ± 0.65%
29.88% ± 0.41%
w/o Neighborhood-Conditioned Skill Induction
72.54% ± 0.90%
71.72% ± 0.86%
54.00% ± 0.71%
31.03% ± 0.43%
w/o Execution-Aligned Skill Refinement
72.36% ± 0.89%
71.09% ± 0.85%
55.56% ± 0.62%
28.73% ± 0.59%
w/o Context-Budgeted Skill Routing
69.83% ± 0.91%
88.02% ± 0.96%
63.39% ± 0.77%
40.80% ± 0.55%
KDASO QwQ-32B
73.42% ± 0.90%
87.38% ± 0.97%
69.17% ± 0.89%
47.70% ± 0.60%
Table 4: Ablation study of KDASO on CA.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Samples
Time Span
Length(tokens)
CA
1,148
2025/10/16 - 2025/10/22
1,001
CR
6,200
2025/10/01 - 2025/10/31
996
CPC
2,455
2025/10/09 - 2025/10/21
1,914
CDC
2,529
2025/10/09 - 2025/10/21
1,683
FD
1,358
2025/10/01 - 2025/10/27
300
Appendix
Table 5: Datasets in DiDi ride-hailing scenarios. All tasks are multi-class classification. Open-source datasets are shown in B . Length(tokens) denotes the total sequence length when the text obtained by converting all modalities into texts and concatenating them is fed into the locally deployed tokenizer, excluding skills.
Benchmark
Dataset
Samples
Length(tokens)
LawBench
LA-ITI
500
49.7
LA-CP
500
660.6
LegalBench
LE-OPP
4,271
43.2
LE-UN
3,813
44.6
LE-LH
4,992
325.3
Reclor
1,000
95.68
Appendix
Table 6: Law-specific and logic-specific datasets. These datasets only contain texts.
Month
2025.08
2025.09
2025.10
2025.11
2025.12
Number of Dnew
3000
3000
3000
3000
3000
Number of Generated Skills
12
15
8
6
6
Number of Merged Skills
4
4
3
1
2
Number of Dropped New Skills
5
6
5
4
3
Number of R′
78
83
83
84
85
Average Length of R′′
137.4
146.3
147.5
149.1
149.8
Appendix
Table 7: Details of Skill Changes from August 2025 to December 2025 on KDASO.
Model
Input tokens
Output tokens
Time Cost(seconds per inference)
QwQ-32B
27,156
1,540
14.79
KDASO QwQ-32B
7,241
1,769
12.13
Appendix
Table 8: Tokens vs. Runtime.
k
O-Precision
O-Recall
B-Precision
B-Recall
QwQ-32B
72.49%
62.30%
57.78%
29.88%
k = 1
65.52%
85.62%
49.17%
33.91%
k = 2
65.57%
86.10%
51.13%
39.08%
k = 3
66.62%
83.55%
40.80%
29.31%
k = 4
72.49%
88.82%
71.14%
60.92%
k = 5
64.88%
85.30%
45.80%
34.48%
Appendix
Table 9: Model performance under different k of KDASO QwQ-32B .
τ
O-Precision
O-Recall
B-Precision
B-Recall
QwQ-32B
72.49%
62.30%
57.78%
29.88%
τ = 0.7
63.26%
88.02%
48.73%
44.25%
τ = 0.8
62.16%
88.18%
50.44%
32.76%
τ = 0.9
71.67%
68.69%
58.03%
37.36%
Appendix
Table 10: Model performance under different τ of KDASO QwQ-32B
Hyperparameter
Values
Learning Rate
{0.01,0.1}
Max. Depth
{1,5,9}
N Estimators
{10,000,20,000,30,000}
Subsample
{0.5,0.8,1.0}
Colsample Bytree
{0.5,0.8,1.0}
Min Child Weight
{1,3,5}
Appendix
Table 11: Hyperparameter Grids of XGBoost.
Dnew Composition
O-Precision
O-Recall
B-Precision
B-Recall
FN only
67.19% ± 0.82%
81.47% ± 0.93%
47.62% ± 0.60%
40.23% ± 0.54%
FP only
71.70% ± 0.85%
72.04% ± 0.86%
44.90% ± 0.58%
25.29% ± 0.37%
FP+FN
69.07% ± 0.83%
76.36% ± 0.89%
43.64% ± 0.57%
27.59% ± 0.38%
Appendix
Table 12: Changes in metrics by changing Dnew to KDASO QwQ-32B . FP and FN are obtained by applying R and latest cases on f to get the confusion matrix.
Method
Precision
Recall
Training Time
XGBoost
67.37%
89.46%
345.56s
Dense Retrieval (Bi-Encoder)
36.10%
92.53%
0.69s
Cross-Encoder
47.10%
94.72%
16.34s
Appendix
Table 13: Comparison of different retrieval methods.
Skill Count
XGBoost (Parallel) [ms]
Dense Retrieval [ms]
Cross-Encoder (Est.) [ms]
100
7.21
5.1
500
36.03
5.5
1,000
72.07
6.0
> 1,000
2,000
144.14
7.0
5,000
360.34
10.0
Appendix
Table 14: Scalability stress test on different numbers of skills.
Backbone
Label Homogeneity ↑
Statute Hit Recall ↑
Induced Skill F1 ↑
TF-IDF
54.3%
48.7%
52.1%
Sentence-BERT
71.2%
65.4%
71.0%
BGE-Large-v1.5 (KDASO)
83.6%
79.2%
78.5%
Law-BERT
84.1%
79.3%
77.2%
Appendix
Table 15: Comparison of embedding backbones for semantic similarity validation.
Scenario
Baseline O-Precision
Baseline O-Recall
KDASO O-Precision
KDASO O-Recall
CA
91.68% ± 1.0%
81.66% ± 0.9%
87.98% ± 1.0%
78.54% ± 0.9%
CR
−
−
81.88% ± 0.9%
63.05% ± 0.8%
CDC
96.15% ± 0.3%
70.00% ± 2.1%
96.57% ± 0.1%
71.99% ± 1.2%
CPC
88.19% ± 1.2%
45.69% ± 2.1%
88.51% ± 0.9%
47.14% ± 1.8%
FD
77.17% ± 0.9%
60.85% ± 0.8%
79.07% ± 0.9%
66.29% ± 0.8%
Appendix
Table 16: Performance comparison between DiDi online baseline and KDASO across five ride-hailing scenarios. " − " means that it’s a new scenario for which no online baseline was previously available.
Ride-hailing Adjudication (CPC)
Generating Model
Reasoning Model
O-Precision
QwQ-32B
DiDi Baseline
87.39% ± 0.95%
DiDi Baseline
DiDi Baseline
85.77% ± 0.94%
DiDi Baseline
QwQ-32B
84.21% ± 0.93%
QwQ-32B
QwQ-32B
88.51% ± 0.90%
Legal Liability Determination (LE-LH)
Appendix
Table 17: O-Precision / Score comparison of KDASO configurations across general and domain-specific models in CPC and LE-LH tasks.
Metric
Qwen3-4B
Qwen3-8B
Qwen3-30B-A3B-Thinking
Qwen3-32B
QwQ-32B
O-Precision
59.00% ± 0.67%
67.44% ± 0.81%
67.87% ± 0.82%
68.68% ± 0.83%
73.25% ± 0.89%
O-Recall
85.30% ± 0.94%
65.81% ± 0.80%
60.70% ± 0.74%
78.11% ± 0.91%
78.75% ± 0.92%
Appendix
Table 18: KDASO average performance across different scales of general LLMs (Qwen series) on three DiDi datasets. All models use identical KDASO orchestration framework.
Generator
Params
QwQ-32B
DeepSeek-V4-Flash
GLM-5
Qwen3.5-27B
27B
78.2
79.5
79.2
QwQ-32B
32B
80.4
80.5
80.2
DeepSeek-V4-Flash
284B/13B MoE
81.0
82.0
81.8
MiniMax M2.5
∼ 400B MoE
79.8
80.8
80.5
Kimi K2.5
1T/32B MoE
78.5
81.2
80.9
Qwen3.5-397B
397B/17B MoE
80.5
81.5
81.2
Appendix
Table 19: Extended LE-LH overall score. Skills generated by different models (rows) are executed by different reasoners (columns).
Model
Params
Type
Base Score
+KDASO
Δ (Gain)
Qwen3-8B
8B
Open-Source General LLMs
33.6
39.6
+6.0
QwQ-32B
32B
Open-Source General LLMs
39.8
44.0
+4.2
Qwen3.5-2B
2B
Open-Source General LLMs
28.5
32.0
+3.5
Qwen3.5-27B
27B
Open-Source General LLMs
40.2
41.8
+1.6
MiniMax M2.5
∼ 400B MoE
Open-Source General LLMs
41.5
44.8
+3.3
Kimi K2.5
1T/32B MoE
Open-Source General LLMs
42.0
45.2
+3.2
Appendix
Table 20: Extended LA-ITI overall score with and without KDASO.
Model Configuration
Score
QwQ-32B
39.8
w/o Neighborhood-Conditioned Skill Induction
40.1
w/o Execution-Aligned Skill Refinement
42.7
w/o Context-Budgeted Skill Routing
43.5
KDASO QwQ-32B
44.0
KDASO QwQ-32B (LE-LH → LA-ITI)
43.6
Appendix
Table 21: Ablation study and cross-domain skill transferability results for KDASO on public legal/logical reasoning benchmarks. Scores reflect overall task performance. Higher values indicate better capability.