KDASO: Knowledge-Data Dual-Driven Automated Skill Optimization for Liability Adjudication Tasks
Organizations: National Key Laboratory for Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · DiDichuxing Co. Ltd, Beijing, China
Abstract
Liability adjudication tasks (e.g., legal charge determination) require analyzing event descriptions against domain-specific skills to determine whether a party should bear liability. While Large Language Models (LLMs) have shown promise in this area by leveraging executable skills for interpretable reasoning, existing approaches face three critical limitations: skill libraries become stale as real-world scenarios dynamically evolve, skill descriptions often suffer from a semantic execution gap where LLMs deviate from intended expert logic, and growing skill collections exceed context budgets causing critical skill omission. To address these challenges, we propose Knowledge-Data Dual-Driven Automated Skill Optimization (KDASO), which continuously evolves a skill library through neighborhood-conditioned induction from recent adjudication data, execution-aligned refinement based on LLM trajectory analysis, and context-budgeted routing for scalable skill selection. Experiments across ride-hailing adjudication, legal liability determination, and logical reasoning judgment demonstrate that KDASO significantly outperforms current general-purpose and domain-specific baselines. KDASO further achieves 8.96% improvement when transferring skills to concept-similar scenarios and maintains consistent performance gains through continuous optimization, underscoring the high transferability and stability of generated skills.
Figures & tables
| Model † | CA | CR | FD | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| O-Precision | O-Recall | B-Precision | B-Recall | O-Precision | O-Recall | B-Precision | B-Recall | O-Precision | O-Recall | B-Precision | B-Recall | |
| NL2FOL | 68.21% 0.54% | 62.13% 1.87% | 41.84% 0.32% | 23.56% 0.45% | 62.46% 0.73% | 50.66% 0.68% | 73.96% 0.91% | 30.31% 0.29% | 77.17% 0.64% | 60.85% 0.77% | 9.72% 0.19% | 43.75% 0.58% |
| NeoBERT | 57.98% 1.62% | 11.02% 0.21% | 52.94% 0.48% | 36.21% 0.56% | 63.95% 0.71% | 24.13% 0.33% | 56.30% 0.59% | 35.66% 0.47% | 71.13% 0.83% | 9.61% 0.18% | 14.43% 0.26% | 17.50% 0.51% |
| ModernBERT | 63.44% 0.75% | 18.85% 0.28% | 47.31% 0.48% | 46.57% 0.69% | 59.02% 0.66% | 16.73% 0.04% | 50.95% 0.52% | 24.24% 0.35% | 70.30% 0.79% | 9.89% 0.20% | 13.86% 0.23% | 17.50% 1.31% |
| SIM-RAG | 71.31% 0.88% | 59.16% 0.74% | 57.78% 0.65% | 29.89% 0.41% | 58.43% 0.92% | 15.48% 0.42% | 51.24% 0.45% | 23.75% 0.34% | 74.27% 0.85% | 37.46% 0.51% | 12.36% 0.21% | 23.75% 0.34% |
| NodeRAG | 72.24% 2.89% | 61.70% 0.76% | 56.82% 0.63% | 28.74% 0.39% | 61.66% 0.90% | 31.21% 0.53% | 50.02% 1.33% | 24.63% 0.36% | 74.91% 0.86% | 45.23% 1.89% | 14.22% 0.25% | 25.49% 0.37% |
| Model Variants | O-Precision | O-Recall | B-Precision | B-Recall |
|---|---|---|---|---|
| QwQ-32B | 72.49% 0.89% | 62.30% 0.97% | 57.78% 0.65% | 29.88% 0.41% |
| w/o Neighborhood-Conditioned Skill Induction | 72.54% 0.90% | 71.72% 0.86% | 54.00% 0.71% | 31.03% 0.43% |
| w/o Execution-Aligned Skill Refinement | 72.36% 0.89% | 71.09% 0.85% | 55.56% 0.62% | 28.73% 0.59% |
| w/o Context-Budgeted Skill Routing | 69.83% 0.91% | 88.02% 0.96% | 63.39% 0.77% | 40.80% 0.55% |
| KDASO QwQ-32B | 73.42% 0.90% | 87.38% 0.97% | 69.17% 0.89% | 47.70% 0.60% |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Samples | Time Span | Length(tokens) |
|---|---|---|---|
| CA | 1,148 | 2025/10/16 - 2025/10/22 | 1,001 |
| CR | 6,200 | 2025/10/01 - 2025/10/31 | 996 |
| CPC | 2,455 | 2025/10/09 - 2025/10/21 | 1,914 |
| CDC | 2,529 | 2025/10/09 - 2025/10/21 | 1,683 |
| FD | 1,358 | 2025/10/01 - 2025/10/27 | 300 |
| Benchmark | Dataset | Samples | Length(tokens) |
| LawBench | LA-ITI | 500 | 49.7 |
| LA-CP | 500 | 660.6 | |
| LegalBench | LE-OPP | 4,271 | 43.2 |
| LE-UN | 3,813 | 44.6 | |
| LE-LH | 4,992 | 325.3 | |
| Reclor | 1,000 | 95.68 |
| Month | 2025.08 | 2025.09 | 2025.10 | 2025.11 | 2025.12 |
| Number of | 3000 | 3000 | 3000 | 3000 | 3000 |
| Number of Generated Skills | 12 | 15 | 8 | 6 | 6 |
| Number of Merged Skills | 4 | 4 | 3 | 1 | 2 |
| Number of Dropped New Skills | 5 | 6 | 5 | 4 | 3 |
| Number of | 78 | 83 | 83 | 84 | 85 |
| Average Length of | 137.4 | 146.3 | 147.5 | 149.1 | 149.8 |
| Model | Input tokens | Output tokens | Time Cost(seconds per inference) |
|---|---|---|---|
| QwQ-32B | 27,156 | 1,540 | 14.79 |
| KDASO QwQ-32B | 7,241 | 1,769 | 12.13 |
| O-Precision | O-Recall | B-Precision | B-Recall | |
|---|---|---|---|---|
| QwQ-32B | 72.49% | 62.30% | 57.78% | 29.88% |
| = 1 | 65.52% | 85.62% | 49.17% | 33.91% |
| = 2 | 65.57% | 86.10% | 51.13% | 39.08% |
| = 3 | 66.62% | 83.55% | 40.80% | 29.31% |
| = 4 | 72.49% | 88.82% | 71.14% | 60.92% |
| = 5 | 64.88% | 85.30% | 45.80% | 34.48% |
| O-Precision | O-Recall | B-Precision | B-Recall | |
|---|---|---|---|---|
| QwQ-32B | 72.49% | 62.30% | 57.78% | 29.88% |
| = 0.7 | 63.26% | 88.02% | 48.73% | 44.25% |
| = 0.8 | 62.16% | 88.18% | 50.44% | 32.76% |
| = 0.9 | 71.67% | 68.69% | 58.03% | 37.36% |
| Hyperparameter | Values |
|---|---|
| Learning Rate | |
| Max. Depth | |
| N Estimators | |
| Subsample | |
| Colsample Bytree | |
| Min Child Weight |
| Composition | O-Precision | O-Recall | B-Precision | B-Recall |
|---|---|---|---|---|
| FN only | 67.19% 0.82% | 81.47% 0.93% | 47.62% 0.60% | 40.23% 0.54% |
| FP only | 71.70% 0.85% | 72.04% 0.86% | 44.90% 0.58% | 25.29% 0.37% |
| FP+FN | 69.07% 0.83% | 76.36% 0.89% | 43.64% 0.57% | 27.59% 0.38% |
| Method | Precision | Recall | Training Time |
|---|---|---|---|
| XGBoost | 67.37% | 89.46% | 345.56s |
| Dense Retrieval (Bi-Encoder) | 36.10% | 92.53% | 0.69s |
| Cross-Encoder | 47.10% | 94.72% | 16.34s |
| Skill Count | XGBoost (Parallel) [ms] | Dense Retrieval [ms] | Cross-Encoder (Est.) [ms] |
|---|---|---|---|
| 100 | 7.21 | 5.1 | |
| 500 | 36.03 | 5.5 | |
| 1,000 | 72.07 | 6.0 | > 1,000 |
| 2,000 | 144.14 | 7.0 | |
| 5,000 | 360.34 | 10.0 |
| Backbone | Label Homogeneity | Statute Hit Recall | Induced Skill F1 |
|---|---|---|---|
| TF-IDF | 54.3% | 48.7% | 52.1% |
| Sentence-BERT | 71.2% | 65.4% | 71.0% |
| BGE-Large-v1.5 (KDASO) | 83.6% | 79.2% | 78.5% |
| Law-BERT | 84.1% | 79.3% | 77.2% |
| Scenario | Baseline O-Precision | Baseline O-Recall | KDASO O-Precision | KDASO O-Recall |
|---|---|---|---|---|
| CA | 91.68% 1.0% | 81.66% 0.9% | 87.98% 1.0% | 78.54% 0.9% |
| CR | 81.88% 0.9% | 63.05% 0.8% | ||
| CDC | 96.15% 0.3% | 70.00% 2.1% | 96.57% 0.1% | 71.99% 1.2% |
| CPC | 88.19% 1.2% | 45.69% 2.1% | 88.51% 0.9% | 47.14% 1.8% |
| FD | 77.17% 0.9% | 60.85% 0.8% | 79.07% 0.9% | 66.29% 0.8% |
| Ride-hailing Adjudication (CPC) | ||
| Generating Model | Reasoning Model | O-Precision |
| QwQ-32B | DiDi Baseline | 87.39% 0.95% |
| DiDi Baseline | DiDi Baseline | 85.77% 0.94% |
| DiDi Baseline | QwQ-32B | 84.21% 0.93% |
| QwQ-32B | QwQ-32B | 88.51% 0.90% |
| Legal Liability Determination (LE-LH) | ||
| Metric | Qwen3-4B | Qwen3-8B | Qwen3-30B-A3B-Thinking | Qwen3-32B | QwQ-32B |
|---|---|---|---|---|---|
| O-Precision | 59.00% 0.67% | 67.44% 0.81% | 67.87% 0.82% | 68.68% 0.83% | 73.25% 0.89% |
| O-Recall | 85.30% 0.94% | 65.81% 0.80% | 60.70% 0.74% | 78.11% 0.91% | 78.75% 0.92% |
| Generator | Params | QwQ-32B | DeepSeek-V4-Flash | GLM-5 |
|---|---|---|---|---|
| Qwen3.5-27B | 27B | 78.2 | 79.5 | 79.2 |
| QwQ-32B | 32B | 80.4 | 80.5 | 80.2 |
| DeepSeek-V4-Flash | 284B/13B MoE | 81.0 | 82.0 | 81.8 |
| MiniMax M2.5 | 400B MoE | 79.8 | 80.8 | 80.5 |
| Kimi K2.5 | 1T/32B MoE | 78.5 | 81.2 | 80.9 |
| Qwen3.5-397B | 397B/17B MoE | 80.5 | 81.5 | 81.2 |
| Model | Params | Type | Base Score | +KDASO | (Gain) |
|---|---|---|---|---|---|
| Qwen3-8B | 8B | Open-Source General LLMs | 33.6 | 39.6 | +6.0 |
| QwQ-32B | 32B | Open-Source General LLMs | 39.8 | 44.0 | +4.2 |
| Qwen3.5-2B | 2B | Open-Source General LLMs | 28.5 | 32.0 | +3.5 |
| Qwen3.5-27B | 27B | Open-Source General LLMs | 40.2 | 41.8 | +1.6 |
| MiniMax M2.5 | 400B MoE | Open-Source General LLMs | 41.5 | 44.8 | +3.3 |
| Kimi K2.5 | 1T/32B MoE | Open-Source General LLMs | 42.0 | 45.2 | +3.2 |
| Model Configuration | Score |
| QwQ-32B | 39.8 |
| w/o Neighborhood-Conditioned Skill Induction | 40.1 |
| w/o Execution-Aligned Skill Refinement | 42.7 |
| w/o Context-Budgeted Skill Routing | 43.5 |
| KDASO QwQ-32B | 44.0 |
| KDASO QwQ-32B (LE-LH LA-ITI) | 43.6 |
LLM Agent Skill Learning
Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.
See all 428 papers