CaDO: Formal Verification-Guided Multi-Agent Framework for Causal Dataset Repair and Generation
Organizations: Shandong University Qingdao, China
Abstract
High-quality causal reasoning datasets are essential for developing models with reliable causal understanding, particularly in specialized domains. However, existing datasets frequently suffer from incorrect annotations and structural imbalances, which can lead to biased predictions and limited generalization. To address this issue, we propose CaDO (Causal-Engine-Guided Multi-Agent Data Optimization), a multi-agent framework for automated diagnosis and repair of causal reasoning datasets. Our framework consists of three key stages: formal causal reasoning, multi-level diagnosis, and automated data repair and generation. First, a Formalizer Agent translates natural language questions into formal causal queries to minimize semantic deviations and make retrieval and verification more explicit and tractable. Second, a dual-level diagnosis mechanism employs independent Critic Agents to identify sample-level errors while analyzing query-type coverage and label distributions to detect dataset-level structural imbalances. Finally, the framework employs reusable repair operators grounded in formal causal semantics to correct erroneous samples, while Explorer and Expander Agent collaboratively produce diverse and semantically consistent new samples. Experimental results show that CaDO significantly outperforms the strongest of five data cleaning baselines in 16 out of 18 experimental settings, improving downstream model accuracy by up to 9.1%. Moreover, at least 95% of the samples flagged by CaDO are defective, and on CLadder, CaDO incorrectly modifies 92.2% fewer correct labels than an LLM-based judge, demonstrating its effectiveness and reliability. Our code and data are publicly available at https://github.com/thisissyu/cado.
Figures & tables
| Verdict | Condition | Repair |
|---|---|---|
| consistent | , rationale consistent | none |
| rationale error | , rationale’s numbers, graph, arithmetic or conclusion inconsistent | rationale |
| label error | on firm ground, rationale follows the text | label and rationale from the engine |
| premise mismatch | on firm ground, rationale uses numbers or links the text does not state | label and rationale from the engine |
| unanswerable | stable, grounded reading, but the text determines no answer | remove |
| unresolved | no reading, a conflict , or without firm ground | none (abstain) |
| CLadder | CauGym | Corr2Cause | |
| Answer | yes/no | value, yes/no | valid or not |
| Query types | 10 | 7 | 6 |
| Clean train | 6,963 | 3,149 | 6,001 |
| Corrupted (wrong) | 3,599 (1,080) | 1,863 (554) | 3,886 (1,166) |
| Dev | 997 | 350 | 1,076 |
| Headline test | 1,989 | 1,400 | 1,162 |
| External | CLadder | CauGym | Corr2Cause | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| evaluation model | Corr. | Best BL | CaDO | Clean | Corr. | Best BL | CaDO | Clean | Corr. | Best BL | CaDO | Clean |
| Qwen3-4B | 76.1 | 89.9 A | 94.4 † | 96.7 | 35.6 | 35.8 G | 38.0 | 39.4 | 88.6 | 91.6 A | 95.6 † | 97.1 |
| Qwen3-8B | 75.8 | 87.2 A | 95.2 † | 96.1 | 38.4 | 38.8 A | 39.6 | 41.3 | 88.6 | 92.3 J | 96.9 † | 97.9 |
| Llama-3.2-3B | 84.8 | 86.3 A | 95.4 † | 93.5 | 23.9 | 23.9 J | 27.1 † | 24.5 | 87.8 | 88.1 A | 95.5 † | 97.0 |
| Llama-3.1-8B | 86.2 | 88.5 A | 94.6 † | 95.4 | 29.8 | 30.3 A | 32.3 † | 33.3 | 88.2 | 91.4 J | 97.0 † | 97.7 |
| Ministral-3-3B | 91.3 | 89.4 A | 96.0 † | 97.2 | 31.4 | 31.6 A | 33.2 † | 33.1 | 89.5 | 91.1 A | 96.6 † | 97.6 |
| CLadder (1,080) | CauGym (554) | Corr2Cause (1,166) | ||||||||||
| Method | Err. | Fixed | Dam. | Rem. | Err. | Fixed | Dam. | Rem. | Err. | Fixed | Dam. | Rem. |
| Corrupted | 30.0 | 0 | 0 | 0 | 29.7 | 0 | 0 | 0 | 30.0 | 0 | 0 | 0 |
| Loss filter | 16.9 | 0 | 0 | 126 | 19.9 | 0 | 0 | 115 | 18.4 | 0 | 0 | 183 |
| Confident learning | 15.3 | 0 | 0 | 200 | 21.7 | 0 | 0 | 28 | 15.7 | 0 | 0 | 35 |
| GPT-Agree | 8.9 | 0 | 0 | 164 | 5.9 | 0 | 0 | 35 | 6.7 | 0 | 0 | 28 |
| GPT-Judge | 8.1 | 714 | 77 | 280 | 4.4 | 431 | 13 | 30 | 4.4 | 869 | 21 | 24 |
| CLadder | CauGym | Corr2Cause | |
|---|---|---|---|
| Flag precision (%) | 95.0 | 99.8 | 100.0 |
| Flag recall (%), all defects | 74.0 | 90.8 | 100.0 |
| label flipped | 80.6 | 90.9 | 100.0 |
| number changed | 81.9 | 92.9 | 100.0 |
| link changed | 50.5 | 90.4 | 100.0 |
| question changed | 85.7 | 92.1 | 100.0 |
| Under-covered | Label-skewed | ||||
| Training data | ETT | NIE | ATE | Yes excess | “No” acc. |
| Corrupted | 29.3 | 82.3 | 93.4 | +24.8 | 49.1 |
| Loss filter | 48.8 | 83.7 | 96.2 | +24.5 | 49.5 |
| Confident learning | 40.8 | 82.0 | 93.8 | +26.5 | 45.4 |
| GPT-Agree | 48.3 | 75.3 | 93.4 | +4.2 | 83.2 |
| GPT-Judge | 64.2 | 66.2 | 89.7 | +4.0 | 83.6 |
| Variant | CLadder | CauGym | Corr2Cause |
|---|---|---|---|
| Verification | |||
| GPT-Judge instead of the engine | 3.3 † | 0.4 | 0.4 |
| one Formalizer Agent reading | +0.6 | +2.1 † | = |
| no Critic Agents | +0.5 | +3.2 † | = |
| repairs of every tier | +0.8 | +2.5 † | = |
| Generation | |||
| CLadder | CauGym | Corr2Cause | |
| LLM calls (M tokens) | |||
| CaDO | 1,635 (2.3) | 967 (2.0) | 2,334 (3.4) |
| GPT-Agree | 3,599 (1.8) | 1,863 (1.7) | 3,886 (5.3) |
| GPT-Judge | 3,599 (3.0) | 1,863 (2.6) | 3,886 (5.3) |
| LLM-AutoDP | 6,192 (5.2) | 3,595 (6.0) | 6,629 (9.0) |
| GPU-hours of pipeline search | |||
Synthetic Data Generation for Language Models
Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.
See all 305 papers