cs.LGAug 18, 2026

CoEvo: Oracle-Grounded Self-Evolution of a Single Model for Multi-Step Causal Reasoning

Authors: Jian Zhang, Bingyi Wang, Yizhi Liu

Organizations: Zhejiang University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-step causal reasoning requires chaining inferences where each step constrains the next. An early error propagates silently, and a correct answer reached via flawed logic evades outcome-level detection. In specialized domains, teacher LLMs err on intermediate steps, safety constraints restrict cloud distillation, and shifting conditions demand adaptation, leaving self-evolution as the practical route. Naive self-evolution can collapse: outcome-only rewards let the model exploit distributional shortcuts, and weak self-evaluation reinforces spurious paths into stable failure patterns. We exploit a key asymmetry: generating a correct chain is hard, but verifying a single step is easy. Many high-stakes domains admit a deterministic, queryable oracle, a physics simulator or rule engine over codified constraints. It checks asserted steps without teacher-level ability and abstains beyond its rules; it can check what the model asserts, never replace it. This enables CoEvo, an oracle-grounded self-evolution framework where a single model alternates between Proposer and Solver. As Solver, the model generates competing chains; intra-group debate exposes disagreement steps, a proxy for the capability boundary, and the oracle adjudicates them into process-level supervision. As Proposer, the same model constructs progressively harder scenarios inside oracle constraints, steering the curriculum toward deep multi-hop chains. Both roles are updated jointly, so training pressure co-evolves with the model. On industrial, clinical, and legal multi-step causal reasoning benchmarks, CoEvo enables an 8B LLM to sustain self-evolution, surpassing distillation baselines and the strongest proprietary reference on path correctness (82.1% vs. 71.4%). The trained model generalizes to unseen categories and systems, preserving root-cause accuracy.

Figures & tables

Figure 1: (a–c) Failure modes of unverified self-evolution; (d) the CoEvo training loop.
 IndustrialClinicalLegalAverage
MethodPathAccPathAccPathAccPathAcc
References (zero-shot)
Qwen3-8B (base)23.0132.3220.5235.8735.2544.6026.2637.60
Claude-Sonnet-4.674.6887.5459.9184.0979.5090.2971.3687.31
Qwen3.7-Max72.9679.2756.9180.4173.3879.5067.7579.73
Distillation (Qwen3-8B)
Table 1: Main results (%): reasoning path correctness (Path) and root-cause/conclusion accuracy (Acc); Average is the unweighted mean across domains. Best 8B-scale result in bold.
Figure 2: Self-evolution dynamics on the industrial probe set (three seeds; ±\pm 1 std error bars). (a) Path correctness; the plateau at iterations 7–8 triggers the stopping rule (appendix). (b) Outcome-only baseline: violation rate and output diversity.
VariantPathAcc
Mutual voting replaces oracle78.7986.96
Learned PRM replaces oracle88.3193.00
Anchor-only reward ( w1=w2=0w_{1}{=}w_{2}{=}0 )83.8991.20
Dense instance checking ( 3.4×3.4\times cost)92.4494.93
Frozen Proposer83.3890.61
Real-task RL, no Proposer84.9190.21
Table 2: Mechanism ablations (%, industrial, 3,942 cases); PRM: process-reward model; full table in appendix.
More in this topic

Causal Reasoning in Language Models

Causal reasoning in language models studies how models identify cause-and-effect structure, predict interventions, and answer counterfactual questions.

See all 102 papers

Explore similar work

CardsList
Aug 4, 2026

CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning

Many critical reasoning tasks, including clinical diagnosis, legal judgment, and industrial fault diagnosis, require step-dependent causal chains in which early errors propagate and correct conclusions can mask invalid reasoning. Although large language models perform well on such tasks, privacy, latency, and controllability motivate distillation into locally deployable models. Standard trajectory imitation does not correct process errors on the student's own rollout distribution. We propose CausalOPD, a curriculum online process distillation framework. A knowledge-augmented teacher first provides trajectories grounded in domain-specific causal rules, entity relations, and structural constraints. The student then generates on-policy trajectories, and the teacher identifies the first wrong step, defined as the earliest transition that verifiably violates available constraints. Starting from the verified prefix, short-horizon reinforcement learning repairs this localized failure. A causal-stage curriculum advances from evidence-level to mechanism-level and conclusion-level errors, following their propagation order. Across three domains, CausalOPD improves average path correctness by 23.4 percentage points over sequence-level online process distillation and reduces the right-label-wrong-reasoning rate from 15.7% to 4.4%. The domain-specific 8B students also surpass both evaluated proprietary references in path correctness across all domains.

Jian Zhang, Bingyi Wang, Yizhi Liu

Zhejiang University

Curriculum RLCausal Reasoning in Language ModelsProcess SupervisionCausal ReasoningOn-Policy Distillation
AskPDF
May 31, 2026

On the Generalization Gap in Self-Evolving Language Model Reasoning

Recent work suggests that large language models (LLMs) can improve through self-evolution (SE), using supervision signals generated by the model itself. In this work, we ask: under a strict closed-loop setup, where the self-evolution algorithm has access only to an unlabeled prompt set and a base model, how close can internally generated supervision come to oracle-supervised training? We analyze four representative strategies in a unified offline self-evolution framework: single-round verification, multi-turn revision with feedback, iterative training, and curriculum learning. Our primary experiments use Knights and Knaves (KK) logical reasoning tasks, which provide deterministic solutions, controlled difficulty levels, and a clean testbed for easy-to-hard generalization. We first show that self-evolution consistently improves over the base model, but plateaus after excessive training compute is invested, and eventually still leaves a non-trivial gap to oracle supervision. We find that multi-turn critic-revision with large models can reach strong self-evolution performance, with Gemma 12B nearly matching oracle-supervised training. Beyond Knights and Knaves, we also evaluate self-evolution on real-world reasoning benchmarks, where gains are also modest. Overall, our results characterize when closed-loop self-evolution can help and show how internally generated supervision remains insufficient under this minimal formulation.

Zhenting Qi, Susanna Maria Baby, Stefanie Anna Baby +5

Google Research · Harvard University · Google +1

LLM Self-RefinementLanguage ModelingLanguage Model Self-ImprovementLLM Reasoning
AskPDF
Sep 1, 2026

Learning What to Practice: Diagnosis-Guided Self-Evolution for Language Models

Self-play supports the self-evolution of language models, but solver performance can plateau or decline across rounds without guidance. Existing unguided methods typically use difficulty, learnability, or diversity signals to keep questions challenging and varied, without identifying which unresolved reasoning weaknesses to target. Existing guided methods rely on external task resources such as human examples, document corpora, or specified difficulty targets. We introduce DiagEvo, which guides question generation using the solver's failure history from self-play, without external task resources. Its diagnostician extracts recurring error causes and stores them in an error-cause memory. The memory groups related causes under skill nodes and tracks each as Active or Mastered according to self-consistency on targeted questions. The challenger uses these states and recurrence counts to balance cause-targeted generation with free exploration. Double-confidence filtering retains intermediate-difficulty questions only when the most common solver answer has a clear vote lead. With the default 4B diagnostician, DiagEvo outperforms all baselines in mean accuracy across nine benchmarks for each solver: Qwen3-4B, Qwen3-8B, and OctoThinker-8B. On Qwen3-8B, DiagEvo reaches 72.3% mean accuracy across five mathematical reasoning benchmarks, 4.5 percentage points above R-Zero. Its overall mean accuracy across nine benchmarks is 57.4%, 3.5 percentage points above SPICE. Ablations show that mixed generation, memory-state updates with cross-state stitching, and double-confidence filtering contribute to these gains.

Xincheng Wei, Yifan Ding, Fucheng Xiong +6

The Chinese University of Hong Kong, Shenzhen · Meituan, LongCat Team · Peking University +2

Self-Training for Language ModelsLanguage Model Self-ImprovementLLM Reasoning
AskPDF