Large language models can produce fluent reasoning traces whose local semantic errors propagate to an incorrect conclusion, while unconstrained self-correction may preserve, amplify, or introduce errors. Existing diffusion language models provide iterative refinement, but usually define noise as token masking or replacement rather than as errors in the reasoning process. We present Semantic Reasoning Denoising (SRD), an operatorized Markov denoising method for natural-language reasoning trajectories. SRD represents semantic noise with executable error operators that describe the error type, its location, and the corrupted and repaired propositions. Composing these operators constructs progressively noisier states. During training, the model learns to identify the semantic noise active in the current trajectory and to reconstruct the paired adjacent lower-noise state. During inference, noise-level-aware denoising repeatedly predicts an inverse operator and checks whether it is applicable, so each executed update makes a localized move toward a stable trajectory. Across six in-domain benchmarks spanning mathematics, code, knowledge, and commonsense, SRD improves the strongest same backbone baseline by 3.2 points on average. On seven cross-dataset transfer targets, it remains competitive with Llama-3-8B-Instruct and improves the strongest Qwen3-8B baseline average by 2.9 points. Analyses of noise sources, objectives, and denoising depth further show that structured semantic-noise prediction and iterative operator execution are central to the improvement.
Figures & tables
Figure 1: Overview of SRD. (a) Real, synthetic, and interpolated failures are aligned into executable semantic-noise operators and adjacent states. (b) A LoRA denoiser predicts the active operator and adjacent lower-noise state. (c) From the base CoT, applicability-checked inverse operators iteratively recover the final answer.
Figure 2: Token-space and semantic denoising. (a) Token-space diffusion corrupts surface tokens without modeling reasoning validity. (b) SRD applies and inverts executable semantic operators over coherent reasoning states to recover the trajectory and answer.
Group
Method
GSM8K
MATH
HEval+
MBPP
MMLU
PIQA
Avg.
Base
CoT
77.2
31.0
52.1
51.3
63.0
81.9
59.4
Learn-to-solve
SFT
79.3
35.6
55.5
52.4
65.5
82.0
61.7
STaR
80.6
39.8
54.7
52.9
66.3
84.2
63.1
DiffCoT
82.1
39.1
55.9
56.1
65.4
87.3
64.3
Learn-to-correct
Self-Refine
80.4
34.5
52.8
51.6
62.7
83.5
60.9
RISE
82.3
38.5
53.2
52.0
68.9
83.4
63.1
Table 1: Main in-domain results; diffusion-model rows are reference-only.
Table 2: Cross-dataset transfer results; reading tasks report F1.
K
Calls
Gen. tok.
Latency (s)
Tok./fwd.
1
2
304
5.7
1.7
2
3
445
6.4
2.5
4
5
735
7.1
4.2
8
9
1366
9.5
5.4
16
17
2378
11.2
8.0
Table 3: MATH inference cost by denoising budget.
Figure 5: Accuracy versus denoising budget K on GSM8K, MATH, CSQA, and BBH; dashed lines show base CoT.
Variant
GSM8K
MATH
CSQA
BBH
Full SRD
84.8
43.1
82.6
83.3
w/o real failures
83.1
43.6
81.7
82.7
w/o synthetic
80.4
41.4
81.2
82.1
w/o interpolation
82.7
44.5
81.6
82.1
operator-only
79.1
40.8
80.9
81.8
adjacent-state-only
80.6
40.3
81.2
82.4
Table 4: Core ablations (best per column in bold ).
Reinforcement learning has become a central paradigm for advancing reasoning in large language models, yet most existing methods still depend on stronger teacher models or heavily curated difficult datasets, limiting scalable capability improvement. In this paper, we introduce DenoiseRL, a reinforcement learning framework that substitutes external supervision with recovery-oriented optimization over failures from weak models. Instead of relying on stronger supervision or carefully engineered data, DenoiseRL learns directly from incorrect reasoning traces by converting them into opportunities for improvement, making training more scalable and less dependent on external resources. This yields a richer and more diverse learning signal, improving exploration efficiency from imperfect model behavior. As a result, DenoiseRL improves reasoning performance and overall training efficiency while reducing the need for expensive data curation or stronger teacher models. Empirically, DenoiseRL consistently outperforms strong on-policy RL baselines across competitive mathematical and general reasoning benchmarks and promotes stronger self-corrective behavior as training difficulty increases, highlighting an effective and scalable alternative pathway for improving reasoning in large language models.
Caijun Xu, Changyi Xiao, Zhongyuan Peng +1
1Fudan University · 2Shanghai Innovation Institute
Large language models produce fluent but often incorrect multi-step reasoning, and naive correction methods risk degrading already-correct answers. We introduce Denoising Iterative Self-Correction (DISC), a test-time procedure that treats verification question outputs as noisy measurements of where a solution may be corrupted. Using these signals, DISC progressively reduces errors across multiple verify-judge-correct passes, analogous to traditional iterative denoising. A binary judgment gate controls correction precision by blocking rewrites that would damage already-correct answers while the verifier and corrector together repair errors. We evaluate this trade-off using two paired diagnostics: an improvement-to-degradation ratio (precision) and a repair rate (recall). Across three benchmarks (BIG-Bench Mistake, HotpotQA, GPQA Diamond) and four models, DISC dominates Chain-of-Verification and Self-Refine on the precision-recall trade-off, reaching 81.6% accuracy with 13x more improvements per degradation than Chain-of-Verification and 5x more than Self-Refine on BIG-Bench Mistake (Sonnet~4.5). On GPQA Diamond, we identify a capability floor below which judges acknowledge contradictions in evidence but cannot translate that recognition into a correction. We further show that cross-model role allocation -- assigning verification and judgment to a model different from the generator -- mitigates self-confirmation bias.
Drafting an answer allows a reasoning model to reconsider each decision in the context of the others. Yet repeated diffusion denoising does not ensure that individually plausible token updates form a better answer together. We introduce Draft Refinement Language Models (DRLMs), which use a single denoiser to propose parallel revisions and evaluate a candidate draft. We train the denoiser to reconstruct every token from uniformly corrupted drafts, teaching both preservation and repair. Its predictions specify revisions directly, without a diffusion reverse transition. Evaluation requires a different use of these predictions: confidence in an existing token reflects both contextual support and evidence from observing the token itself. Using the known corruption process, we correct for this self-token evidence to estimate how well each token is supported by the prompt and the remaining draft. Aggregating these estimates yields a cavity energy computed in one forward pass, without an additional scoring model. Each round proposes a revision, reads the completed candidate in its revised context, and retains it only if its energy is lower. Across Countdown, Sudoku-Extreme, and Maze-Hard, our method outperforms autoregressive and discrete diffusion baselines. With 6M parameters, DRLM reaches 44.4% on Countdown-5 and 93.9% on Sudoku-Extreme. The results establish complete-draft parallel refinement as an effective alternative to reverse diffusion for complex reasoning.