Diverse Reasoning Paths Matter: Symbolic Equivalence Augmentation for Enhancing RL Exploration

Xiaoou Liu; Tiejin Chen; Hanyang Chen; Yuan Ling; Hua Wei

cs.SCOct 15, 2026

Diverse Reasoning Paths Matter: Symbolic Equivalence Augmentation for Enhancing RL Exploration

Authors: Xiaoou Liu, Tiejin Chen, Hanyang Chen, Yuan Ling, Hua Wei

Organizations: Arizona State University · Independent Researcher

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for improving the reasoning capabilities of LLMs. However, the role of supervised fine-tuning (SFT) data in enabling effective downstream RL remains underexplored. In this paper, we investigate a key hypothesis: the diversity of reasoning steps in chain-of-thought (CoT) during SFT is a critical enabler of RLVR performance. To test this hypothesis, we introduce EquivAug, a data augmentation framework that uses symbolic equivalence transformations to generate diverse yet provably correct reasoning paths. By converting CoT solutions into formal symbolic representations, applying equivalence transformations, and reconstructing natural-language explanations, EquivAug produces training data that differs in reasoning steps rather than surface-level lexical variation. Experiments on GSM8K and FOLIO across model scales demonstrate that EquivAug consistently achieves the highest reasoning path diversity after SFT and the strongest accuracy after GRPO. Our results establish that CoT diversity during SFT is positively correlated with downstream RLVR performance.

Figures & tables

Figure 1: Overview of our pipeline. Left: The EquivAug data augmentation framework converts chain-of-thought solutions into symbolic representation, applies equivalence transformation rules to generate structurally diverse yet semantically equivalent variants verified by a symbolic solver, and reconstructs them back into natural language. Right: The augmented data is used for supervised fine-tuning, followed by reinforcement learning with GRPO.

Figure 2: Effect of transformation complexity on performance (GSM8K on Qwen2.5-1.5B).

Figure 3: Out-of-distribution performance of different data augmentation methods on MATH500 and SVAMP.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Test accuracy for different models during GRPO training for GSM8K on Llama3.1-8B.

Figure 5: Entropy of different models during GRPO training for folio on Llama3.1-8B.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
Jun 22, 2026Stanley Wei, Juno KimRL for Language Model Reasoning Efficient Language Model Reasoning
1Princeton University · University of California, Berkeley.
Guided-RLVR: Guided Exploration in Reinforcement Learning with Verifiable Rewards
Oct 27, 2026Bo Yang, Zishuai Zhang, Weihua Li +3RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · School of Artificial Intelligence, Beihang University
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Aug 27, 2026Xingyu Shen, Huishuai Zhang, Peng Li +2RL for Language Model Reasoning RL for Language Models
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies

Jun 22, 2026J/K move · Enter open · S save

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

Stanley Wei, Juno Kim

1Princeton University · University of California, Berkeley.

RL for Language Model Reasoning Efficient Language Model Reasoning Reinforcement Learning with Verifiable Rewards RL Fine-Tuning LLM Reasoning

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.

Abstract

RL for Language Model Reasoning

Explore similar work

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

Guided-RLVR: Guided Exploration in Reinforcement Learning with Verifiable Rewards

Boosting LLM Exploration via Weak-Model Guidance in RLVR