cs.LGNov 11, 2026

The Illusion of Diversity: Aligning LLM Exploration via Effective Entropy

Authors: Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Cong Qin, Binbin Zheng, Chaowen Hu, Zekai Shao

Organizations: Fudan University · Meituan · Tsinghua University · Peking University · 

Abstract

Reinforcement learning with verifiable rewards has shifted the alignment paradigm of large language models from subjective preference to objective correctness, yet a fundamental disconnect persists between exploration diversity and reasoning validity. Existing strategies predominantly rely on "blind" entropy metrics, e.g., token-level or global sequence-level entropy, which structurally fail to distinguish between effective reasoning and potential hallucinations. This misalignment creates an illusion of diversity, where high entropy tends to signal chaotic degeneration rather than valuable exploration. To bridge this gap, we introduce Effective Entropy, a validity-aware entropy concept that quantifies diversity exclusively within the valid subspace and systematically filters out the noise of incorrect trajectories. Built on this concept, we instantiate VEPO, a simple Validity-aware Entropy Policy Optimization method that directly incorporates effective entropy into RLVR training. VEPO induces an adaptive attraction dynamic that drives up the probability of under-explored valid trajectories, a mechanism that matches the practical regime of large-scale autoregressive models where sequence probabilities remain extremely small. Experiments on six mathematical reasoning benchmarks with 1.5B and 7B DeepSeek-R1-Distill-Qwen models show consistent aggregate accuracy gains over GRPO and competitive repeated-sampling success. A direct evaluation with matched correct counts finds more distinct strategies on commonly solved MATH500 problems, supported by paired statistical tests and three-seed training comparisons.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Boosting LLM Exploration via Weak-Model Guidance in RLVR

    Aug 27, 2026Xingyu Shen, Huishuai Zhang, Peng Li +2Entropy Collapse in RLRL for Language Model Reasoning

  2. DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning

    Feb 23, 2026Zhongwei Wan, Yun Shen, Zhihao Dou +9Reinforcement LearningRL for Language Model Reasoning

  3. Targeted Exploration via Unified Entropy Control for Reinforcement Learning

    Apr 16, 2026Chen Wang, Lai Wei, Yanzhi Zhang +5Entropy Collapse in RLReinforcement Learning