Group Contrastive Policy Optimization: Learning Beyond the Self-Exploration Limit with Golden Answers

Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving the reasoning abilities of large language models, yet group-based methods lose their learning signal when every sampled response receives zero reward. We propose Group Contrastive Policy Optimization (GCPO), a hybrid objective that preserves standard on-policy optimization for informative rollout groups and activates a proximal contrastive objective only when exploration completely fails. The auxiliary objective pairs an offline golden reasoning trajectory with the current policy's failed responses, thereby providing a reliable positive reference and prompt-specific hard negatives without treating the golden trajectory as an on-policy rollout. Its old-policy token probabilities are evaluated through teacher forcing and used solely to constrain the update, rather than as an importance-sampling correction for the teacher distribution. A first-order analysis shows that the resulting update increases a length-normalized likelihood margin between reliable reasoning and observed failures. Experiments across two model scales and 10 core mathematical reasoning benchmarks demonstrate that GCPO improves training efficiency and final reasoning performance over competitive supervised and RLVR baselines.