Large language models (LLMs) are increasingly deployed as autonomous agents in non-verifiable domains such as financial services, where they must interpret user intent, rewrite queries, and retrieve information under domain-specific constraints. Although reinforcement learning (RL) can train such agents from the consequences of their actions, recent successes largely rely on verifiable tasks with automatically computed binary rewards. In non-verifiable retrieval tasks, assessing query quality requires scarce subject-matter expert (SME) judgment that is inherently graded rather than binary. With Group Relative Policy Optimization (GRPO), sparse rewards yield no useful learning signal when all sampled outputs within a group receive the same outcome, causing optimization to stall. We introduce a method that converts a small, fixed investment of SME judgment into a dense reward and combines it with retrieval-based metrics to form a hybrid, GRPO-compatible training signal. Experiments show improvements of up to 6% in Recall@3 and 6.7% in MRR@3 compared with sparse-reward baselines.
Figures & tables
Figure 1. The mechanism at the heart of the paper. (a) The query-rewriting agent: only the rewriter πθ is trainable: the downstream retriever and answer synthesizer are frozen, so the reward measures the rewriter’s causal effect on retrieval. (b) One prompt sampled into a group of G=4 rewrites (value of G is illustrative here), scored two ways. A binary top-3 hit reward collapses every rewrite that misses the threshold to 0 including o3 , which retrieves the answer document at rank 7 and is invisible to a top-3 cutoff while the dense SME rubric of Eq. ( 4 ) grades the same rewrites continuously and ranks o3 correctly as the best of the four. (c) The consequence for GRPO: the binary reward leaves std(r1:G)=0 for this group, so every advantage Ai in Eq. ( 1 ) is zero and no gradient reaches πθ (the degenerate case of Eq. ( 3 )); the dense SME reward keeps std(r1:G)>0 because near-misses are still ranked, so the group always spreads and every prompt yields a usable gradient. Dense SME reward is curated using SME judgements [Figure 4 , Appendix].
Table 1. Reward composition used in all experiments. Rretrieval is a retrieval driver modulated by two multiplicative guardrail gates with floors b and c (Eq. ( 5 )); RSME is a weighted rubric over four SME-elicited criteria (Eq. ( 11 )). The top-level blend (Eq. ( 4 )) trades the two off with a (SME weight 1−a ).
Method
R@3
MRR@3
R@10
MRR@10
% grad
No RL (prompting only)
Raw question, no rewrite
71.91
0.6296
83.32
0.6505
N/A
Few-shot LLM rewrite
67.90
0.5876
80.60
0.6107
N/A
Sparse-reward RL
Binary hit@ k
67.96
0.5885
80.82
0.6122
6.5
DeepRetrieval (Literature Search)
68.54
0.5929
80.95
0.6157
17.5
Table 2. Retrieval quality on FlashRAG -NQ. % grad is the fraction of GRPO groups with non-zero intra-group reward variance, i.e. 1−ρdeg of Eq. ( 3 ): the share of prompts that actually produce a gradient this measures the existence and quantity of learnable distinctions in the reward, not whether the resulting gradient points somewhere useful. Best per column in bold.
Run
AQ
OpenCompass
Mistral-7B
disagr.
Retrieval-only, k =3
0.7432
0.6369
0.8496
0.1199
+ SME a=0.85
0.7551
0.6475
0.8628
0.1198
+ SME a=0.70
0.7554
0.6469
0.8639
0.1206
+ SME a=0.55
0.7548
0.6474
0.8621
0.1198
Retrieval-only, k =10
0.7528
0.6444
0.8603
0.1199
+ SME a=0.85
0.7616
0.6531
0.8683
0.1189
Table 3. Judge-independent answer quality. AQ is the mean of OpenCompass and Mistral-7B scores are rubric-free LLM judges that were never part of the training reward; disagr. is their disagreement rate. Adding the SME rubric improves every column at both k .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Model & adapter
Base model
Qwen3-1.7B
Precision
bfloat16
Tuner
LoRA
LoRA rank / α
16 / 32
Target modules
all-linear
Appendix
Table 4. Complete GRPO training configuration, held constant across all reward conditions. Only the reward function and dynamic resampling differ between the dense runs and the binary control: resampling of zero-variance groups is enabled for the dense reward but disabled for the binary baseline, since resampling would mask the very gradient stall the control is meant to expose.
Evaluated@3
Evaluated@10
Reward
matched (train@3)
cross (train@10)
Δ
matched (train@10)
cross (train@3)
Δ
Recall (%)
No rewrite
71.91
–
83.32
–
Retrieval only
71.94
68.98
−2.96
81.96
80.90
−1.06
+ SME a=0.85
74.00
73.58
−0.42
83.94
82.92
−1.02
+ SME a=0.70
73.62
73.56
−0.06
83.91
83.10
−0.81
Appendix
Table 5. Cross-depth transfer. matched is train- k= eval- k ; cross is the same reward trained at the other depth. Δ is cross − matched, so negative means the depth mismatch cost performance. The no-rewrite baseline involves no training and is depth-agnostic. Single run per cell.
Figure 4. From SME judgment to a dense reward. A fixed budget of subject-matter-expert grading is elicited once, distilled into an LLM-executable rubric of named, individually weighted criteria, and validated against the SME-preferred ranking before it is trusted as a reward. Only after the rubric separates gold from baseline rewrites do we optimize the rewriter against it. Colour marks provenance, separating the stages that consume expert time from those that replay that judgment at scale. Four-stage loop turning SME judgment into a dense reward A left-to-right flow of four boxes: Elicit, where SMEs grade a sample and articulate why; Distil, where the rubric becomes an LLM judge prompt; Validate, which checks that the rubric ranks as the SME does; and GRPO, which fine-tunes against the validated reward. Labels beneath the arrows name the artifact that passes between stages. A dashed arrow returns from Validate to Distil, marked reweight or respecify before training. Box colour encodes provenance: SME-authored, LLM-executed, or computed from retrieval.
Reinforcement learning from verifiable rewards (RLVR) drives chain-of-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation. We propose Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock overhead. We prove that this cosine admits a multiplicative decomposition into prediction-error and activation-pattern factors, providing a concrete characterization of what the alignment signal measures. On Qwen3-4B and Qwen3-8B, GAR consistently improves over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific data. Code and data are available at https://github.com/LQgdwind/GAR.
Leqi Zheng, Jinbo Su, Fang Niu +8
Tsinghua University · Renmin University of China · Institute of Information Engineering, CAS +4
While Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a promising post-training paradigm for Large Language Models (LLMs), its dependency on the gold label or domain-specific verifiers limits its scalability to new tasks and domains. In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR), a simple reward that uses only the policy model itself. Given a prompt, VIGOR samples a group of completions and assigns higher within-group rewards to outputs that induce smaller ℓ2 norms of the teacher-forced negative log-likelihood gradients under the current parameters. Intuitively, lower gradient norms suggest the completion aligns better with the current policy, serving as an intrinsic preference signal for policy optimization. To make this intrinsic signal practical for RL, we correct the systematic length bias of averaged token-level gradients with a T scaling, and apply group-wise rank shaping to stabilize reward scales across prompts. Across mathematical reasoning benchmarks, VIGOR outperforms the state-of-the-art Reinforcement Learning from Internal Feedback (RLIF) baseline, and it also exhibits cross-domain transfer to code benchmarks when trained only on math data. For instance, on Qwen2.5-7B-Base post-trained on MATH, VIGOR improves the average math accuracy by +3.31% and the average code accuracy by +1.91% over this baseline, while exhibiting more stable training dynamics. The code is available at https://github.com/ZJUSCL/VIGOR.
Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.