Large language models (LLMs) are increasingly deployed as autonomous agents in non-verifiable domains such as financial services, where they must interpret user intent, rewrite queries, and retrieve information under domain-specific constraints. Although reinforcement learning (RL) can train such agents from the consequences of their actions, recent successes largely rely on verifiable tasks with automatically computed binary rewards. In non-verifiable retrieval tasks, assessing query quality requires scarce subject-matter expert (SME) judgment that is inherently graded rather than binary. With Group Relative Policy Optimization (GRPO), sparse rewards yield no useful learning signal when all sampled outputs within a group receive the same outcome, causing optimization to stall. We introduce a method that converts a small, fixed investment of SME judgment into a dense reward and combines it with retrieval-based metrics to form a hybrid, GRPO-compatible training signal. Experiments show improvements of up to 6% in Recall@3 and 6.7% in MRR@3 compared with sparse-reward baselines.
Figures & tables
Figure 1. The mechanism at the heart of the paper. (a) The query-rewriting agent: only the rewriter πθ is trainable: the downstream retriever and answer synthesizer are frozen, so the reward measures the rewriter’s causal effect on retrieval. (b) One prompt sampled into a group of G=4 rewrites (value of G is illustrative here), scored two ways. A binary top-3 hit reward collapses every rewrite that misses the threshold to 0 including o3 , which retrieves the answer document at rank 7 and is invisible to a top-3 cutoff while the dense SME rubric of Eq. ( 4 ) grades the same rewrites continuously and ranks o3 correctly as the best of the four. (c) The consequence for GRPO: the binary reward leaves std(r1:G)=0 for this group, so every advantage Ai in Eq. ( 1 ) is zero and no gradient reaches πθ (the degenerate case of Eq. ( 3 )); the dense SME reward keeps std(r1:G)>0 because near-misses are still ranked, so the group always spreads and every prompt yields a usable gradient. Dense SME reward is curated using SME judgements [Figure 4 , Appendix].
Table 1. Reward composition used in all experiments. Rretrieval is a retrieval driver modulated by two multiplicative guardrail gates with floors b and c (Eq. ( 5 )); RSME is a weighted rubric over four SME-elicited criteria (Eq. ( 11 )). The top-level blend (Eq. ( 4 )) trades the two off with a (SME weight 1−a ).
Method
R@3
MRR@3
R@10
MRR@10
% grad
No RL (prompting only)
Raw question, no rewrite
71.91
0.6296
83.32
0.6505
N/A
Few-shot LLM rewrite
67.90
0.5876
80.60
0.6107
N/A
Sparse-reward RL
Binary hit@ k
67.96
0.5885
80.82
0.6122
6.5
DeepRetrieval (Literature Search)
68.54
0.5929
80.95
0.6157
17.5
Table 2. Retrieval quality on FlashRAG -NQ. % grad is the fraction of GRPO groups with non-zero intra-group reward variance, i.e. 1−ρdeg of Eq. ( 3 ): the share of prompts that actually produce a gradient this measures the existence and quantity of learnable distinctions in the reward, not whether the resulting gradient points somewhere useful. Best per column in bold.
Run
AQ
OpenCompass
Mistral-7B
disagr.
Retrieval-only, k =3
0.7432
0.6369
0.8496
0.1199
+ SME a=0.85
0.7551
0.6475
0.8628
0.1198
+ SME a=0.70
0.7554
0.6469
0.8639
0.1206
+ SME a=0.55
0.7548
0.6474
0.8621
0.1198
Retrieval-only, k =10
0.7528
0.6444
0.8603
0.1199
+ SME a=0.85
0.7616
0.6531
0.8683
0.1189
Table 3. Judge-independent answer quality. AQ is the mean of OpenCompass and Mistral-7B scores are rubric-free LLM judges that were never part of the training reward; disagr. is their disagreement rate. Adding the SME rubric improves every column at both k .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Model & adapter
Base model
Qwen3-1.7B
Precision
bfloat16
Tuner
LoRA
LoRA rank / α
16 / 32
Target modules
all-linear
Appendix
Table 4. Complete GRPO training configuration, held constant across all reward conditions. Only the reward function and dynamic resampling differ between the dense runs and the binary control: resampling of zero-variance groups is enabled for the dense reward but disabled for the binary baseline, since resampling would mask the very gradient stall the control is meant to expose.
Evaluated@3
Evaluated@10
Reward
matched (train@3)
cross (train@10)
Δ
matched (train@10)
cross (train@3)
Δ
Recall (%)
No rewrite
71.91
–
83.32
–
Retrieval only
71.94
68.98
−2.96
81.96
80.90
−1.06
+ SME a=0.85
74.00
73.58
−0.42
83.94
82.92
−1.02
+ SME a=0.70
73.62
73.56
−0.06
83.91
83.10
−0.81
Appendix
Table 5. Cross-depth transfer. matched is train- k= eval- k ; cross is the same reward trained at the other depth. Δ is cross − matched, so negative means the depth mismatch cost performance. The no-rewrite baseline involves no training and is depth-agnostic. Single run per cell.
Figure 4. From SME judgment to a dense reward. A fixed budget of subject-matter-expert grading is elicited once, distilled into an LLM-executable rubric of named, individually weighted criteria, and validated against the SME-preferred ranking before it is trusted as a reward. Only after the rubric separates gold from baseline rewrites do we optimize the rewriter against it. Colour marks provenance, separating the stages that consume expert time from those that replay that judgment at scale. Four-stage loop turning SME judgment into a dense reward A left-to-right flow of four boxes: Elicit, where SMEs grade a sample and articulate why; Distil, where the rubric becomes an LLM judge prompt; Validate, which checks that the rubric ranks as the SME does; and GRPO, which fine-tunes against the validated reward. Labels beneath the arrows name the artifact that passes between stages. A dashed arrow returns from Validate to Distil, marked reweight or respecify before training. Box colour encodes provenance: SME-authored, LLM-executed, or computed from retrieval.