Using Experience Twice: Dynamic Process Rubrics for Privileged Token-Level Credit Assignment
Organizations: University of Chinese Academy of Sciences · Computer Network Information Center, Chinese Academy of Sciences · Zhejiang University
Abstract
Agent trajectories often contain both successful decisions and failed attempts, yet optimization with sparse trajectory-level rewards struggles to identify which critical decisions should be reinforced or suppressed. We observe that meaningful distinctions among on-policy trajectories can be distilled into natural-language process rubrics and used as process supervision signals. Based on this insight, we propose Online Rubric Self-Distillation (ORSD), which leverages self-derived process rubrics for dense token-level supervision, thereby augmenting outcome-centric reinforcement learning. Specifically, the current policy independently generates multiple rubric bundles at each step according to the same group of trajectories, extracting candidate process rubrics from cross-trajectory behavioral differences. A frozen verifier then evaluates all rubric items against each trajectory in the group, and selects one candidate bundle based on bundle-level alignment between rubric satisfaction and task success. ORSD treats the selected rubric bundle as privileged information during training, then compares the log-likelihoods of the original tokens with and without the rubric. The resulting likelihood gap provides token-specific process credit, which is added to the original trajectory-level GRPO advantage through additive advantage shaping. ORSD thus bridges trajectory-level outcome supervision and fine-grained process supervision, enabling token-level credit assignment using process signals. Extensive experiments on ALFWorld and WebShop show that ORSD accelerates policy improvement and boosts final task performance for both Qwen3-1.7B and Qwen3-4B under the same environment-interaction budget. Our code is available at https://github.com/AnnaXChun/ORSD.
Figures & tables
| Qwen3-1.7B | Qwen3-4B | |||
|---|---|---|---|---|
| Method | Score | Success (%) | Score | Success (%) |
| Baseline | 72.20 | 55.40 | 82.69 | 66.15 |
| Offline Rubric | 74.35 | 58.20 | 85.39 | 72.25 |
| ORSD w/o Rubric–Reward Alignment | 77.53 | 60.85 | 83.86 | 67.60 |
| Random Rubric | 73.24 | 59.60 | 84.94 | 69.90 |
| ORSD ( ) | 78.60 | 63.00 | 89.02 | 77.05 |
| Saved rubric | Rejected rubric |
|---|---|
| WebShop: Buy a women’s top with the requested attributes, color, and size for less than $30. | |
| 1. Identify the correct product category and specific product attributes (e.g., slim fit, loose fit, color, size, price) needed to find the desired item. | 1. Identify the appropriate product category and the key attributes specified in the user instruction. |
| 2. Ensure the search query includes all relevant keywords and filters to narrow down the results effectively. | 2. Use relevant product keywords and available search or filtering options to narrow down candidate items. |
| 3. Search using the relevant product attributes and narrow the candidates accordingly. Do not commit to a product based only on a partial match in the title or description. | 3. Select the appropriate product variant, such as the requested size, color, or other configurable attributes. |
| 4. Verify that the selected product meets the price criteria (lower than $30.00) before proceeding to purchase. | 4. Verify that the selected product satisfies the important constraints in the instruction, including product attributes and price requirements. |
| 5. Confirm that the product is correctly selected and the purchase action is initiated. | 5. Confirm that the final product matches the user’s request and complete the purchase successfully. |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Base models | Qwen3-1.7B; Qwen3-4B |
| Optimization | Full-parameter GRPO |
| Hardware | 4 A100 80GB |
| Prompts / rollouts per prompt | 8 / 8 |
| Optimizer steps | 300 |
| Rollout group size | 8 (ALFWorld and WebShop) |
| Parameter | ALFWorld | WebShop |
|---|---|---|
| Evaluation outcome | official won | official score and score |
| Episode horizon | environment termination | 15 assistant actions |
| Response budget | 16,384 | 30,720 |
| Interaction history | full history | full history |
| Bundle-selection outcome | official success | behavior-regularized return |
| Training reward | Eq. ( 12 ) | Eq. ( 13 ) |
Token-Level Credit Assignment
Token-level credit assignment allocates sequence-level rewards across the words, actions, or reasoning steps that shape a model’s outcome.
See all 60 papers