cs.AIMar 4, 2026
SaveA Rubric-Supervised Critic from Sparse Real-World Outcomes
Organizations: OpenHands · UIUC · CMU
Abstract
Academic benchmarks for coding agents tend to reward autonomous task completion, measured by verifiable rewards such as unit-test success. In contrast, real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse. How can we bridge this gap? In this paper, we propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling. Specifically, we introduce Critic Rubrics, a rubric-based supervision framework with 24 behavioral features that can be derived from human-agent interaction traces alone. Using a semi-supervised objective, we can then jointly predict these rubrics and sparse human feedback (when present). In experiments, we demonstrate that, despite being trained primarily from trace-observable rubrics and sparse real-world outcome proxies, these critics improve best-of-N reranking on SWE-bench (Best@8 +15.9 over Random@8 over the rerankable subset of trajectories), enable early stopping (+17.7 with 83% fewer attempts), and support training-time data curation via critic-selected trajectories.
Figures & tables
Figure 1: Overview of our method: Learning a deployable critic from production traces. We convert real-world human–agent interactions into segments (user request agent actions finish), annotate each segment with trace-observable Critic Rubrics (24 dense behavioral signals), and combine them with sparse production outcome proxies (e.g., PR merge / code survival) to train a semi-supervised, multi-task critic that predicts both rubric features and segment success. The resulting critic supports best-of- reranking, compute-efficient early stopping, and trajectory selection for training-time data curation.
Figure 2: From sparse outcomes to dense feedback in real-world usage. A pull request (PR) provides a coarse outcome signal (merged or not). Each PR contains commits, which we attribute to segments —self-contained units of agent work within multi-turn conversations (§ 2.2 ). This hierarchy grounds supervision at multiple granularities: PR-merge labels apply to all segments linked to the PR, while code survival assigns fine-grained credit based on how much segment-authored code remains in the final diff. Critic Rubrics provide dense, outcome-agnostic supervision for every segment.
| With Labels | ||||
|---|---|---|---|---|
| Conversations | Segments | Code Survival | PR Merge | |
| Train | 37,855 | 150,290 | 4,266 | 8,203 |
| Test | 386 | 1,547 | 1,083 | 1,547 |
| Total | 38,241 | 151,837 | 5,349 (4%) | 9,750 (6%) |
Table 1: Real-world user-agent interaction data composition. Most segments lack success labels, illustrating label sparsity.
Figure 3: Rubric effects differ between benchmarks and real-world data. Each point shows the change in success probability when a rubric feature is present ( ), with 95% confidence intervals; red indicates FDR significance ( , where is the FDR-adjusted -value). Bottom (benchmarks). SWE-bench and SWE-Gym show strong, consistent negative effects for core agent-behavior failures—especially incomplete_implementation , insufficient_testing , and insufficient_debugging —indicating these behaviors reliably predict unit-test failure. Top (real-world). In contrast, effect sizes are smaller and less consistently significant under PR-merge and code-survival proxies, reflecting noisier supervision and multi-turn credit assignment; user follow-up features (e.g., correction or reversion requests) exhibit stronger associations with code survival than with PR merge. Overall, benchmarks highlight stable failure modes, while real-world data exposes proxy-dependent and interaction-dependent effects.
| Real-World Interactions | SWE-bench (Mixed Subset) | ||||||||||||
| Intrinsic | Best@ (%) | Early Stop | |||||||||||
| Model | AUC | F1 | Prec | Rec | AUC | F1 | Prec | Rec | @2 | @4 | @8 | Att | |
| PR Merge Objective | |||||||||||||
| No Real-World Data | 0.48 | 0.85 | 0.75 | 1.00 | 0.59 | 0.81 | 0.69 | 0.99 | 49.2 | 44.2 | 45.6 | +12.9 | 5.12 |
| Success-Only | 0.64 | 0.86 | 0.76 | 0.99 | 0.64 | 0.82 | 0.69 | 0.99 | 58.5 | 57.1 | 57.1 | +13.7 | 4.13 |
| Success + Rubrics | 0.58 | 0.86 | 0.75 | 1.00 | 0.64 | 0.81 | 0.69 | 0.99 | 66.8 | 72.4 | 72.1 | +18.4 | 1.44 |
Table 2: Comprehensive Model Comparison. Success + Rubrics models jointly predict behavioral features and success, while Success-Only models predict success without rubric supervision. Best@ and Early Stopping metrics are on the mixed-outcome subset (148 instances); Early Stopping uses fixed with showing improvement over random. SWE-bench uses combined Sonnet + Opus trajectories. All results use final checkpoint.
| Sonnet 4.5 | Opus 4.5 | Combined | ||||
|---|---|---|---|---|---|---|
| Model | @4 | MRR | @4 | MRR | @8 | MRR |
| No Real-World Data | +3.9 | 0.75 | -8.8 | 0.69 | -12.3 | 0.61 |
| Success-Only | +7.0 | 0.77 | -8.1 | 0.72 | +5.7 | 0.78 |
| Success+Rubrics (BCE-floor) | +3.9 | 0.76 | +2.6 | 0.74 | +15.9 | 0.83 |
| Success+Rubrics (BCE-round) | +1.8 | 0.74 | -1.2 | 0.72 | +14.2 | 0.81 |
| Success+Rubrics (MSE) | +1.8 | 0.74 | +0.4 | 0.73 | -12.3 | 0.63 |
Table 3: Cross-backbone generalization on mixed-outcome instances. : Best@ improvement over random. Success-Only overfits to Sonnet but degrades below random on Opus; rubric-supervised models generalize.
| Selection Strategy | Resolved | Rate | vs Base |
|---|---|---|---|
| Base Model (no SFT) | 233 / 500 | 46.6% | — |
| Critic-selected | 239 / 500 | 47.8% | +1.2 |
| Proxy-filtered (code survival = 1) | 252 / 500 | 50.4 % | +3.8 |
| Random | 231 / 500 | 46.2% | -0.4 |
Table 4: SFT data selection results on SWE-bench Verified. Critic-selected trajectories outperform random selection for agent fine-tuning, demonstrating that critic predictions provide actionable training signal.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Latency (s) | Speedup |
|---|---|---|
| LLM rubric annotator (o3) | 17.0 6.3 | 1.0 |
| Trained critic (Qwen3-4B) | 1.1 0.8 | 16 |
Table 5: Inference latency for rubric prediction. Measured on 10 real-world segments using the full rubric schema (24 features). LLM uses o3 via LiteLLM; critic uses self-hosted vLLM.
| Feature | Type | Description |
| Agent Behavioral Issues | ||
| misunderstood_intention | Binary | Agent misunderstood the user’s goal. |
| did_not_follow_instruction | Binary | Agent ignored explicit instructions. |
| insufficient_analysis | Binary | Agent failed to inspect relevant prior code/docs. |
| insufficient_clarification | Binary | Agent acted despite ambiguous requirements. |
| improper_tool_use_or_setup | Binary | Misused tools or had incorrect dependencies/config. |
Table 6: Comprehensive taxonomy of Critic Rubrics features derived from real-world traces. The “User Follow-Up Patterns” section applies only when a user replies after the agent finishes.
| Real-world Interaction | SWE-b | SWE-g | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Feature | Meta- | |||||||||
| Incomplete implementation | 1681 | +0.01 | 0.43 | 236 | -0.21 | 0.001 | 1007 | -0.21 | 0.001 | 0.001 |
| Insufficient testing | 3092 | -0.01 | 0.56 | 425 | -0.21 | 0.001 | 1319 | -0.14 | 0.001 | 0.001 |
| Insufficient debugging | 1383 | +0.02 | 0.30 | 289 | -0.18 | 0.001 | 1069 | -0.15 | 0.001 | 0.001 |
| Insufficient analysis | 1203 | -0.04 | 0.01 | 278 | -0.15 | 0.001 | 749 | -0.15 | 0.001 | 0.001 |
| Misunderstood intention | 930 | -0.05 | 0.003 | 123 | -0.25 | 0.001 | 282 | -0.16 | 0.001 | 0.001 |
Table 7: Full regression results for rubric feature effects on success. For each dataset, we report: (count when detected), (effect on success rate), and (Fisher’s exact test, FDR-corrected). Meta- is the harmonic mean across datasets. Bold indicates .
| Rubric Feature | AUC | Prev. | |
| Loop Behavior | 0.94 | 3.0% | 105 |
| Infra Issue Agent Caused | 0.91 | 0.5% | 16 |
| Insufficient Clarification Seeking | 0.89 | 4.1% | 146 |
| Insufficient Debugging † | 0.88 | 13.3% | 470 |
| Incomplete Implementation † | 0.82 | 13.6% | 482 |
| Risky Actions Or Permission Issues | 0.82 | 4.4% | 157 |
Table 8: Rubric prediction on 1,547 real-world test segments. This measures learnability/consistency (how well the critic reproduces the annotation protocol), not human-verified correctness. The critic achieves mean AUC of 0.78 across all features, and 0.81 on the top predictive features (marked with ) identified in § 3.3 . Prev. = prevalence; = positive examples.
| Model | Best@2 | Best@4 | Best@8 | MRR |
|---|---|---|---|---|
| Success + Rubrics (Survival) | ||||
| step 2000 | 75.8 | 77.7 | 78.0 | 0.940 |
| step 4000 | 75.8 | 77.8 | 78.3 | 0.945 |
| step 6000 | 75.8 | 77.7 | 78.4 | 0.944 |
| step 8000 | 75.9 | 78.0 | 78.1 | 0.943 |
| step 9658 (final) | 75.7 | 77.6 | 77.8 | 0.941 |
Table 9: Ablation: Training steps. Performance peaks at step 4000–6000, with mild degradation at longer training. We report final checkpoint for consistency.