What RL on Expert-Built Tasks Teaches Nemotron 3.5 Super
Organizations: Surge AI
Abstract
We post-train NVIDIA's Nemotron 3.5 Super (120B total, 12B active parameters) with reinforcement learning (RL) on five expert-built task datasets, one at a time: real-world software engineering, complex constraints, long-horizon tool use, professional chart understanding and enterprise document reasoning. Each run trains a fresh rank-32 LoRA adapter with CISPO on Tinker, with no supervised stage, and each adapter is compared with the base model on the same tasks, harness and settings. On twelve headline benchmarks, eight built by other groups and four by Surge AI, the gains range from +4.5 points on IFBench to +13.8 on GDP_pdf, which rises from 10.4% to 24.2%; SWE-bench Pro V2 rises from 86.9% to 92.1%, Terminal-Bench 2.1 from 63.9% to 70.3%, MultiChallenge from 58.3% to 66.6%, AutomationBench from 14.8% to 27.5% and Chartography from 39.4% to 50.6%. The document adapter, trained without API tools, also gains +8.0 points on AutomationBench. In attempts read side by side with the base model's on tasks each adapter newly solves, the base model's failures recur in a few modes, and each adapter does more often what the base model missed: the tool-use adapter finds the data and the rules before acting, the coding adapter writes its own test or scratch files on SWE-bench Pro V2, the constraints adapter carries earlier turns into its answers, the chart adapter measures the marks in code, and the document adapter reads AutomationBench's spreadsheets.
Figures & tables
| Dataset (adapter) | Domain | Task format | Reward and grader | Training harness |
|---|---|---|---|---|
| Real-World SWE (RL-SWE) | Software engineering | A repository at a fixed commit and an expert-written change request | Pass only if every hidden test passes | OpenCode and mini-swe-agent ports, network blocked |
| Complex Constraints (RL-Constraints) | Instruction following | A single-turn request, often with source material, and a binary rubric | , ; GPT-6.1 Sol judge | One response, no tools |
| Long-Horizon Tool Use (RL-ToolUse) | Agentic tool use | A request inside a simulated company, with a rubric of requirements and prohibitions | Share of requirements met, cut for broken prohibitions; GPT-6.1 Sol judge | The company’s services as MCP tools, network blocked |
| Professional Chart Understanding (RL-Charts) | Chart reading | A professional chart, a value-reading question and an expert answer | Pass only if judged correct; GPT-6.1 Sol judge | Python kernel and crop tool |
| Enterprise Document Reasoning (RL-Docs) | Document reasoning | A PDF, a question and a rubric whose criteria carry expected values | , ; GPT-6.1 Sol judge | Page-view, zoom, search and Python tools |
| Benchmark | Adapter | Tasks | Metric | Grader | Harness |
|---|---|---|---|---|---|
| Built by other groups | |||||
| SWE-bench Pro V2 | RL-SWE | 635 2 | Resolve rate | Hidden tests | OpenCode T , no network |
| Terminal-Bench 2.1 | RL-SWE | 89 5 | Pass rate | Hidden tests | OpenCode T , open internet |
| DeepSWE | RL-SWE | 112 4 | Pass rate of the committed changes | Hidden tests | OpenCode T , context compaction |
| AdvancedIF | RL-Constraints | 1,643 2 | Every rubric item met, mean of three subsets | Claude Opus 5.5 | One response T per turn |
| IFBench | RL-Constraints | 300 8 | Share of constraints met, strict | Programmatic checkers | One response T |
| Benchmark | Adapter | Base | Trained | Gain [95% CI] |
|---|---|---|---|---|
| SWE-bench Pro V2 | RL-SWE | 86.9 | 92.1 | |
| Terminal-Bench 2.1 | RL-SWE | 63.9 | 70.3 | |
| DeepSWE | RL-SWE | 27.9 | 32.6 | |
| ComplexConstraints | RL-Constraints | 27.3 | 35.5 | |
| AdvancedIF | RL-Constraints | 50.6 | 58.3 | |
| IFBench | RL-Constraints | 72.3 | 76.8 |
| Evaluation | Setting | Base | Trained | Gain [95% CI] |
|---|---|---|---|---|
| RL-SWE | ||||
| SWE-bench Pro V2 | Second run, same settings | 86.6 | 91.8 | |
| Terminal-Bench 2.1 | Second run, same settings | 63.8 | 69.4 | |
| Terminal-Bench 2.1 | Another harness, Terminus 2 | 65.8 | 67.4 | |
| SWE-bench Verified | Another benchmark, same harness as SWE-bench Pro V2 | 73.3 | 72.8 | |
| Held-out SWE tasks | Repositories kept out of training; training harnesses and grader | 35.4 | 37.8 | |
| Base failure mode (pairs) | Behavior after RL | Link to training | Rate over all attempts |
|---|---|---|---|
| RL-ToolUse on AutomationBench, 30 pairs | |||
| Data or target not found (12) | Keeps looking: lists the Drive, queries by exact name | Share of requirements met; a stop meets none | Never reads Sheets or Drive: 24.4 7.2, |
| Rule or update kept elsewhere, not read (10) | Reads every source first: the whole inbox, every tab | A broken prohibition drops the reward to zero or below | Never opens the inbox or Slack: 74.3 41.2, |
| RL-SWE on DeepSWE, SWE-bench Pro V2 and Terminal-Bench 2.1, 30 pairs | |||
| Checks that missed the error (8) | Tests the new behavior with a check of its own | Pass only if all hidden tests pass | Own test or scratch file, SWE-bench Pro V2: 14.3 32.6, |
| Lost requirement (13) | Carries each requirement into the code | One dropped requirement scores zero | – |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| RL-SWE | RL-Constraints | RL-ToolUse | RL-Charts | RL-Docs | |
|---|---|---|---|---|---|
| Deployment | early access | documented | early access | documented | documented |
| Peak learning rate | |||||
| Schedule to the reported checkpoint | constant | constant | constant to mid-run, then linear decay | constant | constant |
| Staleness cap (sampler versions) | 4 (soft) | 3 | 2 (soft) | 8 | 2 (admission held) |
| Token cap | 32,768 per turn | 32,768, then 49,152 per response | 32,768 per turn | 32,768 per turn | 32,768 per turn |
| Context | 258,048 | one response | 258,048 | 258,048 | 258,048 |
| Benchmark | Any 13-gram | Max share (%) | Notes |
|---|---|---|---|
| Terminal-Bench 2.1 | 0 | 0.0 | The benchmark’s canary string appears in no training prompt |
| SWE-bench Pro V2 | 0 | 1.0 | No SWE training task shares a repository with the three SWE benchmarks |
| DeepSWE | 0 | 0.4 | |
| SWE-bench Verified | 0 | 6.6 | |
| IFBench | 0 | 0.0 | |
| AdvancedIF | 1 | 5.1 | Without the 13-gram item: |
| Subgroup | Base | Trained | Gain [95% CI] | B/W/T |
|---|---|---|---|---|
| AutomationBench (RL-ToolUse), pass rate, by department | ||||
| HR | 10.0 | 33.0 | 24/1/75 | |
| Sales | 7.0 | 22.0 | 17/2/81 | |
| Operations | 24.0 | 37.0 | 18/5/77 | |
| Finance | 17.0 | 27.0 | 16/6/78 | |
| Marketing | 11.0 | 21.0 | 14/4/82 | |
| Subgroup | Base | Trained | Gain [95% CI] | B/W/T |
|---|---|---|---|---|
| MultiChallenge, pass rate per attempt, by axis | ||||
| Inference memory | 60.7 | 70.0 | 46/16/47 | |
| Instruction retention | 74.8 | 81.2 | 23/5/41 | |
| Reliable version editing | 45.0 | 59.4 | 19/6/15 | |
| Self-coherence | 52.6 | 55.7 | 18/10/20 | |
| AdvancedIF, every rubric item met, by subset | ||||
RL Fine-Tuning
Reinforcement learning fine-tuning adapts pretrained models and policies through task rewards, preference feedback, or environment interaction.
See all 241 papers