Large language model (LLM)-based coding agents achieve impressive results on controlled benchmarks yet routinely produce pull requests that real maintainers reject. The root cause is not functional incorrectness but a lack of organicity: generated code ignores project-specific conventions, duplicates functionality already provided by internal APIs, and violates implicit architectural constraints accumulated over years of development. Simply exposing an agent to the latest repository snapshot is not enough: the snapshot reveals the final state of the codebase, but not the repository-specific change patterns by which that state was reached. We introduce Learning to Commit, a framework that closes this gap through Online Repository Memory. Given a repository with a strict chronological split, the agent performs supervised contrastive reflection on earlier commits: it blindly attempts to resolve each historical issue, compares its prediction against the oracle diff, and distils the gap into a continuously growing set of skills-reusable patterns capturing coding style, internal API usage, and architectural invariants. When a new PR description arrives, the agent conditions its generation on these accumulated skills, producing changes grounded in the project's own evolution rather than generic pretraining priors. Evaluation is conducted on genuinely future, merged pull requests that could not have been seen during the skill-building phase, and spans multiple dimensions including functional correctness, code-style consistency, internal API reuse rate, and modified-region plausibility. Experiments on an expert-maintained repository with rich commit history show that Online Repository Memory effectively improves organicity scores on held-out future tasks.
Figures & tables
Framework / Paradigm
Chronological Eval
Historical Memory
Zero Data Leakage
Oracle Supervision
Organicity Eval
SWE-bench [ 1 ]
✗
✗
✗
✗
✗
SWE-Bench++ [ 14 ]
✗
✗
✓
✗
✗
RepoMem [ 8 ]
✗
✓
✗
✗
✗
SWE-CI [ 7 ]
✓
✗
✗
✗
✗
SWE-Bench-CL [ 13 ]
✓
✓
✗
✗
✗
Learning to Commit (Ours)
✓
✓
✓
✓
✓
Table 1: Comparison of representative repository-level benchmarks and agentic learning paradigms. Columns progress from basic capabilities to our unique contributions: leveraging historical context, continual learning across tasks, strict data leakage prevention, oracle-supervised memory updates, and multi-dimensional organicity evaluation. Symbol legend: ✓ = fully satisfies; ✗ = not supported.
Figure 1: Overview of the Learning to Commit framework. Real repositories are time-split into past commits (for skill learning) and future commits (for evaluation). In Step 1, the agent performs on-policy contrastive reflection: it blindly attempts each historical commit, then compares its prediction against the oracle diff to extract repository-specific development patterns into a structured skill document. In Step 2, the agent generates organic patches for unseen future tasks, guided by the accumulated skills. In Step 3, generated patches are evaluated against oracle commits across multiple dimensions including file IoU, line deviation, and multi-dimensional LLM judge scores.
Setting
File IoU (Skill / Base)
Steps (Skill / Base)
Line Dev. (Skill / Base)
seq-all
80% / 61%
56.8 / 71.9
0.69 / 1.59
par-bycat
80% / 68%
62.7 / 62.7
0.62 / 0.88
par-all
81% / 72%
67.7 / 68.3
0.80 / 1.06
seq-bycat
71% / 71%
75.0 / 76.4
1.22 / 1.13
Table 2: Deterministic code metrics across four experimental conditions. Bold indicates the better result in each pair. File IoU measures localisation accuracy ( ↑ ), Steps measures solve efficiency ( ↓ ), and Line Deviation measures patch bloat relative to the oracle ( ↓ , closer to 0 is better).
Setting
Claude Judge
Gemini Judge
par-bycat
54%
57%
seq-all
55%
58%
par-all
46%
54%
seq-bycat
26%
46%
Table 3: Overall pairwise win rates (%) of the skill-conditioned agent, as judged by two independent LLMs. Bold indicates skill agent wins more than 50%.
Dimension
Skill Win
Base Win
Tie
Q1: Scope Alignment
25%
41%
34%
Q2: Logic Similarity
50%
25%
25%
Q3: Redundancy & Halluc.
54%
41%
5%
Q4: Code Style
27%
38%
36%
Table 4: Dimension-level breakdown for the par-bycat setting (averaged over Claude and Gemini judges).
Historical pull requests record the design decisions, compatibility constraints, and implementation patterns behind a codebase's current state. Experience relevant to a new task can span related changes whose descriptions emphasize different concerns. We introduce Chronos, a test-time framework that makes this connected history available to large language model (LLM)-based code agents. Chronos distills merged pull requests into structured experience cards and connects them through a typed graph of code-level, developer-intent, and organizational relations. Semantic search identifies entry cards, and weighted multi-hop expansion retrieves connected changes for selective reading. The same memory guides candidate generation and patch selection: a patch-focused change agent and a validation-strategy agent each develop a patch, and an evolution steward consults history to select between them. On SWE-Bench Verified, the full workflow improves SWE-Agent across all six evaluated LLM backbones, raising the mean resolution rate from 69.2% to 72.9% and reaching 79.8% with MiniMax M2.5. With the same backbone, it raises resolution rates from 48.3% to 51.7% on SWE-Bench Pro and from 41.0% to 43.5% on FEA-Bench Lite. Both experience-guided single-agent variants also outperform the base agent. In a human evaluation on 100 tasks with ten cards retrieved per task, graph-grounded retrieval increases the mean number of useful cards from 1.24 to 2.87 over flat semantic retrieval. These results demonstrate the value of PR relations for retrieving useful repository experience and of the evaluated workflows for applying that experience during patch generation and selection.
Xin Yin, Yiang Zhang, Zhiyuan Peng +3
Zhejiang University, China · National Industrial Information Security Development Research Center, China
Developers frequently write uninformative git commit messages such as "fix" or "update stuff", degrading the value of version-control history for code review, debugging, and onboarding. We present CommitLLM, a three-stage pipeline that generates concise, Conventional Commits-compliant messages from code diffs using a fine-tuned small language model. The system combines (1) QLoRA fine-tuning of Mistral-7B-Instruct-v0.2 on the CommitPackFT dataset, (2) constrained decoding to enforce brevity, and (3) deterministic post-processing to strip conversational artifacts and enforce format. On a 50-sample evaluation, CommitLLM achieves 98% format compliance (vs. 22% for vanilla Mistral), reduces average output length from 154.8 to 37.9 characters, and improves LLM-as-a-Judge scores from 1.97 to 3.68 out of 5. Notably, the post-processing layers contribute more to quality improvement than the fine-tuning itself, suggesting that for structured-output tasks, treating the LLM as a component in a deterministic pipeline is more effective than optimizing the model alone. The entire system runs on a single consumer GPU (NVIDIA T4, 16 GB VRAM).
Fine-tuning large language models for code editing has typically relied on mining commits and pull requests. The working hypothesis has been that commit messages describe human intent in natural language, and patches to code describe the changes that implement that intent. However, much of the previously collected data is noisy: commit messages are terse, human-written commits commingle several unrelated edits, and many commits come from simple, rule-based bots. The recent adoption of software engineering agents changes this landscape. Code changes co-authored by humans and agents are often accompanied by substantially more explicit natural-language descriptions of intent and rationale. Moreover, when these changes land in public repositories, they are implicitly filtered by humans: maintainers discard low-quality commits to their projects. We present AgentPack, a corpus of 1.8M code edits co-authored by Claude Code, OpenAI Codex, and Cursor Agent across public GitHub projects up to early October 2025. We describe the identification and curation pipeline, quantify adoption trends of these agents, and analyze the structural properties of the edits. Finally, we show that models fine-tuned on AgentPack can outperform models trained on prior human-only commit corpora, highlighting the potential of using public data from software engineering agents to train future code-editing models.
Yangtian Zi, Zixuan Wu, Aleksander Boruch-Gruszecki +2