Teaching Code Agents to Ask Without Leaking Private Code

Wenjia Jiang; Jieke Shi; Zhensu Sun; Chengran Yang; Shidong Pan; Peixin Zhang; Yifan Jia; James Hoang

cs.AIOct 17, 2026

Teaching Code Agents to Ask Without Leaking Private Code

Authors: Wenjia Jiang, Jieke Shi, Zhensu Sun, Chengran Yang, Shidong Pan, Peixin Zhang, Yifan Jia, James Hoang,

Organizations: Singapore Management University · University of Alberta · Alberta Machine Intelligence Institute (Amii) · Columbia University ·

Abstract

Local coding agents can extend their reach by consulting a stronger cloud model. Such consultations, however, often expose the repository's private identifiers, file paths, and source code. Host-side masking hides recognized identifiers but leaves the surrounding code and the agent's phrasing unchanged, while training for less disclosure risks suppressing consultation or producing questions too vague to answer. We propose utility-verified abstraction (UVA), which teaches the agent to describe private entities by their role and type while retaining the technical details an expert needs. From solved repair trajectories, we rewrite each question into this form and keep the rewrite only if it discloses fewer private entities and the repair, replayed from the same execution state with the repair trajectories. Across two benchmarks, the trained agent consults more often than before training. On repository-level issue repair, it solves 37.6% of issues, against 25.3% for the same model working alone, a gain from the training as a whole rather than from consultation alone. Its questions disclose 3.7-fold fewer private entities and 1.4-fold less verbatim code than the untrained agent's questions to the same expert.

Figures & tables

Figure 1: One consultation, asked two ways. A local coding agent works through a repair and, at one step, consults a third-party cloud expert. (a) The untrained agent quotes three private identifiers of the repository in its question. (b) Our trained agent describes the same entities by role and type. The expert gives both the same advice and both repairs succeed, so the abstract question gets the same help while the identifiers stay on the device.

Figure 2: Overview of UVA. (a) Consultations are collected from successful student rollouts. (b) An offline teacher describes private entities by their roles and types. A rewrite is retained only if it strictly reduces measured disclosure and the repair passes the task’s tests when replayed from the saved state. (c) The student is fine-tuned with LoRA on the verified pairs together with repair trajectories. At inference, it generates consultations directly without a rewriter. The example and disclosure counts are illustrative.

	SWE-bench Verified					Terminal-Bench
Policy	$\mathrm{pass}@1\uparrow$	$\mathrm{pass}@3\uparrow$	Ask%	$L_{\mathsf{P1}}\downarrow$	$\mathsf{OCO}\downarrow$	$\mathrm{pass}@1\uparrow$	Ask%	$L_{\mathsf{P1}}\downarrow$	$\mathsf{OCO}\downarrow$
Solo (no ask)	25.3	31.4	0	n/a	n/a	20.2	0	n/a	n/a
Natural	24.6	32.4	4.8	4.42	8.4	24.7	7.9	4.18	4.0
Privacy prompt	23.8	31.2	1.4	4.64	9.5	20.2	7.9	3.17	2.3
Host sanitizer	25.0	28.6	4.6	0.97	6.0	22.5	5.6	0.71	3.6
Ours , by cloud expert

Table 1: Solve rate and outbound disclosure of each consultation policy. The first four rows are the baselines, all run with the untrained student, and the Ours rows are the student trained by UVA. The baselines solve about as many tasks with an expert as without one. With the same expert, the trained student solves more tasks and consults more often. Each of its questions also discloses fewer private entities and less verbatim code. Its private-entity count stays above the host sanitizer’s.

Figure 3: Ablation on training strategy and training data, and the overall costs. In (a) and (b), the trained students consult Claude Opus 4.8, while the marks taken from Table 1 and the zero-pair point are the untrained student with DeepSeek-V3.2. (a) Disclosure is

L_{\mathsf{P1}}

; the triangles mark Natural and Host sanitizer . (b) Nested budgets of

100

–

400

accepted pairs; disclosure is

\mathsf{OCO}

. (c) Cloud consultation cost per 1,000 tasks against the solve rate, each point labelled with its price.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Student	Qwen3.5-9B ( Qwen Team, 2026 ) , sampled at temperature $0.8$ with thinking disabled
Experts	DeepSeek-V3.2 for every consulting policy; GPT-5.6 Sol and Claude Opus 4.8 for UVA only
Teacher	DeepSeek-V3.2, prompted as in Appendix B.2 , temperature $0.3$ , at most $600$ output tokens
Training tasks	SWE-smith ( Yang et al., 2025 ) , whose source repositories are disjoint from the $12$ of SWE-bench Verified
Collection	A consultation is forced at step $10$ and every $12$ steps after it, at most five per rollout, within $75$ steps; the reply enters the trajectory and the forcing instruction does not
Gate	Eq. 3 with $\kappa_{\mathsf{P1}}=2$ on the detector-only count, the strict reduction, $V$ of Eq. 4 , and the replay from the snapshot at $h$

Appendix

Table 2: Implementation choices and hyperparameters.

Expert	Verified	Terminal-Bench
None ∗ ( Solo )	25.3	20.2
None	36.2	24.7
DeepSeek	37.6	29.2
GPT-5.6	38.3	36.0
Opus 4.8	40.6	43.8

Appendix

Table 3: Solve rate of the trained student with consultation disabled , against the untrained student working alone and the trained student with each expert. An asterisk marks the untrained student; every other row is the student trained by UVA, the

300

-pair student of Table 1 . Both columns are

\mathrm{pass}@1

, over the

500

Verified instances with

3

samples each and a single attempt on the

89

Terminal-Bench tasks.

	Natural	Host sanitizer	Ours
Ask%	$4.8$	$4.6$	$7.4$
Calls per asking rollout	$1.40$	$1.35$	$1.39$
Consulting rollouts	$13/72$	$2/23$	$52/111$
Non-consulting rollouts	$356/1{,}428$	$123/477$	$512/1{,}389$
$\mathrm{pass}@1$ , consulting	$18.1$	$8.7$	$46.8$
$\mathrm{pass}@1$ , non-consulting	$24.9$	$25.8$	$36.9$

Appendix

Table 4: Solve rate by whether a rollout consulted, and expected disclosure per task , on SWE-bench Verified. Natural is over its

1{,}500

rollouts and Host sanitizer over the one run of

500

behind its disclosure columns in Table 1 . Ours consults DeepSeek-V3.2. The rollout rows count solved rollouts over all rollouts in the group, and disclosure per task is Ask% times the calls per asking rollout times

L_{\mathsf{P1}}

per question.

Figure 4: Consultation and submission timing per policy. The upper half of each violin is the distribution of the steps at which the policy consulted, and the mirrored lower half that of the step at which each submitting rollout handed in its patch. Darker shading marks the interquartile range, and each median line carries its step. All halves share one kernel width and peak height and span only the steps their data cover, so the violins compare the shapes of the distributions, and Ask% in Table 1 gives how often each policy consults.

Task	sympy__sympy-20438 , “ is_subset gives wrong results”. One consultation, at step $21$ of $116$ , after the student has reproduced the failure.
Question	The failing test shows that ProductSet and FiniteSet with the same elements are not considered equal (b == c is False but set(b) == set(c) is True). When is_subset is called, it falls back to comparing self.intersect(other) == self, which fails for ProductSet because the intersection returns a FiniteSet . Additionally, the simplify () of Eq(b, c) fails with AttributeError: ’Complement’ object has no attribute ’equals’. What is the structural cause of this issue and how should is_subset be fixed?
Disclosure	$L_{\mathsf{P1}}=4$ , all four entities also named by the issue text; $\mathsf{OCO}=19$ .
Expert reply	The structural cause is that ProductSet does not implement eq or equals to compare element-wise with FiniteSet […] Immediate step: Override is_subset in ProductSet to compare element containment without relying on intersect .
Outcome	The student adds that override to sympy/sets/sets.py and guards the equals call in sympy/core/relational.py , and the tests fail. The reference patch edits sympy/core/relational.py and the set-comparison handlers under sympy/sets/handlers/ , and leaves sets.py untouched.

Appendix

Table 5: A Privacy prompt consultation that discloses four entities and misdirects the repair. Private entities counted by

L_{\mathsf{P1}}

are in bold; the expert reply is truncated.

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 326 papers

May 25, 2026

RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations

Code agents are currently having skillful performance on repository-level software engineering benchmarks, but it remains unclear whether success on end-to-end tasks such as issue resolution truly reflects repository context reasoning, the ability to identify the task-relevant information across multiple files and reason over the relations among them. To investigate this question, we introduce RepoMirage, a two-stage evaluation suite built on SWE-Bench Verified that adopts perturbation as a diagnostic tool to increase the demand for context reasoning by transforming how the repository is exposed. First, RepoMirage-Perturb applies three types of semantics-preserving repository-level perturbations, revealing a clear performance drop when correct solving requires broader context access. RepoMirage-Extend further turns perturbation-targeted structural bottlenecks into explicit tasks beyond issue resolution, where the average performance declines from 66.8% in the original setting to 25.3%, indicating a significant deficiency in repository context reasoning. Further trajectory analysis reveals an exploration drift, where agents access broader repository context but fail to turn it into effective structure information. Motivated by this observation, we propose RepoAnchor, a structure-first prototype workflow that separates repository exploration from downstream problem solving, and show that explicit structural scaffolding yields notable gains. These results uncover an previously overlooked gap in repository context reasoning for code agents and suggest that stronger structure-aware methods are potential to improve them.

Hanyu Li, Yichi Zhang, Speed Zhu +3

Beijing University of Posts and Telecommunications · Tsinghua University · Tencent

AI Coding Agents Software Engineering Benchmarks Automated Evaluation LLM Reasoning

Ask PDF

Aug 17, 2026

Executable Code Knowledge: Code as a Native, Validation-Carrying Knowledge Representation for AI Coding Agents

AI coding agents need more than relevant snippets: they need business semantics, validation evidence, relations, and assurance that their context is current. Existing systems usually infer or externalize this knowledge through retrieval, summaries, graphs, rules, or reverse specifications. We investigate a complementary representation in which selected code units directly carry agent-usable knowledge. We introduce Executable Code Knowledge (ECK) and define an Executable Code Knowledge Unit (ECKU) as a source-bound object combining stable identity, semantics, executable behavior, contracts, evidence, relations, provenance, validation state, and a query interface. Our Python prototype supports code-local authoring, manifest export, evidence execution, exact changed-line impact, freshness checking, and agent-facing projections. Across three real Python repositories and 26 controlled patch tasks, direct ECK provides executable test coverage for 11/11 evidence-bearing tasks and exact selectors for 9/11; hiding declared evidence reduces exact recovery to 1/11 (paired exact McNemar p=0.0078). ECK-derived rules recover 11/11 exact selectors, showing that rules are effective delivery artifacts while ECK supplies source binding, validation state, impact, and freshness. Exact changed-line impact matches independently authored labels on all 26 patches (12 unit links; precision, recall, and F1 all 1.000). AST-bounded fingerprints classify 50 positive changes and 17 unrelated same-file controls correctly, whereas static rules snapshots detect none of the 50 stale cases. Model-backed patch-review and cross-layer studies measure projection fidelity rather than independent impact discovery. These results support a hybrid architecture: retrieval for coverage, ECK for source and evidence governance, and projections for delivery.

Xueping Gao

Alibaba Cloud Hangzhou, China

Knowledge Representation AI Coding Agents

Ask PDF

Aug 6, 2026

CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents

Modern LLM coding agents such as Claude Code and OpenHands share a common inefficiency: they spend much of their token budget finding the file to patch, rather than patching it. On SWE-Bench Verified, a 30B OpenHands agent averages 23 rounds and 631K tokens per resolved issue, with many calls spent on grep, glob, and view_file during repository exploration. We introduce CodeGrep, a 14B retrieval agent trained end-to-end with GRPO to issue multi-turn parallel grep, glob, and read tool calls and return candidate files to a frozen downstream coding agent. On all 500 SWE-Bench Verified instances, CodeGrep preserves resolve rate while substantially improving efficiency: 27.0% versus 25.8% for the no-retrieval baseline, with 15% fewer rounds and 19% fewer tokens on resolved instances. Across retrievers, downstream utility follows a precision threshold: BM25 with precision 0.375 degrades the agent, Jina with precision 0.445 is neutral, and CodeGrep with precision 0.677 crosses the threshold at which retrieval begins to reduce rollout cost. To enable this study, we mine supervision from 67K open-source agent trajectories using CATM and build a Git-worktree environment for multi-turn agent RL. In our setting, applying the efficiency signal at the advantage layer rather than the reward layer reduces KL drift and translates cleanly into downstream efficiency. We will release the model, training pipeline, RL environment, and evaluation harnesses.

Wuya Chen, Yihao yang, Yang Cao +1

1Netease Guangzhou AI Lab · 2Independent Researcher

AI Coding Agents LLM Agent Workflow Optimization RL for LLM Agents Code Search

Ask PDF

Abstract

AI Coding Agents

Explore similar work

RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations

Executable Code Knowledge: Code as a Native, Validation-Carrying Knowledge Representation for AI Coding Agents

CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents