Teaching Code Agents to Ask Without Leaking Private Code
Authors: Wenjia Jiang, Jieke Shi, Zhensu Sun, Chengran Yang, Shidong Pan, Peixin Zhang, Yifan Jia, James Hoang, Xiwei Xu, Zhenchang Xing, David Lo, Zhou Yang
Organizations: Singapore Management University · University of Alberta · Alberta Machine Intelligence Institute (Amii) · Columbia University · New York University · Singapore University of Technology and Design · AIDX TECH · Commonwealth Scientific and Industrial Research Organisation (CSIRO) · Canada CIFAR AI Chair
Local coding agents can extend their reach by consulting a stronger cloud model. Such consultations, however, often expose the repository's private identifiers, file paths, and source code. Host-side masking hides recognized identifiers but leaves the surrounding code and the agent's phrasing unchanged, while training for less disclosure risks suppressing consultation or producing questions too vague to answer. We propose utility-verified abstraction (UVA), which teaches the agent to describe private entities by their role and type while retaining the technical details an expert needs. From solved repair trajectories, we rewrite each question into this form and keep the rewrite only if it discloses fewer private entities and the repair, replayed from the same execution state with the repair trajectories. Across two benchmarks, the trained agent consults more often than before training. On repository-level issue repair, it solves 37.6% of issues, against 25.3% for the same model working alone, a gain from the training as a whole rather than from consultation alone. Its questions disclose 3.7-fold fewer private entities and 1.4-fold less verbatim code than the untrained agent's questions to the same expert.
Figures & tables
Figure 1: One consultation, asked two ways. A local coding agent works through a repair and, at one step, consults a third-party cloud expert. (a) The untrained agent quotes three private identifiers of the repository in its question. (b) Our trained agent describes the same entities by role and type. The expert gives both the same advice and both repairs succeed, so the abstract question gets the same help while the identifiers stay on the device.
Figure 2: Overview of UVA. (a) Consultations are collected from successful student rollouts. (b) An offline teacher describes private entities by their roles and types. A rewrite is retained only if it strictly reduces measured disclosure and the repair passes the task’s tests when replayed from the saved state. (c) The student is fine-tuned with LoRA on the verified pairs together with repair trajectories. At inference, it generates consultations directly without a rewriter. The example and disclosure counts are illustrative.
SWE-bench Verified
Terminal-Bench
Policy
pass@1↑
pass@3↑
Ask%
LP1↓
OCO↓
pass@1↑
Ask%
LP1↓
OCO↓
Solo (no ask)
25.3
31.4
0
n/a
n/a
20.2
0
n/a
n/a
Natural
24.6
32.4
4.8
4.42
8.4
24.7
7.9
4.18
4.0
Privacy prompt
23.8
31.2
1.4
4.64
9.5
20.2
7.9
3.17
2.3
Host sanitizer
25.0
28.6
4.6
0.97
6.0
22.5
5.6
0.71
3.6
Ours , by cloud expert
Table 1: Solve rate and outbound disclosure of each consultation policy. The first four rows are the baselines, all run with the untrained student, and the Ours rows are the student trained by UVA. The baselines solve about as many tasks with an expert as without one. With the same expert, the trained student solves more tasks and consults more often. Each of its questions also discloses fewer private entities and less verbatim code. Its private-entity count stays above the host sanitizer’s.
Figure 3: Ablation on training strategy and training data, and the overall costs. In (a) and (b), the trained students consult Claude Opus 4.8, while the marks taken from Table 1 and the zero-pair point are the untrained student with DeepSeek-V3.2. (a) Disclosure is LP1 ; the triangles mark Natural and Host sanitizer . (b) Nested budgets of 100 – 400 accepted pairs; disclosure is OCO . (c) Cloud consultation cost per 1,000 tasks against the solve rate, each point labelled with its price.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Value
Student
Qwen3.5-9B ( Qwen Team, 2026 ) , sampled at temperature 0.8 with thinking disabled
Experts
DeepSeek-V3.2 for every consulting policy; GPT-5.6 Sol and Claude Opus 4.8 for UVA only
Teacher
DeepSeek-V3.2, prompted as in Appendix B.2 , temperature 0.3 , at most 600 output tokens
Training tasks
SWE-smith ( Yang et al., 2025 ) , whose source repositories are disjoint from the 12 of SWE-bench Verified
Collection
A consultation is forced at step 10 and every 12 steps after it, at most five per rollout, within 75 steps; the reply enters the trajectory and the forcing instruction does not
Gate
Eq. 3 with κP1=2 on the detector-only count, the strict reduction, V of Eq. 4 , and the replay from the snapshot at h
Appendix
Table 2: Implementation choices and hyperparameters.
Expert
Verified
Terminal-Bench
None ∗ ( Solo )
25.3
20.2
None
36.2
24.7
DeepSeek
37.6
29.2
GPT-5.6
38.3
36.0
Opus 4.8
40.6
43.8
Appendix
Table 3: Solve rate of the trained student with consultation disabled , against the untrained student working alone and the trained student with each expert. An asterisk marks the untrained student; every other row is the student trained by UVA, the 300 -pair student of Table 1 . Both columns are pass@1 , over the 500 Verified instances with 3 samples each and a single attempt on the 89 Terminal-Bench tasks.
Natural
Host sanitizer
Ours
Ask%
4.8
4.6
7.4
Calls per asking rollout
1.40
1.35
1.39
Consulting rollouts
13/72
2/23
52/111
Non-consulting rollouts
356/1,428
123/477
512/1,389
pass@1 , consulting
18.1
8.7
46.8
pass@1 , non-consulting
24.9
25.8
36.9
Appendix
Table 4: Solve rate by whether a rollout consulted, and expected disclosure per task , on SWE-bench Verified. Natural is over its 1,500 rollouts and Host sanitizer over the one run of 500 behind its disclosure columns in Table 1 . Ours consults DeepSeek-V3.2. The rollout rows count solved rollouts over all rollouts in the group, and disclosure per task is Ask% times the calls per asking rollout times LP1 per question.
Figure 4: Consultation and submission timing per policy. The upper half of each violin is the distribution of the steps at which the policy consulted, and the mirrored lower half that of the step at which each submitting rollout handed in its patch. Darker shading marks the interquartile range, and each median line carries its step. All halves share one kernel width and peak height and span only the steps their data cover, so the violins compare the shapes of the distributions, and Ask% in Table 1 gives how often each policy consults.
Task
sympy__sympy-20438 , “ is_subset gives wrong results”. One consultation, at step 21 of 116 , after the student has reproduced the failure.
Question
The failing test shows that ProductSet and FiniteSet with the same elements are not considered equal (b == c is False but set(b) == set(c) is True). When is_subset is called, it falls back to comparing self.intersect(other) == self, which fails for ProductSet because the intersection returns a FiniteSet . Additionally, the simplify () of Eq(b, c) fails with AttributeError: ’Complement’ object has no attribute ’equals’. What is the structural cause of this issue and how should is_subset be fixed?
Disclosure
LP1=4 , all four entities also named by the issue text; OCO=19 .
Expert reply
The structural cause is that ProductSet does not implement eq or equals to compare element-wise with FiniteSet […] Immediate step: Override is_subset in ProductSet to compare element containment without relying on intersect .
Outcome
The student adds that override to sympy/sets/sets.py and guards the equals call in sympy/core/relational.py , and the tests fail. The reference patch edits sympy/core/relational.py and the set-comparison handlers under sympy/sets/handlers/ , and leaves sets.py untouched.
Appendix
Table 5: A Privacy prompt consultation that discloses four entities and misdirects the repair. Private entities counted by LP1 are in bold; the expert reply is truncated.