Edit Recommendation: Can LLMs Anticipate Edit Requests from Users, and When?
Organizations: NTT, Inc.
Abstract
After a large language model (LLM) agent generates an artifact, users often send edit requests for it. This paper studies whether an LLM can anticipate these requests before they are sent. Such anticipation is useful in practice because it reduces the effort of user review and feedback. We formalize this task as edit recommendation and build EditRecBench, a benchmark for evaluating the performance of LLMs on this task, from real-world conversation logs in WildChat-4.8M. Through benchmarking and failure analysis, we identify a practical challenge that some cases are inherently difficult to predict in real-world settings. To address this, we introduce decision model-based gating to filter out cases where predictions are unlikely to be correct before they are presented to the user. Our trained model outperforms existing decision models and raises pass@5 among shown suggestions by 12--25 points at a gating threshold. Our code is available at https://github.com/ntt-dkiku/llm-edit-recommendation.
Figures & tables
| pass@1 | pass@3 | pass@5 | pass@5 (no new info) | pass@5 (new info) | |
| Empirical ceiling | .40 [.38,.43] | .53 [.51,.56] | .61 [.58,.63] | .70 [.67,.73] | .30 [.25,.36] |
| Shuffled | .01 [.00,.01] | .01 [.01,.02] | .02 [.01,.03] | .03 [.02,.04] | .00 [.00,.01] |
| Stock requests (generic) | .06 [.05,.07] | .10 [.08,.11] | .11 [.09,.13] | .14 [.12,.16] | .02 [.00,.04] |
| Stock requests (popular) | .06 [.04,.07] | .13 [.12,.15] | .17 [.15,.20] | .20 [.17,.23] | .09 [.06,.13] |
| Llama 3.1 8B | .04 [.03,.06] | .07 [.06,.09] | .10 [.08,.12] | .13 [.11,.15] | .01 [.00,.02] |
| UserLM-8b | .03 [.02,.04] | .08 [.06,.09] | .11 [.09,.13] | .14 [.12,.16] | .02 [.00,.04] |
| Gating model | gpt-6-luna | gpt-oss-120b | Gemma 4 12B | Qwen3.5-9B | avg. | |
|---|---|---|---|---|---|---|
| Zero-shot | d3-mini | .549 [.52,.58] | .549 [.51,.59] | .564 [.53,.60] | .605 [.56,.65] | .567 [.55,.59] |
| pplx-decider-flash-v1.2 | .555 [.52,.59] | .574 [.54,.61] | .599 [.56,.64] | .622 [.58,.66] | .587 [.57,.61] | |
| gpt-6-luna (Decisions API) | .508 [.47,.54] | .569 [.53,.61] | .563 [.52,.60] | .549 [.51,.59] | .547 [.53,.57] | |
| Trained | Logistic regression | .563 [.53,.60] | .564 [.53,.60] | .598 [.56,.64] | .589 [.55,.63] | .579 [.56,.60] |
| Ours (base: Qwen3.5-2B) | .657 [.62,.69] | .631 [.59,.67] | .678 [.64,.71] | .687 [.64,.73] | .663 [.64,.68] | |
| Ours (base: Qwen3.5-4B) | .662 [.63,.69] | .665 [.63,.70] | .710 [.67,.74] | .723 [.68,.76] | .690 [.67,.71] |
| Gating model | gpt-6.1-sol | gpt-oss-20b | Llama 3.1 8B | UserLM-8b | avg. | |
|---|---|---|---|---|---|---|
| Zero-shot | d3-mini | .547 [.51,.58] | .575 [.53,.61] | .585 [.53,.64] | .566 [.51,.62] | .568 [.54,.59] |
| pplx-decider-flash-v1.2 | .546 [.51,.58] | .580 [.54,.62] | .616 [.56,.67] | .587 [.53,.64] | .583 [.56,.61] | |
| gpt-6-luna (Decisions API) | .531 [.50,.56] | .568 [.53,.61] | .593 [.53,.64] | .520 [.47,.57] | .553 [.53,.57] | |
| Trained | Logistic regression | .570 [.54,.60] | .543 [.50,.58] | .564 [.51,.62] | .540 [.48,.59] | .554 [.53,.58] |
| Ours (base: Qwen3.5-2B) | .617 [.58,.65] | .666 [.63,.70] | .611 [.55,.66] | .591 [.54,.64] | .621 [.60,.64] | |
| Ours (base: Qwen3.5-4B) | .640 [.61,.67] | .677 [.64,.71] | .723 [.67,.77] | .663 [.61,.72] | .675 [.66,.70] |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Predictor | Reasoning | Sampling |
|---|---|---|
| gpt-6.1-sol | medium effort | API defaults |
| gpt-6-luna | medium effort | API defaults |
| gpt-oss-120b | medium effort | 1.0, top- 1.0 |
| gpt-oss-20b | medium effort | 1.0, top- 1.0 |
| Gemma 4 12B | thinking on | 1.0, top- .95, top- 64 |
| Qwen3.5-9B | thinking on | 0.6, top- .95, top- 20 |
| Stock requests (generic) | Stock requests (popular) | |
|---|---|---|
| 1 | Make it shorter and more concise. | Make it shorter and more concise. |
| 2 | Add more detail and examples. | Fix the errors the user reported in the code so that it runs without error, and provide the complete corrected version. |
| 3 | Fix the mistakes and errors in it. | Please revise your last response so that it adds the specific content, fields, or details that are currently missing, expanding each item the way the request calls for while keeping everything else as it is. |
| 4 | Rewrite it in a different tone and style. | Please simplify your last response so it is easier to read and understand, using plainer words and shorter, less complicated sentences while keeping the same meaning. |
| 5 | Continue and expand it with the next part. | Please rewrite the entire response in Chinese, translating all of the content rather than only parts of it. |
| Human | ||||
|---|---|---|---|---|
| LLM | clear | borderline | unrelated | ethics |
| clear (accepted) | 93 | 5 | 2 | 0 |
| borderline (rejected) | 28 | 18 | 3 | 0 |
| unrelated (rejected) | 0 | 1 | 0 | 0 |
| count | value [95% CI] | |
|---|---|---|
| Agreement | 93 / 100 | .93 [.86, .97] |
| unanimous votes | 91 / 96 | .95 [.88, .98] |
| split votes | 2 / 4 | .50 [.15, .85] |
| Precision of matches | 46 / 50 | .92 [.81, .97] |
| Precision of non-matches | 47 / 50 | .94 [.84, .98] |
| Recall | 46 / 49 | .94 [.84, .98] |
| Artifact Actual edit request Predicted edit request | |
|---|---|
| Match | Code remove comments Just give me the interface without the comments, example, or explanation. List No just first and last name Write names without middle names Document please include beverage pairings and sweet treats as well Please add a dessert and beverage options to each of the six menus. Translation Use devnagri Write it in Hindi script instead of Roman Hindi. |
| No match | List no bullet point remove the intro, just give me the charges and statutes List not so conversational the description, make it something like “direct, fresh.. Can you make it even simpler and more natural? Some of the wording still sounds a bit robotic, especially “holistic” and “visual focal point”. Document maybe more on the design aspect Can you shorten the description to a single punchy tagline for my portfolio? Code Ok, excellent, but I want it to be a script that runs this and I can plot it too! It would be nice to return the solution matrix from wave1D_spectral rather than just plot. Could you modify the function signature to something like [x, u] = wave1D_spectral(L, Nx, c, tmax, dt, u0, v0) and make it more reusable? |
| Gating model | gpt-6-luna | gpt-oss-120b | Gemma 4 12B | Qwen3.5-9B | avg. |
|---|---|---|---|---|---|
| Context only | |||||
| d3-mini | .495 [.46,.53] | .496 [.46,.54] | .475 [.44,.51] | .485 [.44,.53] | .488 [.47,.51] |
| pplx-decider-flash-v1.2 | .510 [.48,.55] | .510 [.47,.55] | .558 [.52,.60] | .536 [.49,.58] | .528 [.51,.55] |
| Ours (base: Qwen3.5-2B) | .545 [.51,.58] | .553 [.51,.59] | .562 [.52,.60] | .537 [.49,.58] | .549 [.53,.57] |
| Ours (base: Qwen3.5-4B) | .574 [.54,.61] | .591 [.55,.63] | .596 [.56,.63] | .555 [.51,.60] | .579 [.56,.60] |
| With predicted edit requests | |||||
| Gating model | gpt-6.1-sol | gpt-oss-20b | Llama 3.1 8B | UserLM-8b | avg. |
|---|---|---|---|---|---|
| Context only | |||||
| d3-mini | .534 [.50,.57] | .503 [.46,.54] | .531 [.48,.58] | .517 [.47,.57] | .521 [.50,.54] |
| pplx-decider-flash-v1.2 | .506 [.47,.54] | .546 [.51,.58] | .544 [.49,.60] | .515 [.46,.57] | .528 [.50,.55] |
| Ours (base: Qwen3.5-2B) | .530 [.50,.56] | .557 [.52,.60] | .509 [.45,.57] | .498 [.44,.55] | .523 [.50,.55] |
| Ours (base: Qwen3.5-4B) | .547 [.51,.58] | .610 [.57,.65] | .567 [.52,.62] | .544 [.48,.60] | .567 [.54,.59] |
| With predicted edit requests | |||||
Large Language Model-Based Recommendation
Large language model-based recommendation uses language models to personalize ranking, retrieval, and conversational suggestions from text and behavior.
See all 238 papers