Revision-Based Learning from Interactive Experience and Feedback

Jiawen Li; Chenming Ge; Alice Gao; Rishab Vempati; Yi-Hao Peng; Jason Wu

cs.AIOct 19, 2026

Revision-Based Learning from Interactive Experience and Feedback

Authors: Jiawen Li, Chenming Ge, Alice Gao, Rishab Vempati, Yi-Hao Peng, Jason Wu

Organizations: Purdue University · University of Michigan · University of Washington · MIT

Abstract

Code generation models are increasingly used to produce interactive artifacts, such as websites and user interfaces. Many properties that determine whether these interfaces are useful only become apparent through realistic interaction, making them difficult to incorporate into model training. In this paper, we introduce RELIEF (REvision-based Learning from Interactive Experience and Feedback), a method for improving code generation models using feedback on their generated interfaces. RELIEF uses a coding agent to revise these interfaces, then incorporates the improvements into their original generation traces to produce supervision for fine-tuning. To evaluate RELIEF, we collected a dataset of 200 synthetically generated websites paired with 355 interaction recordings and accompanying feedback from 24 users. We used critiques derived from code inspection, automated interaction, explicit human feedback, and human interaction recordings to fine-tune variants of two code generation models. Our results show that feedback effectiveness varies with base model capabilities, with critiques inferred from human interaction recordings leading to the largest usability improvements overall under our tested models.

Figures & tables

Figure 1: RELIEF revises generated interfaces using feedback from human users, computer-use agents, or code agents. It then reconstructs generation traces that would have produced the revised interfaces and fine-tunes the interface-generation model on these traces.

Figure 2: Feedback and code revisions per website, grouped by base model. We measure feedback volume by the number of written comments for human feedback and identified problems for other sources, and revision size by lines of code added and removed. Code agents identify several times more problems than other sources and lead to the largest revisions. Human interaction recordings also yield more critiques and larger revisions than written human feedback.

Figure 3: The share of each source’s critiques in eight themes (average across base models). Human interaction and CUA critiques focus most on search, written human feedback on filters and the cart, and code critiques spread across forms, products, and the cart.

Figure 4: Overview of Git-guided trace reconstruction. We replay the rejected trace

\tau^{0}

with step-wise commits, compute the diff between the rejected and revised websites, and used a process similar to git blame to attribute changes to generation steps

a_{n}

, which are then retroactively revised to construct the trace for supervision

\hat{\tau}^{+}

Inkling Small
Fix Condition	Eligible	Completion	$\Delta$ (pp)
Original	36/37	81%	–
Code Critique	37/37	86%	+5
CUA Critique	37/37	86%	+5
Human Interaction	37/37	85%	+4
Human Feedback	35/37	83%	+2

Table 1: Static eligibility and task completion before and after feedback-guided revisions, evaluated on 100 tasks across 37 websites.

\Delta

(pp) denotes the change from the original baseline in percentage points. Bold indicates the highest completion rate per model.

Inkling Small
Condition	Eligible @1	Task Completion
Base Model	12/14	145/184 (78.8%)
Code Critique	13/14	132/184 (71.7%)
CUA Critique	14/14	152/184 (82.6%)
Human Interaction	12/14	156/184 (84.8%)
Human Feedback	13/14	139/184 (75.5%)

Table 2: Task completion results across two base models and four feedback sources. Eligible Sites @1 reports the number of websites passing eligibility checks on the first attempt. Task Completion reports success across 184 tasks after retries. Bold indicates the highest rate within each model.

Figure 5: Arena Elo ratings (95% CI) of models fine-tuned on each critique source. Critiques inferred from human interaction recordings (green) yield the highest-rated model for both Qwen and Inkling and are the only source rated above the base model (dashed line) on both.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW
AdamW $(\beta_{1},\beta_{2})$	$(0.9,0.999)$
AdamW $\epsilon$	$10^{-8}$
Weight decay	0
Learning rate	$5\times 10^{-6}$ (constant)
Effective batch size	32

Appendix

Table 3: Model hyperparameters used in training experiments.

Characteristic	Value
Age
Mean (SD)	24.13 (2.94)
Range	19–31
Web Design Experience
None	4 (16.7%)
A little (university course)	8 (33.3%)

Appendix

Table 4: Demographic characteristics of the 24 participants in our user study. Age is reported as mean and standard deviation (SD), while categorical variables are reported as counts and percentages.

Panel of Figure 6 — Figure 6: User study interfaces. Left: Participants complete tasks on a designated website while their interactions are recorded. Right: After completing the tasks, participants review their interaction recordings and annotate usability issues they encountered.

Figure 7: Human preference evaluation interface. Annotators compared websites generated by different model variants while completing the same task. Model conditions were hidden, and presentation order was randomized.

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 313 papers

RLPF: Reinforcement Learning from Performance Feedback for Code Generation
Jul 29, 2026Huihao Jing, Haozhe Cui, Wenbin Hu +9RL for Code Generation Reinforcement Learning
The Hong Kong University of Science and Technology (HKUST)
Learning to Detect UI Principle Violations via Reinforcement Learning
Jul 22, 2026Nishi Mehta, Swathi Alse, Himani Kumawat +2Automated Evaluation Language Model Auditing
University of California, Santa Cruz · Carnegie Mellon University
Feedback Over Form: Why Execution Feedback Matters More Than Pipeline Topology in 1-3B Code Generation
Apr 23, 2026Charles Junichi McAndrewsLLM Self-Refinement Execution-Guided Code Generation

Jul 29, 2026J/K move · Enter open · S save

RLPF: Reinforcement Learning from Performance Feedback for Code Generation

Huihao Jing, Haozhe Cui, Wenbin Hu, Shaojin Chen+8

The Hong Kong University of Science and Technology (HKUST)

RL for Code Generation Reinforcement Learning Code Optimization RL Post-Training

Open paper AskPDF arXiv

Code models are increasingly trained with execution feedback, but most training signals still stop at correctness. This leaves an important gap for systems code: two programs can pass the same tests while differing greatly in runtime. We study how to train code agents to prefer faster correct implementations, rather than treating efficiency only as an evaluation metric. The key difficulty is that runtime is a fragile reward. It is meaningful only after a program is correct, varies across tasks, and gives little guidance when most sampled programs fail to compile or run. We propose RLPF, reinforcement learning from performance feedback, which turns execution outcomes into a staged reward. Failed programs are ordered by execution progress, while correct programs are ranked by their relative improvement from the baseline toward the expert reference. This gives useful feedback before correctness and performance-sensitive feedback after correctness. Fine-tuning Qwen3-32B with RLPF on PerfCodeBench raises correct-and-runnable solutions from

11.1\%

54.6\%

and improves relative efficiency from

8.1\%

38.6\%

. The trained model becomes competitive with stronger open-weight systems, and its optimization behavior transfers modestly to EffiBench-X. Additional studies show that model-generated references provide useful but weaker supervision, and that the full composite reward is more reliable than correctness-only or runtime-only baselines. These results suggest that code agents can be trained not only to pass tests, but also to optimize the programs they write.

Abstract

Code Generation

Explore similar work

RLPF: Reinforcement Learning from Performance Feedback for Code Generation

Learning to Detect UI Principle Violations via Reinforcement Learning

Feedback Over Form: Why Execution Feedback Matters More Than Pipeline Topology in 1-3B Code Generation