cs.LGOct 13, 2026
SaveInteraction-as-a-Judge
Organizations: Carnegie Mellon University · Arena · University of California, Berkeley · University of California, Los Angeles
Abstract
Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess these applications interactively, current automated LLM evaluations primarily focus on code analysis and unit tests. This mismatch can cause LLM judges to miss failures that are immediately apparent to human users. We study Interaction-as-a-Judge, an evaluation paradigm where computer-use agents (CUAs) directly interact with an application. We demonstrate that Interaction-as-a-Judge can improve both evaluation and generation of interactive applications. To this end, we collect and release two complementary datasets: pairs of applications with human preferences and corresponding agent traces, and applications with human-verified requirement labels. Relative to standard non-interactive approaches, we find that Interaction-as-a-Judge improves evaluation in terms of predicting human preferences () and requirement verification (). We also observe scaling benefits: across both preference alignment and requirement verification, higher-quality interactions improve performance. Finally, through both inference- and training-time experiments, we demonstrate that Interaction-as-a-Judge can also improve the quality of generated artifacts. At inference time, Interaction-as-a-Judge repairs of broken requirements within five attempts, compared to for code-only self-feedback. At training time, using Interaction-as-a-Judge as a reward signal improves win-rate over the base model by pp while reducing both generated code length by and build failure rates by pp compared to a code-only reward signal.
Figures & tables
Figure 1: We present Interaction-as-a-Judge, an evaluation paradigm that uses interaction traces to improve evaluation and generation. We compare against non-interactive judges and evaluate on pairwise preferences and reward modeling. We also show how to apply Interaction-as-a-Judge for inference-time scaling and as a reward signal for standard post-training via reinforcement learning.
Figure 2: We study three approaches to computer-use. Iterative computer-use is analogous to standard frame-by-frame computer-use. One-shot computer-use plans once at the start to save costs. Heuristic computer-use uses a pre-determined algorithm (e.g., random) to choose actions.
Figure 3: We train a reward model with varying levels of interaction and observe that including interaction data improves accuracy in terms of alignment with human preferences. GPT 5.6 Luna is the only exception where adding interaction offers comparable performance to code only. We fine-tune from Qwen-3.5-9B as our base model and train each approach 5 times. Bolded values are statistically significant against code+screenshot, calculated by a sample-wise paired t-test.
Figure 4: We evaluate Interaction-as-a-Judge’s ability for requirement verification (left) and requirement discovery (right). For requirement verification, we see that Interaction-as-a-Judge is better than a code only Agent-as-a-Judge, especially at identifying broken requirements. Additionally, we observe that performance scales as model capabilities improve. We find that requirement discovery is still challenging for both approaches.
Figure 5: We compare Interaction-as-a-Judge against a code-only Agent-as-a-Judge at iterative self-feedback and requirement repair. Interaction-as-a-Judge repairs of initially broken requirements compared to repaired by the code-only Agent-as-a-Judge. We see this improvement even when evaluating using Sol as a code-only Agent-as-a-Judge. Additionally, the gap between self-reported accuracy and accuracy measured by GPT 5.6 Sol is less for Interaction-as-a-Judge.
Figure 6: We train a Qwen-3.5-9B model using code-only and interaction-only reward models. As expected, each performs better on its respective training objective, but the interaction-only model leads to improvements in neutral criteria such as build success rate and decreased code length. We use an auto-rubrics based judge and find a significant improvement over the base model. The build reward baseline naturally degrades in reward for both reward models.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Tool | Arguments and effect |
| mouse_click | ; presses and releases the left mouse button. |
| mouse_down | ; holds the left button until a later release. |
| mouse_up | ; releases the held left button. |
| mouse_move | and optional steps (default 8); interpolates cursor movement, preserving button state. |
| mouse_drag | from_x , from_y , to_x , to_y , optional steps (12), and hold_ms (50); performs a complete drag. |
| double_click | ; performs a double click. |
Table 1: Preference-agent action space. Positions are iframe-relative pixels; durations are milliseconds. A dispatched observation request also consumes a step in the implemented dispatcher.
| Mean | Median | Max | |
| ArenaPreferences | |||
| Prompt length (words) | 408 | 68 | 16,240 |
| Files per website | 15.6 | 14 | 233 |
| Lines of code | 1,940 | 1,540 | 17,666 |
| Lines of code (App.tsx only) | 399 | 276 | 4,329 |
| Interaction time before vote (seconds) | 230 | 188 | 600 |
Table 2: Summary statistics for ArenaPreferences (per website, across battles) and ArenaCorrectness ( traces, game versions). Lines of code exclude lockfiles.
Figure 7: Per-category preference agreement (%) for Gemini 3 Flash. Error bars show 95% normal confidence intervals over battles.
Figure 8: Per-category preference agreement (%) for GPT 5.6 Luna. Error bars show 95% normal confidence intervals over battles.
Figure 9: Per-category reward-model agreement (%) with full code input, averaged over five seeds. Error bars show 95% Wilson intervals over pooled seed–battle outcomes.
Figure 10: Per-category reward-model agreement (%) with App.tsx input, averaged over five seeds. Error bars show 95% Wilson intervals over pooled seed–battle outcomes.
| Parameter | Value |
| Backbone | Qwen3.5-9B |
| Objective | Pairwise Bradley–Terry loss |
| Optimizer | Distributed Adam; , , |
| Learning rate | , constant; no warmup |
| Weight decay / gradient clipping | 0 / maximum gradient norm 50 |
| Epochs / global batch size | 1 / 32 preference pairs |
Table 3: Hyperparameters for the main preference reward models. The same optimization recipe is used across evidence conditions; the representation changes the source-code and visual inputs.
| Evidence | Full source ( ) | App.tsx ( )) | No source ( |
| Code | 10.05 | 0.43 | — |
| Code + screenshot | 13.07 | 0.49 | — |
| Heuristic trace | 57.71 | 2.26 | 1.42 |
| One-shot trace | 57.26 | 2.30 | 1.42 |
| Iterative trace | 59.93 | 2.42 | 1.53 |
Table 4: Candidate inputs at or above the 32,768-token ceiling in the main test predictions (%).
| Reward model | Seeds | Code | Code + Screenshot | Code + Iterative Interaction |
| Qwen3.5-4B | 5 | |||
| Qwen3.5-9B | 10 | |||
| Qwen3.5-27B | 10 |
Table 5: Agreement (%) on 3,327 gaming battles for three reward-model sizes. All rows use the same evaluation IDs. Video includes action and reasoning text in this table. Error terms are seed standard deviations.
| Evidence | Agreement (%) |
| Code | |
| Code + trace text | |
| Code + video, without trace text | |
| Code + video + trace text |
Table 6: Trace-text ablation on 3,327 gaming battles with ten seeds per condition. “Trace text” comprises actions and intermediate reasoning. Code-only and video-plus-text entries also appear in Table 5 .
Figure 11: Image fidelity and trace length on 3,327 gaming battles. Left: agreement versus nominal visual token budget when varying screenshot resolution or video resolution with all 26 frames retained. Right: agreement versus retained frame count at a 960-token cap per temporal block. Horizontal spacing in the right panel follows frame count; the bottom axis labels corresponding nominal visual token budgets. Video budgets are the per-block cap times for frames, so the 1- and 2-frame prefixes share a 960-token budget but occupy distinct positions. The 26-frame, 12,480-token endpoint is the same experiment in both panels. Points show means and error bars show one sample standard deviation across ten training seeds. Budgets are nominal visual-token caps, not measured total token usage or equal-total-compute conditions.
| Evaluation interaction policy | |||
| Training policy | Heuristic | One-shot | Iterative |
| Heuristic | 67.95 | 68.05 | 68.85 |
| One-shot | 67.72 | 69.03 | 68.67 |
| Iterative | 67.29 | 68.39 | 70.07 |
Table 7: Reward-model agreement (%) under changes in the interaction policy. Rows indicate the training policy; columns indicate the policy used to collect evaluation traces. Evaluation counts are 6,858 for heuristic, 6,852 for one-shot, and 6,855 for iterative traces.
Figure 12: We train on heuristic computer-use and then evaluate on preferences using iterative computer-use. Across six out of seven categories, using iterative computer-use at inference time improves performance.
| Reward model | Seeds | Mini recordings | Qwen recordings |
| Qwen3.5-4B | 5 | ||
| Qwen3.5-9B | 10 | ||
| Qwen3.5-27B | 10 |
Table 8: Agreement (%) of fixed code-plus-video reward models when changing the recording pipeline on 3,284 gaming battles with identical prompts and labels. Error terms are sample standard deviations across training seeds. Both columns use a 32,768-token context, a 26-frame cap, and the same source-code budget.
| Parameter | Value |
| Initial policy | Qwen3.5-9B |
| Training seed | 42 |
| Prompts per rollout batch | 384 |
| Samples per prompt | 8 (3,072 trajectories per batch) |
| Training sampling | Temperature 1; top- ; top- disabled |
| Reasoning mode | Thinking disabled |
Table 9: Policy-training hyperparameters for the code-only and standard interaction-only reward arms. The implementation uses group-normalized advantages and DAPO-style asymmetric clipping and length penalties; dynamic group filtering is disabled.
More in this topic
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 516 papers