cs.CLSep 19, 2026

Automatic multimodal UX improvement recommendations from LLM agent user simulations

Authors: Anu Chowdhury, Bin Wu, Hossein A. Rahmani, Emine Yilmaz

Organizations: Centre for Artificial Intelligence, University College London

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Evaluating user experience (UX) on live websites through user testing is expensive, subjective, and difficult to scale. LLM agents offer a promising route to automating UX testing by simulating realistic user behaviour. However, existing simulation approaches typically lack multimodality and require time-consuming manual review to extract actionable insights. We formalise UX improvement recommendation from simulation data as a structured natural language generation and ranking problem, and establish an evaluation protocol using expert annotation and LLM-as-a-Judge. We present AMUSER, a multimodal framework which simulates user behaviour and automatically generates prioritised UX improvement recommendations from resulting data. We evaluate AMUSER on commercial websites and show that its recommendations substantially outperform those from text-only simulation (NDCG@3 = 0.758 versus 0.359) at an 89% lower simulation cost. Our results suggest an asymmetric role of multimodality: visual access during simulation improves recommendations through richer traces, while providing visual inputs during recommendation generation can modestly degrade quality. We also discuss practical deployment lessons from applying AMUSER to commercial websites.

Figures & tables

Figure 1: Selected steps from an AMUSER simulation (young professional persona, searching for size 12 cotton trousers on sale). The agent navigates the site, producing thoughts and emotion labels at each step; the resulting trace DD is converted into prioritised recommendations RR (abridged recommendations shown for brevity).
Figure 2: Overview of the AMUSER framework. A persona analysis module conditions a simulation agent, which produces multimodal traces; a recommendation module generates prioritised product, UI, and UX recommendations.
Figure 3: Performance of baselines for medium and high priority recommendations by category relative to AMUSER. AMUSER outperforms across all categories.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Simulation MethodNDCG@3NDCGAllProductUIUX
Homepage HTML-only0.4020.5150.2690.3330.1730.299
UXAgent0.5030.6280.3950.3670.3980.419
AMUSER (ours)0.8460.9220.6390.7930.4470.682
Appendix
Table 3: GPT-5.5 robustness check on main results. Findings closely match the original GPT-5.4 results.
AblationNDCG@3NDCGAllProductUIUX
AMUSER0.8780.9370.6630.8120.5000.676
w/o images+0.052+0.026+0.047+0.061+0.060+0.022
w/o actions & thoughts−- 0.039−- 0.010−- 0.053+0.037−- 0.090−- 0.083
Appendix
Table 4: GPT-5.5 robustness check on a 33% sample of ablations. Deltas shown relative to AMUSER.
Figure 4: An example of an AMUSER recommendation implemented on a retail category page. The “Show numeric sizes alongside letters” recommendation surfaced in Figure 1 was implemented via JavaScript injection (due to lack of access to the site’s codebase) using Claude Code with Opus 4.8 in a single shot. This provides a concrete example that AMUSER’s prioritised recommendations RR can be actioned by site owners. Full recommendation title and description in Appendix C.3.1 .
AMUSERAMUSER w/o imagesUXAgent (non-multimodal)
Compact card layout*Compact card layout*Maintain current clarity
Emphasise distance visually*Maintain map-list sync*Enhance visual hierarchy
Appendix
Table 5: Illustrative (abridged) UI recommendations for a “find a nearby bank branch” simulation where multimodality was necessary in friction identification. Starred (*) recommendations require multimodal simulation to identify. In its recommendation description, UXAgent claims distance is ’prominently shown’; however, AMUSER’s trace shows it was below the fold. Text-only simulation has no viewport to detect this but the judge correctly flags UXAgent’s recommendation as incorrect. Recall that in the "w/o images" treatment, simulations are generated with multimodality but these images are not provided to the recommendation module at generation time. The user actions and frustrated thoughts captured by multimodal simulation persist in the text trace which explains why the recommendation module captures these frictions, without being shown image data.
Figure 5: Mean usefulness of recommendations as a function of simulation length (total steps), fit with a linear regression per category. Longer simulations are associated with higher-rated recommendations across all treatments and categories.
More in this topic

User Simulation

User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.

See all 127 papers

Explore similar work

CardsList

  1. Avenir-UX: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding

    Feb 25, 2026Wee Joe Tan, Zi Rui Lucas Lim, Shashank Durgad +2Computer-Use AgentsWeb Agents

    University College London London, United Kingdom Work done at UCL Nexus Labs.

  2. Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

    Jun 11, 2026Ruichao Mao, Zhou Fang, Teng Guo +9Multimodal ReasoningMultimodal Model Evaluation

    Ant Group

  3. UXBench: Measuring the Actionability of LLM-Generated UX Critiques

    Jun 15, 2026Wenjie Wang, Yue Huang, Zipeng Ling +11LLM-as-a-JudgeLLM Agent Evaluation

    University of Notre Dame · University of Pennsylvania · University of Rochester +4