Empirical evaluation is central to recommender-systems (RecSys) research, but turning experimental designs into executable code remains a manual and error-prone task. We present AutoRecLab, a Python-based autonomous RecSys lab that automates RecSys experiments from natural-language prompts. Given a research idea, AutoRecLab derives explicit experiment requirements, builds and validates a prototype, and iteratively expands it into the requested full experiment. The workflow combines retrieval-augmented generation (RAG) for documentation lookup, static type verification, and execution-steered tree search. In our demonstration, AutoRecLab autonomously implements an explicit-to-implicit feedback conversion study. In a baseline comparison across six algorithms and three datasets, 8 of 9 runs succeed at an average cost of approx- imately $1 per run with GPT-5.4-mini.
Figures & tables
Figure 1. Plot generated by AutoRecLab for the explicit-to-implicit conversion experiment. The panels report NDCG@10 (top) and Precision@10 (bottom) on MovieLens 1M for different algorithms and rating thresholds: greater than or equal to (ge) 1, greater than 3, and greater than 4. Two vertically stacked grouped bar charts, each with three bars per feedback-conversion condition. For NDCG@10, ItemKNN is highest when ratings greater than or equal to 1 are converted to implicit feedback, whereas ImplicitMF is highest for thresholds greater than 3 and greater than 4. For Precision@10, ItemKNN is highest for ratings greater than or equal to 1 and greater than 3, whereas ImplicitMF is highest for ratings greater than 4. The popularity baseline is lowest in every condition, and all three algorithms reach their lowest values when only ratings greater than 4 are converted.
Run Number
Metric
1.1
1.2
1.3
2.1
2.2
2.3
3.1
3.2
3.3
Dataset
MU
MU
MU
ML1M
ML1M
ML1M
VI
VI
VI
Algos
6
6
6
6
6
6
6
6
6
Cost ($)
1.09
1.08
1.10
0.95
0.98
0.91
1.04
1.01
0.90
Runtime P
4.37h
12.23h
40m
23m
58m
1.52h
17.32h
7m
19m
Runtime F
30.2h
33.65h
16.23h
32.33h
32.08h
22.43h
5m
24.48h
16.58h
Table 1. Statistics for the AutoRecLab baseline experiment across nine runs (P: Prototype, F: Final Refinement)
Figure 2. AutoRecLab workflow. Requirements are derived from the research task. During prototyping, tree search and MCP-based documentation retrieval produce a working prototype. The refinement loop expands this prototype to the full experiment and returns the experimental summary, code, and plots. A block diagram of the AutoRecLab workflow. A user provides a Research Prompt, which enters Requirements Engineering and is separated into Full and Prototype Requirements. Prototype Requirements enter the Prototyping Stage, where code is iterated with the OmniRec API, Tree Search, and MCP-assisted documentation retrieval. The resulting Prototype enters the Refinement Stage and is combined with the Full Requirements in a Final Refinement loop that returns a summary, code, and plots to the user.
Auto-research agents have shown the potential to automate hypothesis generation, experiment execution, and iterative refinement. However, scaling this paradigm to industry-scale recommendation models introduces two challenges: (1) long feedback loops, where model training can take days, making serial iteration prohibitively slow and requiring parallel exploration across multiple research directions; and (2) system complexity, where large configurations, fragile infrastructure dependencies, and multi-day GPU jobs require robust and recoverable execution. We present Auto-RecSys, an autonomous research system for long-horizon experimentation on industry-scale recommendation models. Auto-RecSys addresses these challenges through three harness designs: (1) distributed asynchronous execution for running multiple experiments in parallel across servers, (2) centralized cross-server memory for persistent and recoverable execution across sessions and failures, and (3) cognitive-procedural separation, where natural-language skill files guide LLM reasoning while deterministic scripts enforce operational correctness. Auto-RecSys further employs a dual-loop self-evolving architecture: an Execution Evolution Loop in which model-specific playbooks accumulate operational knowledge by recording failed attempts and crystallizing successful pipelines, and an Idea Evolution Loop in which experimental outcomes inform subsequent ideation. Evaluated on recommendation models, Auto-RecSys significantly reduces the human time required per experiment cycle and improves execution reliability as its playbooks mature.
Optimizing modern recommender models still depends heavily on engineers manually iterating over architectural, objective, and training-strategy changes. While LLM-based agents can automate this trial-and-error process, allowing the LLM to both select modification directions and generate concrete hypotheses often leads to unstable search under limited experiment budgets. Inspired by the above challenge, we propose RecHarness, a Bandit-Routed Agentic Harness for automated recommender model optimization. RecHarness separates the optimization process into two steps: a bandit router selects the next modification direction according to historical validation feedback, while the LLM generates a concrete optimization hypothesis and executable code edit within the selected direction. To sustain long-horizon exploration, RecHarness uses a jump-basin mechanism to activate a structural-jump arm when local edits stagnate. Across multiple recommendation tasks, datasets, and model backbones, RecHarness achieves more stable performance improvements and uses limited trial budgets more effectively than LLM-reasoning search. During a 7-day online A/B test on a large-scale short-video advertising platform, the selected candidate improves ADVV by 2.084%, Revenue by 0.534%, and Exposure by 0.559%. Code is available at https://github.com/6lyc/RecHarness.
Haoran Ling, Yuecheng Li, Zeyu Song +5
Georgia Institute of Technology · Kuaishou Technology, China
Improving an industrial recommender is an iterative research-and-engineering process rather than a direct path from idea to deployment. In DASHEN, NetEase's gaming-community app, algorithm engineers typically identify promising directions from research papers, technical reports, and prior production experiments; reproduce or adapt the underlying methods; implement them in the production codebase; and evaluate the resulting models through training and offline experiments. Promising candidates are then advanced to online A/B tests, and those demonstrating robust gains are submitted to Launch Review---the internal gate for full-traffic rollout. Large language models (LLMs) can assist with individual stages of this workflow, but the overall process remains human-dependent without a harness that can reliably coordinate them across long-running, often multi-day experimental cycles. We present AutoLR, initially built as Auto Launch Review and later extended upstream into an autonomous research-to-launch harness. AutoLR combines three system mechanisms: a multi-expert council that debates and adversarially reviews proposals; a deterministic evidence-weighted exploration--exploitation selector that allocates a limited trial budget across candidate directions and uses Council reranking; and a layered knowledge system that combines external research, production-system knowledge, and DASHEN-specific domain knowledge---such as game communities, player characteristics, and content-interaction patterns---with posterior evidence from configurations, patches, logs, failures, and offline outcomes. LLM agents perform semantic reasoning and code generation, while deterministic controllers retain authority over execution, metric extraction, guardrails, and persistent state transitions.