PEAR: Progressive Evidence-Based AutoResearch for Industrial Search Systems

Yifan Wang; Shipeng Zhu; Fei Xiong; Yuqin Yang; Yonghui Huang; Kunyao Wu; Yue Wang; Weichao Meng

cs.IRSep 28, 2026

PEAR: Progressive Evidence-Based AutoResearch for Industrial Search Systems

Authors: Yifan Wang, Shipeng Zhu, Fei Xiong, Yuqin Yang, Yonghui Huang, Kunyao Wu, Yue Wang, Weichao Meng, +1 more

Organizations: Global E-Commerce Agentic Search Team

Abstract

AutoResearch improves systems through iterative experimentation: agents propose candidate modifications, evaluate them, and use the results to guide subsequent exploration. Applying this paradigm to industrial search presents two challenges. (1) Common AutoResearch approaches follow a keep-if-better rule, retaining the highest-scoring candidate for subsequent experiments. Under non-stationary traffic, transient gains may be mistaken for persistent improvements, impairing reliable accumulation of search knowledge. (2) Candidate modifications can be evaluated at multiple fidelity levels, from low-cost proxies to online validation, differing in cost, objective alignment, and statistical reliability. Existing methods rely on individual signals or task-specific procedures, lacking a unified basis for using evidence across levels to guide search. We introduce Progressive Evidence-Based AutoResearch (PEAR) with two complementary components. Evidence-driven AutoResearch maintains an independent, hypothesis-guided research state for each strategy task within a predefined objective and intervention scope. Each state evolves through a Plan-Execute-Evaluate-Update transition that links experimentation to context-aware evidence interpretation and hypothesis revision. Confidence-Gated Verifier Ladder organizes evaluation into four levels of increasing fidelity: Offline Replay, Shadow-Traffic Evaluation, Rapid Online Evaluation, and Decision-Grade Online Evaluation. A unified confidence-based gate promotes candidates only when evidence supports a statistically significant positive effect, enabling broad low-cost exploration while reserving costly online experiments for promoted candidates. In a real-world industrial search system, strategies optimized with PEAR significantly increased Main Order/DAU by 2.7336% and 3.2957% relative to their respective baselines in two A/B experiments.

Figures & tables

Figure 1 : Overview of PEAR. Evidence-driven AutoResearch maintains parallel strategy-level research states, while the Confidence-Gated Verifier Ladder progressively evaluates candidates, governs their promotion using evidence of increasing fidelity, and returns evaluation results to the corresponding research states.

Task phase	Experimental rounds	Groups	Unique configs.	Promoted from L2	Promotion rate
A	19	133	20	2	1.50%
B–I	12	84	23	5	5.95%
B–II	2	14	6	1	7.14%
B–III	21	147	53	2	1.36%
C	16	76	54	3	3.95%

Table 1 : L2 search scale and candidate selection. Unique configurations are deduplicated within each phase; promotion rate is the number of candidates promoted from L2 divided by the number of experimental groups.

Experimental round / research question	Controlled interventions	Evidence-driven update
1. Which constraint axis accounts for the observed gain?	Apply one-axis changes to global and format-specific constraints; also test the coupled global change $G=(3,2)$ .	Only $G=(3,2)$ yields a significant positive order signal. The effects of increasing $w$ , increasing $q$ , and experimental variation remain entangled.
2. Can $G=(3,2)$ be reproduced, and which variable explains its gain?	Replicate $G=(3,2)$ ; compare window-only $G=(3,1)$ , capacity-only $G=(2,2)$ , and boundary settings including $G=(3,3)$ .	$G=(3,3)$ produces a stronger signal. Replications of $G=(3,2)$ yield mixed directional effects and indicate potential degradation in the exit metric. The next experimental round prioritizes capacity and replication.
3. Is the gain driven by the coupled change or by capacity alone?	Replicate $G=(3,3)$ and $G=(3,2)$ ; test capacity-only $G=(2,2)$ and wider-window boundary settings.	$G=(2,2)$ achieves the strongest observed result. The revised hypothesis attributes the gain primarily to capacity relaxation without requiring a wider window.
4. Are the marginal effects of window and capacity stable?	Concentrate replications on $G=(2,2)$ , $G=(3,2)$ , and $G=(3,3)$ ; retain window-only $G=(3,1)$ as a counterfactual.	$G=(3,3)$ remains directionally favorable on order and exit signals but is not significant. The capacity-centered hypothesis is retained for further replication.

Table 2 : Representative hypothesis-revision trajectory for Task A. The table shows four early experimental rounds from a 19-round search;

G=(w,q)

denotes the global window and capacity constraint.

Phase	Experimental rounds	Groups	Unique configs.	Promoted from L2	L4 outcome
I	12	84	23	5	No significant gain
II	2	14	6	1	Positive, not significant
III	21	147	53	2	Significant order gain

Table 3 : Outcomes from Decision-Grade Online Evaluation across search phases for Task B.

Figure 2 : Hypothesis-driven search trajectory for Task B. Candidate evaluations update a retained hypothesis state through validation, rejection, and branch termination until Decision-Grade Online Evaluation.

	Click	Order	GMV
Signed relative error	$-16.02\%$	$\mathbf{-3.49\%}$	$-18.23\%$

Table 4 : Signed relative errors between the aggregate means of proxy predictions and observed online outcomes.

Candidate	L2: Shadow-Traffic Evaluation	L3: Rapid Online Evaluation	L4: Decision-Grade Online Evaluation
A	Order proxy $+45.88\%$	Observed order $+15.48\%$	Main Order/DAU $+2.73\%$
B	Order proxy $+61.64\%$	Observed order $+29.76\%$	Main Order/DAU $+2.16\%$

Table 5 : Candidate effects across evaluation stages for two Task A candidates. L2 reports changes in the order proxy, L3 reports changes in observed orders, and L4 reports changes in Main Order/DAU.

Task	SearchPV/DAU	ASN	GMV/DAU	Main Order/DAU	SKU Order/DAU	PayPV/PV	Main OPMS	SKU OPMS
A	$+0.0063\%$	$\mathbf{+1.1512\%}$	$+2.2136\%$	$\mathbf{+2.7336\%}$	$\mathbf{+2.7259\%}$	$+1.0441\%$	$\mathbf{+2.7407\%}$	$\mathbf{+2.7375\%}$
B	$+0.0470\%$	$\mathbf{+1.3841\%}$	$+1.6633\%$	$\mathbf{+3.2957\%}$	$\mathbf{+2.9826\%}$	$\mathbf{+1.6298\%}$	$\mathbf{+3.2595\%}$	$\mathbf{+2.9448\%}$

Table 6 : Relative metric changes in online A/B evaluations of strategies optimized with PEAR for two search optimization tasks. All values are relative to the corresponding baseline; dark-green boldface indicates statistical significance at

p<0.05

Explore similar work

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer
Jul 19, 2026Jingjie Ning, Xiaochuan Li, Shanshan Zhong +2Research Automation Auto Research
School of Computer Science, Carnegie Mellon University · 2DP Technology
GEAR: Genetic AutoResearch for Agentic Code Evolution
May 8, 2026Ahmadreza Jeddi, Minh Ngoc Le, Hakki C. Karaimer +2
University of Toronto · Vector Institute · AI Center-Toronto, Samsung Electronics +1
Efficiency Matters in Autonomous Research
Jul 27, 2026Haiqian Yang, Yuan CaoResearch Automation Search Algorithms

Jul 19, 2026, cs.MAJ/K move · Enter open · S save

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

Jingjie Ning, Xiaochuan Li, Shanshan Zhong, Ji Zeng+1

School of Computer Science, Carnegie Mellon University · 2DP Technology

Research Automation Auto Research Reusable Knowledge Feedback Materials

Open paper AskPDF arXiv

Auto Research uses language-model agents to propose, implement, and evaluate machine-learning changes in a closed loop, but is usually judged by its terminal pipeline. A terminal score cannot reveal which technical decision produced a gain or distinguish a reusable discovery from a change adapted to development feedback. We introduce intervention-centered Auto Research, which validates research decisions rather than only final artifacts and makes their reliability measurable. Feature, Model, Representation, and Data axes are searched independently with inner five-fold feedback. Each axis winner is frozen before an outer-holdout matrix compares all alternatives on evidence the loop never sees. Across 701 agent-executed attempts spanning ten Matbench endpoints, outer evidence confirms the selected intervention on nine of ten endpoints and preserves 89.3% of non-tied intervention orderings. It also rejects an aggregate Representation gain that inner feedback endorsed. The resulting matrix reveals an information-dependent hierarchy. Composition-only tasks support several routes to improvement, whereas structure-informed tasks favor local geometry features and complementary tree ensembles. A subsequent compatibility test combines already frozen Feature and Model code without further search or tuning and raises mean outer-holdout improvement from 19.0% to 26.3%. By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.

Abstract

Explore similar work

Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer

GEAR: Genetic AutoResearch for Agentic Code Evolution

Efficiency Matters in Autonomous Research