From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs

Yangzhe Peng; Xiaoyang Wang; Yiyang Zhao; Lijun Wu; Kun He

cs.AISep 26, 2026

From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs

Authors: Yangzhe Peng, Xiaoyang Wang, Yiyang Zhao, Lijun Wu, Kun He

Organizations: Huazhong University of Science and Technology · Shanghai Artificial Intelligence Laboratory · Zhejiang University · Shanghai Innovation Institute

Abstract

Comparative Process Reward Models (PRMs) provide critical step-level guidance for autonomous web agents by evaluating state-conditioned preferences between candidate actions. However, existing preference training data synthesized via multi-policy sampling suffers from a severe scarcity of Grounded Minimal Contrastive Pairs (GMCPs)-where competing candidates target genuine on-page elements with identical action types. In representative baselines preference data (namely, WebArbiter), GMCPs account for merely 24.19%, biasing PRMs during training to rely on shallow shortcuts (such as element hallucinations and action type mismatches) rather than acquiring genuine contextual decision semantics. To address these challenges, we propose SURFPRM, a graph-guided process preference synthesis framework for comparative Web PRMs. SURFPRM structures web demonstrations into a persistent Interaction Element Graph that acts as an environment-grounded negative action proposal mechanism, systematically synthesizing contrastive negative actions across spatial, temporal, and spatiotemporal confusion axes. This elevates the GMCP proportion from 24.19% to 74.60%, producing the curated SURFPRM-DATA dataset. Across six open-source backbones (3B to 9B parameters), PRMs trained on SURFPRM-DATA outperform baseline-trained models on average on WEBPRMBENCH and rival leading proprietary LLMs. In downstream reward-guided trajectory search on WEBARENA-LITE, SURFPRM provides step-level guidance for both GPT-4o (+14.21%) and GPT-4o-mini (+12.83%) policies, yielding substantial improvements in complex web task success rates.

Figures & tables

Figure 1: Overview of the surfPRM Framework. Demonstration trajectories are aggregated into an Interaction Element Graph $G=(V,E_{s},E_{\tau})$ capturing spatial containment and temporal paths. The graph acts as a Negative Action Proposal Mechanism across spatial, temporal, and spatiotemporal axes. Teacher models filter false negatives and synthesize gold rationales and verdicts for comparative PRM training.

Figure 2: Negative Action Synthesis across Three Confusion Axes. Candidate negatives are proposed from the Interaction Element Graph along spatial, temporal, and spatiotemporal axes, targeting distinct confusion modes (formal definitions in Appendix A ).

Proposal Rule	Axis	Count
Textually Near, Graph-Distant	Spatial	5,022
Away from Goal	Spatiotemp.	4,020
Act Too Early	Temporal	304
Structurally Near	Spatial	223
Repeat Past	Temporal	166
Create a Cycle	Spatiotemp.	86

Table 1: Distribution of Proposal Rules in surfPRM-data (10k Pairs).

Closed-Source LLM
Model	WebArena		AssistantBench		WorkArena		Average
Model	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
GPT-5*	84.83	71.64	81.67	63.33	81.14	64.62	82.55	66.53
Claude-4.6-Sonnet	84.58	71.14	80.00	60.00	83.02	66.51	82.53	65.88
Gemini-3.7-Flash	83.71	68.66	83.33	73.33	78.77	64.15	81.94	68.71
Jev-1.13 (choice mode)	84.83	70.65	79.17	60.00	78.42	58.49	80.81	63.05

Table 2: Step-Level Process Preference Discrimination on WebPRMBench. Evaluated on Pairwise and BoN Accuracy (%). Average reports macro-average across the three environments.

*

denotes baseline results directly reported from the WebArbiter paper. Models fine-tuned on surfPRM-data are shaded; bold marks best within each group in Average.

Policy Model	WebPRM Guidance	Shopping	CMS	Reddit	GitLab	MAP	Avg.	$\bm{\Delta}$
GPT-4o-mini	w/o Trajectory Search*	21.74	22.86	19.05	34.38	19.35	23.48	–
	GPT-4o-mini*	24.44	22.86	26.32	33.33	15.38	24.47	+0.99
	WebShepherd-8B*	26.09	45.71	23.81	40.62	35.48	34.34	+10.87
	Qwen3.5-9B + WebArbiter 10k	34.78	38.89	33.33	38.24	19.35	32.92	+9.44
	Qwen3.5-9B + Ours 10k	36.96	30.56	50.00	38.24	25.81	36.31	+12.83
GPT-4o	w/o Trajectory Search*	23.91	31.43	28.57	56.25	19.35	31.90	–

Table 3: Downstream Trajectory Search on WebArena-Lite. Success rates (%) across 5 environments with Best-of-

N

search guided by Web PRMs. Avg. is macro-average;

\Delta

is absolute gain over unguided execution.

*

denotes baseline results directly reported from the WebArbiter paper. Rows trained on surfPRM-data are shaded; bold marks best within each policy group.

Figure 5: Axis ablation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Value
Judged pairs	10,000
API calls	11,210
First-attempt pass	9,528 (95.3%)
Passed after retries	293
2nd / 3rd / 4th / 5th attempt	168 / 67 / 40 / 18
Exhausted all 5 attempts (filtered)	179

Appendix

Table 4: Teacher Verification and Gold Supervision Synthesis Cost for the 10,000 Sampled Candidate Pairs.

Model / Training Configuration	WebArena		AssistantBench		WorkArena		Average
Model / Training Configuration	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
Qwen2.5-3B-Instruct	70.02	35.82	78.33	46.67	68.99	29.25	72.45	37.25
+ Only-Temporal	74.63	45.27	78.33	33.33	65.45	26.42	72.80	35.01
+ Only-Spatial	81.84	57.71	75.00	50.00	73.70	40.57	76.85	49.43
+ Only-Spatiotemporal	82.21	53.73	78.33	43.33	73.47	33.02	78.00	43.36
+ All Axes	81.59	55.22	85.00	56.67	75.35	41.98	80.65	51.29

Appendix

Table 5: Ablation on Graph Confusion Axes. Performance comparison of comparative Web PRMs fine-tuned on individual graph confusion axes versus the complete multi-axis mixture (evaluated on WebPRMBench with Qwen2.5-3B-Instruct). Pairwise accuracy and state-level Best-of-

N

(BoN) accuracy are reported in percent (%). The row corresponding to our full confusion mixture is shaded, and bold indicates the best result in the Average columns.

Model / SFT Data	GMCP Ratio	WebArena		AssistantBench		WorkArena		Average
Model / SFT Data	GMCP Ratio	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
Qwen2.5-3B-Instruct	–	70.02	35.82	78.33	46.67	68.99	29.25	72.45	37.25
+ WebArbiter 10k	24.19%	81.84	55.22	69.17	40.00	71.58	35.38	74.20	43.53
+ GMCP40 10k	40.00%	81.47	50.75	70.83	26.67	74.06	39.62	75.45	39.01
+ GMCP60 10k	60.00%	82.71	56.72	76.67	36.67	74.29	40.09	77.89	44.49
+ Ours 10k	74.60%	81.59	55.22	85.00	56.67	75.35	41.98	80.65	51.29

Appendix

Table 6: Effect of the Grounded Minimal Contrastive Pair (GMCP) Ratio. Qwen2.5-3B-Instruct results with approximately 10k SFT pairs per fine-tuned variant. Pairwise and state-level Best-of-

N

(BoN) accuracies are reported in percent (%). The best results in the Average column are in bold .

Model / Training Configuration	WebArena		AssistantBench		WorkArena		Average
Model / Training Configuration	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
Qwen2.5-3B-Instruct	70.02	35.82	78.33	46.67	68.99	29.25	72.45	37.25
+ WebArbiter 10k SFT	81.84	55.22	69.17	40.00	71.58	35.38	74.20	43.53
+ No-Graph 10k SFT	84.45	59.70	70.00	30.00	72.05	36.32	75.50	42.01
+ Ours 10k SFT	81.59	55.22	85.00	56.67	75.35	41.98	80.65	51.29

Appendix

Table 7: Ablation on Graph Necessity. Comparison of comparative Web PRMs trained with the WebArbiter baseline, the No-Graph variant, and surfPRM on WebPRMBench . Pairwise accuracy and state-level Best-of-

N

(BoN) accuracy are reported in percent (%). The best results in the Average columns are in bold .

Model	WebArena		AssistantBench		WorkArena		Average
Model	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
Qwen3.5-9B	81.88 $\pm$ 0.50	68.32 $\pm$ 1.25	81.67 $\pm$ 0.84	66.67 $\pm$ 3.34	78.69 $\pm$ 0.68	54.72 $\pm$ 0.95	80.75 $\pm$ 0.21	63.24 $\pm$ 1.58
+ WebArbiter 10k SFT	85.28 $\pm$ 0.36	68.33 $\pm$ 0.29	89.72 $\pm$ 0.96	71.11 $\pm$ 1.92	77.48 $\pm$ 0.20	52.83 $\pm$ 0.81	84.16 $\pm$ 0.51	64.09 $\pm$ 1.01
+ Ours 10k SFT	87.31 $\pm$ 0.57	70.98 $\pm$ 1.04	87.78 $\pm$ 1.74	71.11 $\pm$ 3.85	83.57 $\pm$ 0.65	61.95 $\pm$ 2.76	86.22 $\pm$ 0.20	68.01 $\pm$ 0.24

Appendix

Table 8: Multiple Repeated Evaluation Results on WebPRMBench . Performance of comparative Web PRMs across repeated evaluation across 3 runs. Pairwise Accuracy (%) and State-level Best-of-

N

(BoN) Accuracy (%) are reported as

\text{mean}\pm\text{std}

. Rows fine-tuned on our synthesized data ( surfPRM-data ) are shaded, and bold indicates the best mean result in the Average columns.

Process Reward Models

Process reward models assess intermediate reasoning steps and actions to guide learning, search, and evaluation.

See all 137 papers

Next-token Preference: Scaling Unsupervised Reward Modeling
Feb 11, 2026Jingxuan Fan, Yueying Li, Zhenting Qi +4Reward Modeling Preference Learning
Harvard University · Cornell University · Microsoft Research +1
The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
Jun 8, 2026Aakriti Agrawal, Souradip Chakraborty, Armin Saghafian +6Process Reward Models RL for Language Model Reasoning
University of Maryland, College Park · Capital One · University of Central Florida
IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models
Jan 2, 2026Haonan Song, Qingchen Xie, Huan Zhu +12Reward Modeling Preference Modeling
HUJING Digital Media & Entertainment Group (XingYun Lab), Beijing, China · Department of Automation, Tsinghua University, Beijing, China · Beihang University, Beijing, China +2

Feb 11, 2026J/K move · Enter open · S save

Next-token Preference: Scaling Unsupervised Reward Modeling

Jingxuan Fan, Yueying Li, Zhenting Qi, Dinghuai Zhang+3

Harvard University · Cornell University · Microsoft Research+1

Reward Modeling Preference Learning Reward Model Evaluation

Open paper AskPDF arXiv

Learning from feedback advances the capabilities and safety of frontier models, but preference supervision remains costly and difficult to scale. We introduce next-token preference (NTP), an unsupervised reward-modeling method that constructs preference pairs from the continuation structure of raw web text. NTP uses each prefix's original continuation as the chosen response and nearby continuations as in-batch negatives, without candidate generation or human or model-generated preference annotations. Training on 11M tokens of math-focused web data yields steady gains on RewardBench and RewardBench 2, with gains across model families and sizes. Across mathematical and general-domain web corpora, our method improves overall RewardBench 2 accuracy by 7.6-8.3 percentage points over initialization under the same token budget. Training on math-focused text yields gains of up to +16.1 on in-domain math subsets and consistent improvements on out-of-domain safety and general subsets. These reward models improve best-of-N selection on math, code, safety, and instruction following, and support math policy optimization competitive with supervised reward models of similar size. NTP also improves the label efficiency of subsequent supervised fine-tuning. Using half the labeled pairs from a curated preference dataset, NTP initialization surpasses base initialization trained on the full dataset. These results suggest that web text contains transferable preference signals that can reduce dependence on costly preference annotations.

Abstract

Process Reward Models

Explore similar work

Next-token Preference: Scaling Unsupervised Reward Modeling

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models