From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs

Yangzhe Peng; Xiaoyang Wang; Yiyang Zhao; Lijun Wu; Kun He

cs.AISep 26, 2026

From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs

Authors: Yangzhe Peng, Xiaoyang Wang, Yiyang Zhao, Lijun Wu, Kun He

Organizations: Huazhong University of Science and Technology · Shanghai Artificial Intelligence Laboratory · Zhejiang University · Shanghai Innovation Institute

Abstract

Comparative Process Reward Models (PRMs) provide critical step-level guidance for autonomous web agents by evaluating state-conditioned preferences between candidate actions. However, existing preference training data synthesized via multi-policy sampling suffers from a severe scarcity of Grounded Minimal Contrastive Pairs (GMCPs)-where competing candidates target genuine on-page elements with identical action types. In representative baselines preference data (namely, WebArbiter), GMCPs account for merely 24.19%, biasing PRMs during training to rely on shallow shortcuts (such as element hallucinations and action type mismatches) rather than acquiring genuine contextual decision semantics. To address these challenges, we propose SURFPRM, a graph-guided process preference synthesis framework for comparative Web PRMs. SURFPRM structures web demonstrations into a persistent Interaction Element Graph that acts as an environment-grounded negative action proposal mechanism, systematically synthesizing contrastive negative actions across spatial, temporal, and spatiotemporal confusion axes. This elevates the GMCP proportion from 24.19% to 74.60%, producing the curated SURFPRM-DATA dataset. Across six open-source backbones (3B to 9B parameters), PRMs trained on SURFPRM-DATA outperform baseline-trained models on average on WEBPRMBENCH and rival leading proprietary LLMs. In downstream reward-guided trajectory search on WEBARENA-LITE, SURFPRM provides step-level guidance for both GPT-4o (+14.21%) and GPT-4o-mini (+12.83%) policies, yielding substantial improvements in complex web task success rates.

Figures & tables

Figure 1: Overview of the surfPRM Framework. Demonstration trajectories are aggregated into an Interaction Element Graph $G=(V,E_{s},E_{\tau})$ capturing spatial containment and temporal paths. The graph acts as a Negative Action Proposal Mechanism across spatial, temporal, and spatiotemporal axes. Teacher models filter false negatives and synthesize gold rationales and verdicts for comparative PRM training.

Figure 2: Negative Action Synthesis across Three Confusion Axes. Candidate negatives are proposed from the Interaction Element Graph along spatial, temporal, and spatiotemporal axes, targeting distinct confusion modes (formal definitions in Appendix A ).

Proposal Rule	Axis	Count
Textually Near, Graph-Distant	Spatial	5,022
Away from Goal	Spatiotemp.	4,020
Act Too Early	Temporal	304
Structurally Near	Spatial	223
Repeat Past	Temporal	166
Create a Cycle	Spatiotemp.	86

Table 1: Distribution of Proposal Rules in surfPRM-data (10k Pairs).

Closed-Source LLM
Model	WebArena		AssistantBench		WorkArena		Average
Model	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
GPT-5*	84.83	71.64	81.67	63.33	81.14	64.62	82.55	66.53
Claude-4.6-Sonnet	84.58	71.14	80.00	60.00	83.02	66.51	82.53	65.88
Gemini-3.7-Flash	83.71	68.66	83.33	73.33	78.77	64.15	81.94	68.71
Jev-1.13 (choice mode)	84.83	70.65	79.17	60.00	78.42	58.49	80.81	63.05

Table 2: Step-Level Process Preference Discrimination on WebPRMBench. Evaluated on Pairwise and BoN Accuracy (%). Average reports macro-average across the three environments.

*

denotes baseline results directly reported from the WebArbiter paper. Models fine-tuned on surfPRM-data are shaded; bold marks best within each group in Average.

Policy Model	WebPRM Guidance	Shopping	CMS	Reddit	GitLab	MAP	Avg.	$\bm{\Delta}$
GPT-4o-mini	w/o Trajectory Search*	21.74	22.86	19.05	34.38	19.35	23.48	–
	GPT-4o-mini*	24.44	22.86	26.32	33.33	15.38	24.47	+0.99
	WebShepherd-8B*	26.09	45.71	23.81	40.62	35.48	34.34	+10.87
	Qwen3.5-9B + WebArbiter 10k	34.78	38.89	33.33	38.24	19.35	32.92	+9.44
	Qwen3.5-9B + Ours 10k	36.96	30.56	50.00	38.24	25.81	36.31	+12.83
GPT-4o	w/o Trajectory Search*	23.91	31.43	28.57	56.25	19.35	31.90	–

Table 3: Downstream Trajectory Search on WebArena-Lite. Success rates (%) across 5 environments with Best-of-

N

search guided by Web PRMs. Avg. is macro-average;

\Delta

is absolute gain over unguided execution.

*

denotes baseline results directly reported from the WebArbiter paper. Rows trained on surfPRM-data are shaded; bold marks best within each policy group.

Figure 5: Axis ablation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Value
Judged pairs	10,000
API calls	11,210
First-attempt pass	9,528 (95.3%)
Passed after retries	293
2nd / 3rd / 4th / 5th attempt	168 / 67 / 40 / 18
Exhausted all 5 attempts (filtered)	179

Appendix

Table 4: Teacher Verification and Gold Supervision Synthesis Cost for the 10,000 Sampled Candidate Pairs.

Model / Training Configuration	WebArena		AssistantBench		WorkArena		Average
Model / Training Configuration	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
Qwen2.5-3B-Instruct	70.02	35.82	78.33	46.67	68.99	29.25	72.45	37.25
+ Only-Temporal	74.63	45.27	78.33	33.33	65.45	26.42	72.80	35.01
+ Only-Spatial	81.84	57.71	75.00	50.00	73.70	40.57	76.85	49.43
+ Only-Spatiotemporal	82.21	53.73	78.33	43.33	73.47	33.02	78.00	43.36
+ All Axes	81.59	55.22	85.00	56.67	75.35	41.98	80.65	51.29

Appendix

Table 5: Ablation on Graph Confusion Axes. Performance comparison of comparative Web PRMs fine-tuned on individual graph confusion axes versus the complete multi-axis mixture (evaluated on WebPRMBench with Qwen2.5-3B-Instruct). Pairwise accuracy and state-level Best-of-

N

(BoN) accuracy are reported in percent (%). The row corresponding to our full confusion mixture is shaded, and bold indicates the best result in the Average columns.

Model / SFT Data	GMCP Ratio	WebArena		AssistantBench		WorkArena		Average
Model / SFT Data	GMCP Ratio	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
Qwen2.5-3B-Instruct	–	70.02	35.82	78.33	46.67	68.99	29.25	72.45	37.25
+ WebArbiter 10k	24.19%	81.84	55.22	69.17	40.00	71.58	35.38	74.20	43.53
+ GMCP40 10k	40.00%	81.47	50.75	70.83	26.67	74.06	39.62	75.45	39.01
+ GMCP60 10k	60.00%	82.71	56.72	76.67	36.67	74.29	40.09	77.89	44.49
+ Ours 10k	74.60%	81.59	55.22	85.00	56.67	75.35	41.98	80.65	51.29

Appendix

Table 6: Effect of the Grounded Minimal Contrastive Pair (GMCP) Ratio. Qwen2.5-3B-Instruct results with approximately 10k SFT pairs per fine-tuned variant. Pairwise and state-level Best-of-

N

(BoN) accuracies are reported in percent (%). The best results in the Average column are in bold .

Model / Training Configuration	WebArena		AssistantBench		WorkArena		Average
Model / Training Configuration	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
Qwen2.5-3B-Instruct	70.02	35.82	78.33	46.67	68.99	29.25	72.45	37.25
+ WebArbiter 10k SFT	81.84	55.22	69.17	40.00	71.58	35.38	74.20	43.53
+ No-Graph 10k SFT	84.45	59.70	70.00	30.00	72.05	36.32	75.50	42.01
+ Ours 10k SFT	81.59	55.22	85.00	56.67	75.35	41.98	80.65	51.29

Appendix

Table 7: Ablation on Graph Necessity. Comparison of comparative Web PRMs trained with the WebArbiter baseline, the No-Graph variant, and surfPRM on WebPRMBench . Pairwise accuracy and state-level Best-of-

N

(BoN) accuracy are reported in percent (%). The best results in the Average columns are in bold .

Model	WebArena		AssistantBench		WorkArena		Average
Model	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN	Pairwise	BoN
Qwen3.5-9B	81.88 $\pm$ 0.50	68.32 $\pm$ 1.25	81.67 $\pm$ 0.84	66.67 $\pm$ 3.34	78.69 $\pm$ 0.68	54.72 $\pm$ 0.95	80.75 $\pm$ 0.21	63.24 $\pm$ 1.58
+ WebArbiter 10k SFT	85.28 $\pm$ 0.36	68.33 $\pm$ 0.29	89.72 $\pm$ 0.96	71.11 $\pm$ 1.92	77.48 $\pm$ 0.20	52.83 $\pm$ 0.81	84.16 $\pm$ 0.51	64.09 $\pm$ 1.01
+ Ours 10k SFT	87.31 $\pm$ 0.57	70.98 $\pm$ 1.04	87.78 $\pm$ 1.74	71.11 $\pm$ 3.85	83.57 $\pm$ 0.65	61.95 $\pm$ 2.76	86.22 $\pm$ 0.20	68.01 $\pm$ 0.24

Appendix

Table 8: Multiple Repeated Evaluation Results on WebPRMBench . Performance of comparative Web PRMs across repeated evaluation across 3 runs. Pairwise Accuracy (%) and State-level Best-of-

N

(BoN) Accuracy (%) are reported as

\text{mean}\pm\text{std}

. Rows fine-tuned on our synthesized data ( surfPRM-data ) are shaded, and bold indicates the best mean result in the Average columns.

Process Reward Models

Process reward models assess intermediate reasoning steps and actions to guide learning, search, and evaluation.

See all 137 papers

Feb 11, 2026

Next-token Preference: Scaling Unsupervised Reward Modeling

Learning from feedback advances the capabilities and safety of frontier models, but preference supervision remains costly and difficult to scale. We introduce next-token preference (NTP), an unsupervised reward-modeling method that constructs preference pairs from the continuation structure of raw web text. NTP uses each prefix's original continuation as the chosen response and nearby continuations as in-batch negatives, without candidate generation or human or model-generated preference annotations. Training on 11M tokens of math-focused web data yields steady gains on RewardBench and RewardBench 2, with gains across model families and sizes. Across mathematical and general-domain web corpora, our method improves overall RewardBench 2 accuracy by 7.6-8.3 percentage points over initialization under the same token budget. Training on math-focused text yields gains of up to +16.1 on in-domain math subsets and consistent improvements on out-of-domain safety and general subsets. These reward models improve best-of-N selection on math, code, safety, and instruction following, and support math policy optimization competitive with supervised reward models of similar size. NTP also improves the label efficiency of subsequent supervised fine-tuning. Using half the labeled pairs from a curated preference dataset, NTP initialization surpasses base initialization trained on the full dataset. These results suggest that web text contains transferable preference signals that can reduce dependence on costly preference annotations.

Jingxuan Fan, Yueying Li, Zhenting Qi +4

Harvard University · Cornell University · Microsoft Research +1

Reward Modeling Preference Learning Reward Model Evaluation

Ask PDF

Jun 8, 2026

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback. However, we identify a hidden bias in PRMs caused by severe imbalance in step-level training data. Standard cross-entropy training amplifies this bias, causing PRMs to overcredit plausible but incorrect steps and produce high false-positive rates. We show that these false positives have an asymmetric downstream effect: false negatives mainly slow exploration, whereas false positives actively steer Best-of-N selection, guided decoding, and policy optimization toward flawed reasoning. This suggests that PRM training should shift from pointwise label fitting to reliable relative comparisons. To address this, we propose PRISM (Precision Ranking for Improved Step Modeling), a policy-aware PRM training framework that learns from contrastive step-level comparisons and hard negatives generated by a temporal lookahead strategy, requiring no new human labels. We further use a difficulty-aware curriculum to optimize the contrastive step margin. Across PRMBench and ProcessBench, PRISM substantially reduces false positives (22% on PRMBench) and improves macro F1 over strong discriminative PRMs. When applied to policy optimization and search tasks, including guided decoding and Best-of-N selection, it consistently improves accuracy (up to 22% for guided decoding and 33% for Best-of-N) and robustness. More broadly, trustworthy process supervision is not just about assigning high rewards, but about rewarding the right reasoning for the right reasons.

Aakriti Agrawal, Souradip Chakraborty, Armin Saghafian +6

University of Maryland, College Park · Capital One · University of Central Florida

Process Reward Models RL for Language Model Reasoning Step-Level Credit Assignment in RL Step-Level Credit Assignment

Ask PDF

Jan 2, 2026

IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models

Generative Reward Models (GRMs) have demonstrated strong performance in reward modeling, due to their interpretability and potential for refinement through reinforcement learning (RL). However, widely used pairwise GRMs create a computational bottleneck in reinforcement learning from human feedback (RLHF), when calibrating or aggregating preference signals over n candidates, often incurring O(n^2) pairwise judgments. To address this issue, we propose Intergroup Relative Preference Modeling (IRPM), an RL-based method that extends the Bradley-Terry preference-learning paradigm via intergroup comparisons to train pointwise GRMs from pairwise preference data. IRPM derives a pointwise reward for each response by contrasting groups of chosen vs. rejected samples, enabling pointwise scores comparable across candidate sets and O(n) reward evaluation for a variable number of candidates during RL training, while preserving interpretability and scalability. Experiments show that IRPM achieves state-of-the-art performance among pointwise GRMs on JudgeBench and RewardBench, and approaches the performance of leading pairwise GRMs. In addition, IRPM achieves substantial gains in post-training evaluations, demonstrating its effectiveness. Our models and code are available at https://github.com/XINGYUN-AI-LAB/IRPM.

Haonan Song, Qingchen Xie, Huan Zhu +12

HUJING Digital Media & Entertainment Group (XingYun Lab), Beijing, China · Department of Automation, Tsinghua University, Beijing, China · Beihang University, Beijing, China +2

Reward Modeling Preference Modeling Pairwise Preference Learning

Ask PDF

Abstract

Process Reward Models

Explore similar work

Next-token Preference: Scaling Unsupervised Reward Modeling

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models