In scientific reasoning tasks, the veracity of the reasoning process is as critical as the final outcome. While Process Reward Models (PRMs) offer a solution to the coarse-grained supervision problems inherent in Outcome Reward Models (ORMs), their deployment is hindered by the prohibitive cost of obtaining expert-verified step-wise labels. This paper addresses the challenge of training reliable PRMs using abundant but noisy "weak" supervision. We argue that existing Weak-to-Strong Generalization (W2SG) theories lack prescriptive guidelines for selecting high-quality training signals from noisy data. To bridge this gap, we introduce the Dual-Consensus Weak-to-Strong (DC-W2S) framework. By intersecting Self-Consensus (SC) metrics among weak supervisors with Neighborhood-Consensus (NC) metrics in the embedding space, we stratify supervision signals into distinct reliability regimes. We then employ a curriculum of instance-level balanced sampling and label-level reliability-aware masking to guide the training process. We demonstrate that DC-W2S enables the training of robust PRMs for complex reasoning without exhaustive expert annotation, proving that strategic data curation is more effective than indiscriminate training on large-scale noisy datasets.
Figures & tables
Figure 1: Dual-Consensus weak-to-strong supervision framework for efficient PRM training in biological reasoning, where expert step-level verification is costly or unavailable.
Figure 2: Performance comparison across four cell lines. BoN (Ours w/ Full Set (Multi-Label)) achieves the highest average F1 scores, outperforming single-label and SFT baselines by effectively aggregating multiple weak supervisory signals. Note: RPE1 is OOD for PRM; SFT is trained with RPE1.
Figure 3: Benchmarking Performance and Efficiency of BoN (Ours w/ CellProfiler). Top (instance-level) and bottom (label-level) efficiency analyses show that we can achieve comparable performance while using fewer training instances and fewer weak step labels, thereby suggesting the effectiveness of the DC-W2S.
Method
RPE1 (OOD)
DE
DoC
Avg
Only Query (Greedy)
19.65
34.84
27.25
Only Query (Majority Voting)
20.69
35.86
28.28
Only Query (BoN w/ Full Set)
53.17
51.94
52.56
SUMMER (Greedy)
37.52
68.85
53.19
SUMMER (Majority Voting)
38.88
72.31
55.60
Table 1: Performance comparison of different policy input mode.
Method
RPE1 (OOD)
DE
DoC
Avg
Greedy Decoding
78.33
34.89
56.61
Majority Voting
78.51
30.69
54.60
Coverage (Upper Bound)
79.18
91.99
85.59
Our PRM
BoN w/ Full Set (Multi-Label)
78.88
62.30
70.59
Table 2: Results of DC-W2S using RBIO1 policy model.
Method
Acc
Weighted F1
Greedy Decoding
80.69
87.41
Majority Voting
82.76
87.23
Coverage (Upper Bound)
95.86
97.43
Our PRM
BoN w/ Full Set (Multi-Label)
83.45
87.40
BoN w/ CellProfiler (100k)
84.48
88.89
Table 3: Results of DC-W2S on BioReason KEGG Task.
Figure 4: Effect of embedding choice. Performance on RPE1–DE for different label-pattern configurations across semantic and biologically grounded embedding spaces. Embeddings with richer biological structure (e.g., ESM, CellProfiler) yield larger gains from neighborhood-reliable supervision (P3).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
HepG2
Jurkat
K562
RPE1
Method
DE
DoC
DE
DoC
DE
DoC
DE
DoC
Avg
Greedy Decoding
40.05
71.15
41.12
59.91
38.47
83.47
37.52
68.85
55.07
Majority Voting
43.56
74.51
44.63
61.69
41.49
85.10
38.88
72.31
57.77
Baselines
BoN w/ Llama-PRM800K
47.27
71.27
48.71
58.81
44.48
82.84
45.20
69.91
58.56
BoN w/ Qwen-PRM800K
44.78
73.53
45.65
61.77
41.79
84.87
42.03
72.58
58.38
Appendix
Table 4: Baseline PRM comparison across all cell lines and tasks.
HepG2
Jurkat
K562
RPE1
Best-of-N (N=8)
DE
DoC
DE
DoC
DE
DoC
DE
DoC
Avg
Ours w/ Full Set (Annotated by Qwen3-32B)
61.84
79.89
62.33
69.53
58.10
88.32
53.06
74.95
68.50
Ours w/ Full Set (Annotated by Qwen3-4B)
62.04
80.11
62.86
70.07
57.41
89.32
51.42
75.86
68.63
LLM-as-a-Judge (LF-Analogical, Qwen3-32B)
36.43
64.76
38.55
53.77
33.30
82.54
32.37
64.07
50.72
Appendix
Table 5: Performance comparison between PRMs trained on different annotation models and using the annotation model (Qwen3-32B) directly as a judge.
Method
HepG2
Jurkat
K562
RPE1
Avg
DE
DoC
Avg
DE
DoC
Avg
DE
DoC
Avg
DE
DoC
Avg
Only Query (Greedy)
20.26
41.37
30.82
22.03
36.37
29.20
19.23
47.37
33.30
19.65
34.84
27.25
30.14
Only Query (Majority Voting)
22.40
43.45
32.93
25.98
37.01
31.50
23.08
45.68
34.38
20.69
35.86
28.28
31.77
Only Query (BoN w/ Full Set)
58.09
66.18
62.14
60.80
55.91
58.36
58.39
72.51
65.45
53.17
51.94
52.56
59.62
SUMMER (Greedy)
40.05
71.15
55.60
41.12
59.91
50.52
38.47
83.47
60.97
37.52
68.85
53.19
55.07
SUMMER (Majority Voting)
43.56
74.51
59.04
44.63
61.69
53.16
41.49
85.10
63.30
38.88
72.31
55.60
57.77
Appendix
Table 6: Performance Comparison of Policy Input Modes (IID vs OOD).
Method
HepG2
Jurkat
K562
RPE1
Avg
DE
DoC
Avg
DE
DoC
Avg
DE
DoC
Avg
DE
DoC
Avg
Greedy Decoding
74.72
36.04
55.38
76.01
30.56
53.28
69.79
42.88
56.34
78.33
34.89
56.61
55.40
Majority Voting
74.97
39.20
57.08
76.10
28.74
52.42
69.96
34.47
52.21
78.51
30.69
54.60
54.08
Coverage (Upper Bound)
76.18
89.92
83.05
77.02
90.94
83.98
70.87
91.12
80.99
79.18
91.99
85.59
83.40
Our PRM
BoN w/ Full Set
75.65
75.09
75.37
76.43
68.70
72.56
70.46
81.79
76.13
78.88
62.30
70.59
73.66
Appendix
Table 7: Detailed Results of DC-W2S using RBIO1 policy model.
Table 12
Figure 6: Performance Comparison of Different Gene Embedding Approaches (HEPG2).
Figure 7: Performance Comparison of Different Gene Embedding Approaches (JURKAT).
Figure 8: Performance Comparison of Different Gene Embedding Approaches (K562).
Figure 9: Performance Comparison of Different Gene Embedding Approaches (RPE1).
Figure 10: Comparison of label pattern across embeddings and regimes.
Figure 11: Scaling trend of BoN (w/ CellProfiler) on HEPG2.
Figure 12: Scaling trend of BoN (w/ CellProfiler) on JURKAT.
Figure 13: Scaling trend of BoN (w/ CellProfiler) on K562.
Figure 14: Scaling trend of BoN (w/ CellProfiler) on RPE1.
Figure 15: Aggregation method comparison on BoN (w/ Full Set) on RPE1(OOD).
Figure 16: Results of additional metrics on HEPG2.
Figure 17: Results of additional metrics on JURKAT.
National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +3