Learning Individual Scientific Taste with Privileged Hindsight Distillation
Authors: Hanwei Fan, Yanqiao Wang, Yiqi Zhu, Xuanyu Lei, Peng Li, Yang Liu
Organizations: Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China · Qiuzhen College, Tsinghua University, Beijing, China
Scientific taste shapes which research directions researchers pursue, and it varies across individuals. Existing work evaluates research directions along multiple evaluation criteria or models scientific taste using community feedback signals such as citations and reviews, but neither directly captures how individuals translate assessments under the same criteria into different preferences. We model scientific taste as a researcher-specific mapping from criterion-level assessments in a shared, evidence-grounded evaluation space with proposal-visible evidence, to relative preferences. A population-level mapping learned from retrospective supervision provides a cold start when individual feedback is scarce. We introduce Privileged Hindsight Distillation (PHD), which uses later-emerging scientific evidence during training to improve proposal-time criterion assessment and the population mapping, while exposing only proposal-visible information at inference. We evaluate our approach primarily in quantum research, where structured public reviews are largely unavailable, and use pairwise proposal comparison as a direct interface for observing individual taste. With only 15 comparisons per researcher, our method captures sharply different tastes, including opposing preferences over the same research-direction pairs when they trade off across evaluation criteria. In an interactive study, personalized rankings also surface proposals associated with greater subsequent user engagement, connecting learned taste to downstream research exploration.
Figures & tables
Figure 1: Modeling scientific taste. Proposal-stage evidence places research directions in a shared evaluation space, while researchers may map the same criterion profiles to different preferences. Later-emerging evidence can provide retrospective supervision for population-level learning but is unavailable at proposal-time inference.
Figure 2: Overview of our framework. (a) The framework learns a shared evaluation space, distills a population-level taste mapping, and sequentially adapts it to individual researchers; PHD refines the task-specific artifacts used by the first two components. (b) PHD iteratively identifies student failures, uses privileged hindsight evidence to diagnose them and revise the artifacts, and retains only revisions that pass leakage screening and held-out validation. At inference, the student uses only proposal-visible information and the final accepted artifacts.
In-context
Agentic
Parametric
SC
PP
SynthesizeMe
FERMI
CIPHER
Ours
VPL
LoRe
FSPO
Real ↑
0.479
0.481
0.477
0.419
0.524
0.693
0.629
0.595
0.642
Sim. ↑
0.423
0.519
0.473
0.366
0.399
0.591
0.556
0.547
0.559
Table 1: Individual scientific taste prediction. RW-SP across real and simulated researchers.
Evaluation Space
Population
Individual RW-SP ↑
Backbone
E-MAE ↓
E-OA ↑
P-SP ↑
Ours
PP
CIPHER
gpt-5.5
0.183 / 0.216
0.721 / 0.713
0.833 / 0.667
0.693
0.481
0.524
gpt-5.4-mini
0.233 / 0.292
0.665 / 0.604
0.667 / 0.567
0.613
0.407
0.332
claude-opus-5
0.217 / 0.275
0.675 / 0.612
0.500 / 0.367
0.664
0.634
0.516
gemini-3.6-flash
0.216 / 0.258
0.699 / 0.662
0.767 / 0.367
0.647
0.422
0.405
Table 2: Robustness across backbones. E-MAE/E-OA denote MAE/Ordering Accuracy for Evaluation Space Learning, and P-SP denotes StrictPair for Population Taste Distillation. Shared-component entries report Ours / w/o PHD . For individual adaptation, we report RW-SP for our complete method and alternative baselines under the same backbone.
Simulated researchers
Real researchers
Metric
SC
PP
SM
SC
ΔCP↑
0.169∣0.120
0.130∣0.067
0.146∣0.151
0.125
Win Rate ↑
0.667∣0.583
0.571∣0.577
0.641∣0.635
0.684
ΔNDCG↑
0.064∣0.006
0.020∣0.018
0.037∣0.044
0.023
Table 3: Downstream evaluation and sensitivity to population prior. (a) Each column compares our personalized ranker against the specified comparator. For simulated researchers, each entry reports Quantum | Cross-domain . Real-user deployment uses Single Call (SC), while simulated cohorts additionally include Profile Prompting (PP) and SynthesizeMe (SM). (b) Individual RW-SP after replacing the population prior while keeping the backbone, support feedback, and adaptation procedure fixed.
Variant
E-MAE ↓
E-OA ↑
P-SP ↑
w/o PHD
0.292
0.604
0.567
w/o Ipriv
0.242
0.649
0.533
Ours
0.233
0.665
0.667
Table 4: Ablation studies. (a) Effect of PHD and privileged hindsight on the two shared components under proposal-only inference. (b) Effect of individual residuals and conditional rules on RW-SP. The w/o PHD variant starts from the seed Rubric B and a randomly initialized population prior.
Support count
0
5
10
12
13
14
15
Ours
0.538
0.535
0.580
0.603
0.648
0.599
0.613
w/o replay
0.538
0.535
0.580
0.575
0.615
0.578
0.602
Table 5: Replay under sequential feedback. Effect of replay-based rule acceptance on held-out RW-SP as support feedback accumulates. The ablation bypasses the cross-fitted replay gate for accepting personal conditional rules.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
Reference
Seed
Learned
Generality
3
3
2
Novelty
2
3
2
Importance
2
2
2
Parsimony
2
0
1
Depth
2
2
2
Algorithmic complexity
2
1
1
Appendix
Table 6: Evaluation-space learning on a held-out proposal. Rubric A provides the reference coordinates. Seed and Learned show the proposal-time predictions before and after PHD learning.
Evidence source
Content
Student-visible
Proposal text
Complete proposal claims, assumptions, equations, and methodology
Yes
Proposal-time literature
Prior work and external evidence available when the proposal is evaluated
Yes
Criterion annotation
Six-dimensional Rubric-A assessment and its supporting evidence
No
Completed work
Source-paper methodology, experimental findings, and realized analyses
No
Community response
Follow-up citations, scites in SciRate, citation function and polarity, adoption distance, methodological integration, and primitive reuse
No
Appendix
Table 7: Evidence available to the hindsight reference panel. Proposal-visible sources may also be accessed by the deployable workflow, whereas privileged sources remain teacher-side information.
Table 8: An illustrative case of divergent scientific taste. Two researchers evaluate the same ten proposal pairs using identical criterion features. Predictions are shown as U04/U05. Bold indicates a correct individual prediction. A star indicates that both opposing preferences are correctly recovered. Proposal titles are abbreviated for readability.
Component
Method
Backbone
Input
Support
Alternative prior
Auto-J
autoj-13b
Proposal pair
0
Alternative prior
MLR-Judge
claude-opus-5 , gemini-3.6-flash
Independent proposal scores
0
Alternative prior
SciJudge
SciJudge-4B
Title, abstract, date
0
Individual
SC
gpt-5.5
Proposal pair, support comparisons
15
Individual
PP
gpt-5.5
Proposal pair, researcher profile
15
Individual
SynthesizeMe
gpt-5.5
Proposal pair, persona, demonstrations
15
Appendix
Table 9: Baseline configurations. Backbones, input representations, and target-researcher feedback for the evaluated methods. External population judges are evaluated as alternative priors under the downstream individual-adaptation protocol. Detailed implementation settings are provided below.
Figure 3: Interactive proposal exploration system. The illustration presents the web interface, its main functionalities, and an example of how researchers explore candidate directions and provide feedback. The learned individual taste mapping informs personalized proposal ranking throughout the interaction process.