Learning Individual Scientific Taste with Privileged Hindsight Distillation
Authors: Hanwei Fan, Yanqiao Wang, Yiqi Zhu, Xuanyu Lei, Peng Li, Yang Liu
Organizations: Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China · Qiuzhen College, Tsinghua University, Beijing, China
Scientific taste shapes which research directions researchers pursue, and it varies across individuals. Existing work evaluates research directions along multiple evaluation criteria or models scientific taste using community feedback signals such as citations and reviews, but neither directly captures how individuals translate assessments under the same criteria into different preferences. We model scientific taste as a researcher-specific mapping from criterion-level assessments in a shared, evidence-grounded evaluation space with proposal-visible evidence, to relative preferences. A population-level mapping learned from retrospective supervision provides a cold start when individual feedback is scarce. We introduce Privileged Hindsight Distillation (PHD), which uses later-emerging scientific evidence during training to improve proposal-time criterion assessment and the population mapping, while exposing only proposal-visible information at inference. We evaluate our approach primarily in quantum research, where structured public reviews are largely unavailable, and use pairwise proposal comparison as a direct interface for observing individual taste. With only 15 comparisons per researcher, our method captures sharply different tastes, including opposing preferences over the same research-direction pairs when they trade off across evaluation criteria. In an interactive study, personalized rankings also surface proposals associated with greater subsequent user engagement, connecting learned taste to downstream research exploration.
Figures & tables
Figure 1: Modeling scientific taste. Proposal-stage evidence places research directions in a shared evaluation space, while researchers may map the same criterion profiles to different preferences. Later-emerging evidence can provide retrospective supervision for population-level learning but is unavailable at proposal-time inference.
Figure 2: Overview of our framework. (a) The framework learns a shared evaluation space, distills a population-level taste mapping, and sequentially adapts it to individual researchers; PHD refines the task-specific artifacts used by the first two components. (b) PHD iteratively identifies student failures, uses privileged hindsight evidence to diagnose them and revise the artifacts, and retains only revisions that pass leakage screening and held-out validation. At inference, the student uses only proposal-visible information and the final accepted artifacts.
In-context
Agentic
Parametric
SC
PP
SynthesizeMe
FERMI
CIPHER
Ours
VPL
LoRe
FSPO
Real ↑
0.479
0.481
0.477
0.419
0.524
0.693
0.629
0.595
0.642
Sim. ↑
0.423
0.519
0.473
0.366
0.399
0.591
0.556
0.547
0.559
Table 1: Individual scientific taste prediction. RW-SP across real and simulated researchers.
Evaluation Space
Population
Individual RW-SP ↑
Backbone
E-MAE ↓
E-OA ↑
P-SP ↑
Ours
PP
CIPHER
gpt-5.5
0.183 / 0.216
0.721 / 0.713
0.833 / 0.667
0.693
0.481
0.524
gpt-5.4-mini
0.233 / 0.292
0.665 / 0.604
0.667 / 0.567
0.613
0.407
0.332
claude-opus-5
0.217 / 0.275
0.675 / 0.612
0.500 / 0.367
0.664
0.634
0.516
gemini-3.6-flash
0.216 / 0.258
0.699 / 0.662
0.767 / 0.367
0.647
0.422
0.405
Table 2: Robustness across backbones. E-MAE/E-OA denote MAE/Ordering Accuracy for Evaluation Space Learning, and P-SP denotes StrictPair for Population Taste Distillation. Shared-component entries report Ours / w/o PHD . For individual adaptation, we report RW-SP for our complete method and alternative baselines under the same backbone.
Simulated researchers
Real researchers
Metric
SC
PP
SM
SC
ΔCP↑
0.169∣0.120
0.130∣0.067
0.146∣0.151
0.125
Win Rate ↑
0.667∣0.583
0.571∣0.577
0.641∣0.635
0.684
ΔNDCG↑
0.064∣0.006
0.020∣0.018
0.037∣0.044
0.023
Table 3: Downstream evaluation and sensitivity to population prior. (a) Each column compares our personalized ranker against the specified comparator. For simulated researchers, each entry reports Quantum | Cross-domain . Real-user deployment uses Single Call (SC), while simulated cohorts additionally include Profile Prompting (PP) and SynthesizeMe (SM). (b) Individual RW-SP after replacing the population prior while keeping the backbone, support feedback, and adaptation procedure fixed.
Variant
E-MAE ↓
E-OA ↑
P-SP ↑
w/o PHD
0.292
0.604
0.567
w/o Ipriv
0.242
0.649
0.533
Ours
0.233
0.665
0.667
Table 4: Ablation studies. (a) Effect of PHD and privileged hindsight on the two shared components under proposal-only inference. (b) Effect of individual residuals and conditional rules on RW-SP. The w/o PHD variant starts from the seed Rubric B and a randomly initialized population prior.
Support count
0
5
10
12
13
14
15
Ours
0.538
0.535
0.580
0.603
0.648
0.599
0.613
w/o replay
0.538
0.535
0.580
0.575
0.615
0.578
0.602
Table 5: Replay under sequential feedback. Effect of replay-based rule acceptance on held-out RW-SP as support feedback accumulates. The ablation bypasses the cross-fitted replay gate for accepting personal conditional rules.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
Reference
Seed
Learned
Generality
3
3
2
Novelty
2
3
2
Importance
2
2
2
Parsimony
2
0
1
Depth
2
2
2
Algorithmic complexity
2
1
1
Appendix
Table 6: Evaluation-space learning on a held-out proposal. Rubric A provides the reference coordinates. Seed and Learned show the proposal-time predictions before and after PHD learning.
Evidence source
Content
Student-visible
Proposal text
Complete proposal claims, assumptions, equations, and methodology
Yes
Proposal-time literature
Prior work and external evidence available when the proposal is evaluated
Yes
Criterion annotation
Six-dimensional Rubric-A assessment and its supporting evidence
No
Completed work
Source-paper methodology, experimental findings, and realized analyses
No
Community response
Follow-up citations, scites in SciRate, citation function and polarity, adoption distance, methodological integration, and primitive reuse
No
Appendix
Table 7: Evidence available to the hindsight reference panel. Proposal-visible sources may also be accessed by the deployable workflow, whereas privileged sources remain teacher-side information.
Table 8: An illustrative case of divergent scientific taste. Two researchers evaluate the same ten proposal pairs using identical criterion features. Predictions are shown as U04/U05. Bold indicates a correct individual prediction. A star indicates that both opposing preferences are correctly recovered. Proposal titles are abbreviated for readability.
Component
Method
Backbone
Input
Support
Alternative prior
Auto-J
autoj-13b
Proposal pair
0
Alternative prior
MLR-Judge
claude-opus-5 , gemini-3.6-flash
Independent proposal scores
0
Alternative prior
SciJudge
SciJudge-4B
Title, abstract, date
0
Individual
SC
gpt-5.5
Proposal pair, support comparisons
15
Individual
PP
gpt-5.5
Proposal pair, researcher profile
15
Individual
SynthesizeMe
gpt-5.5
Proposal pair, persona, demonstrations
15
Appendix
Table 9: Baseline configurations. Backbones, input representations, and target-researcher feedback for the evaluated methods. External population judges are evaluated as alternative priors under the downstream individual-adaptation protocol. Detailed implementation settings are provided below.
Figure 3: Interactive proposal exploration system. The illustration presents the web interface, its main functionalities, and an example of how researchers explore candidate directions and provide feedback. The learned individual taste mapping informs personalized proposal ranking throughout the interaction process.
Scientific discovery depends on expert judgement and foresight, which we call scientific taste: the ability to judge and propose research ideas with the potential for long-term scientific impact. Scientific taste is largely concentrated among highly experienced researchers, whose expertise is usually limited to a few specialised fields. If AI could learn scientific taste, it could reduce reliance on human experts and accelerate scientific discovery. Whether AI can learn this ability remains an open question. We introduce Reinforcement Learning from Community Feedback (RLCF) to learn judgement and ideation. Scientific Judge learns from community feedback, such as citations. Scientific Thinker learns to propose research ideas with high potential impact. Experiments show that Scientific Judge outperforms strong LLM baselines and that learned judgement generalises to future-year papers, other community metrics, and unseen fields. Furthermore, Scientific Thinker proposes research ideas with higher potential impact than those proposed by baselines. These results suggest that AI can learn scientific taste, marking an important step towards AI systems that could help accelerate scientific discovery.
Jingqi Tong, Mingzhe Li, Hangcheng Li +20
Fudan University · Shanghai Innovation Institute · OpenMOSS Team +2
We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models. We define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. TasteVal measures the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes. We operationalize experimental research taste as compute efficiency; a Researcher who reaches the same score as an expert human using half the serial experimental compute has twice the experimental taste. Experimental taste thus acts as a multiplier on experimental compute, making it a key input to forecasts of AI progress. TasteVal consists of 8 novel, challenging, open-ended tasks representative of frontier AI R&D. To isolate taste from coding ability, the model under evaluation acts as a Researcher that iteratively designs experiments while a fixed Coder agent implements them and reports their results. The Researcher executes until either the 40 H100 hour or 120 wall-clock hour budgets are exhausted. We recruit 24 human experts, at least 2 per task, and take the best expert attempt per task as the expert baseline. We evaluate 20 models released between 2023 and 2026. The best-performing model, Opus 5.5, exceeds our expert baseline, with a compute multiplier of 2.3x (95% CI 1.15-4.37), at roughly 1/30 of our baseliners' average per-run cost. On TasteVal, the compute multiplier of frontier models has doubled approximately every 3.0 months since December 2025 (95% CI 1.7-5.0), up from every 14 months between 2023 and December 2025. Measured by final normalized performance, frontier models show no trend break, doubling every 14.6 months. To keep TasteVal uncontaminated, we do not release the tasks.
As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience
Rongcan Pei, Fang Guo, Qinglin Qi +7
Tongji University · Westlake University · Zhejiang University +2