VerifiedLab: Harnessing Knowns for Unknowns

Yu Yao; Jiayi Dong; Yang Yang; Shuiwang Ji; Ju Li

cs.AIOct 27, 2026

VerifiedLab: Harnessing Knowns for Unknowns

Authors: Yu Yao, Jiayi Dong, Yang Yang, Shuiwang Ji, Ju Li

Organizations: Massachusetts Institute of Technology, Cambridge, MA, USA. · University of California, Los Angeles, CA, USA. · Texas A&M University, College Station, TX, USA.

Abstract

Across human history, knowledge has undergone several transitions, from orality to writing and then to digital archives. At each transition, humans established new conventions to preserve knowledge for the new era. Nowadays, as research agents are increasingly deployed for laboratory experiments, knowledge is crossing another boundary: from records written for humans to those for machines. In this work, we introduce VerifiedLab, a multi-agent harness framework for this human-to-machine transition of knowledge. VerifiedLab transforms research output from passive artifacts into active systems that accelerate downstream use, adoption and prediction of future trends. In VerifiedLab, a team of AI scientists with distinct roles reasons over records verified field by field: a principal investigator plans, a librarian computes the evidence directly from the records, specialists draft, and a gate and a critic check each claim against the measured reliability of its data before the harness decides whether to release or withhold it. We demonstrate the efficacy of VerifiedLab through extensive experiments on perovskite solar cells: VerifiedLab recovers known physical trends, anticipates devices reported a year later, and lets AI scientists inherit the literature together with a measure of how far it can be trusted. By scaling the inference-time verification compute, VerifiedLab introduces a new pathway for knowledge dissemination and machine harness for AI co-scientists.

Figures & tables

Fig. 1: Scientific knowledge has crossed a boundary each time it changed medium. Left, the media that carried knowledge; right, the rule that made each trustworthy. The apex is the boundary being crossed now, where AI scientists read papers and this work supplies the rule. Icons: Heroicons (MIT), Font Awesome Free (CC BY 4.0).

Fig. 2: Overview of the verified-literature harness. a , Two coupled agent loops. In the harness loop (blue), a generator agent fills one record per paper under the annotation Skill $A^{\alpha}$ , a verifier agent labels every field under the verification Skill $V^{\beta}$ , and the audit re-versions both Skills. In the discovery loop (purple), a PI agent, a librarian and specialist agents reason over the verified database, and a critic rejects claims without provenance (dashed red). Published proposals re-enter the harness as papers (grey). b , The objects that a human laboratory, an LLM reading PDFs and the Verified Lab use for input, policy, evidence, checking and provenance. c , A recipe refined over three iterations with specialist feedback; $n$ and the share above 22% are computed over the matching verified records. d , A real record reconstructed without and with the annotation Skill (red, errors; green, released record), with the verifier’s feedback. e , Verifier decisions on candidate–reference pairs and the rule applied. f , A claim before and after the critic. Icons: Heroicons (MIT), Font Awesome Free (CC BY 4.0).

Fig. 3: Research agents reconstruct experimental records. a , Field accuracy of 18 generators on Collection A (100 papers, 9,700 field decisions each) with (filled) and without (open) the annotation Skill; colours, model families; dashed line, empty-record baseline. b , Accuracy by subfield for three generators. c , Error composition for the five strongest generators, by subfield (left) and by the competence a field tests (right; localized scalars, 24 fields; relational structures, 30; conditions, 43). Omission, wrong value and over-prediction are defined in Methods. Decisions are those of the o3 verifier with the verification Skill.

Fig. 4: Written verification rules bring verifiers to expert agreement. a , Agreement with expert decisions for three proprietary verifiers, with and without the verification Skill, on Collection B (

n

= 1,460 decisions) and Collection A (

n

= 1,940). b , Agreement of ten open-weight verifiers with the audited o3 decisions (9,700 decisions) without (open) and with (filled) the Skill;

\Delta

, paired difference in percentage points. c , Acceptance of equivalent pairs (

n

= 4,620) against rejection of non-equivalent pairs (

n

= 548), without (open) and with (filled) the Skill; inset, dashed region enlarged. d , Within each family, the largest minus the smallest checkpoint without rules (grey) and the smallest with rules minus the largest without (blue). Error bars, 95% record-clustered bootstrap intervals (2,000 resamples).

Fig. 5: Reliable fields rediscover known trends and predict devices reported a year later. a – c , The 10,258 champion devices read from the high-reliability categories: a , reported PCE (dashed line, median); b , PCE against bandgap with the Shockley–Queisser limit (dashed) and the 1.5–1.7 eV window (shaded); c , PCE against cell area with the maximum, mean and median per logarithmic bin. d – g , Forecasting 2025 devices from recipe fields alone: d , reported PCE by publication year and the temporal split (box, interquartile range; whiskers, 1.5 $\times$ ); e – g , the fine-tuned T5 encoder on the 858 test devices, with predicted against reported PCE (band, $\pm$ 2 points), the cumulative absolute error and the ROC curve for PCE $>$ 20%; the expert-descriptor LightGBM and the training-mean predictor are shown for reference. h , Benchmark on the same split: MAE, the ratio of the mean absolute deviation of the targets to the MAE (MAD:MAE), $R^{2}$ and the AUC for PCE $>$ 20%; mean $\pm$ s.d. over three seeds; bold, best per metric.

Fig. 6: A team of AI scientists reasoning over verified records. a , Architecture, after the Virtual Lab 70 . A human sets the goal; a PI agent recruits specialists and a critic; every agent reaches the literature through the librarian, which returns records tagged by category reliability and provenance, and a forecaster trained on records to 2023 scores candidate devices. b , Six phases of a design question. Dialogue is illustrative and abridged; all numbers are computed from the released records and the forecaster. The recipes in Phase 5 are held-out 2025 devices chosen for illustration. c , One design round: the device at each iteration, the agent acting on it, the record the librarian returns with its verifier labels (TP, true positive), and the claim after the critic strikes the clause that rests on an unreliable category. $n$ and the share above 22% are computed over the matching verified records (Methods).

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 389 papers

Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills
Jun 10, 2026Shi Liu, Jiayao Chen, Chengwei Qin +3AI for Science Document Information Extraction
Southern University of Science and Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China +1
LabAgent: Customize Any Research Hubs for Scientific Discoveries Using AI Agents
Sep 11, 2026Lei Liu, Yikun Zhang, Jialin Chen +7LLM Agent Harnesses Scientific Reproducibility
Department of Biostatistics, Yale University · Department of Computer Science, Northeastern University · EWSC Center, Broad Institute of MIT and Harvard +5
Reliable AI Needs to Externalize Implicit Knowledge: A Human-AI Collaboration Perspective
May 3, 2026Hengyu Liu, Tianyi Li, Zhihong Cui +5AI Agent Reliability Human-in-the-Loop AI
Department of Computer Science, Aalborg University, Aalborg, Denmark · Department of Informatics, University of Oslo, Oslo, Norway · School of Computing, Macquarie University, Sydney, Australia

Jun 10, 2026J/K move · Enter open · S save

Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills

Shi Liu, Jiayao Chen, Chengwei Qin, Yanqing Hu+2

Southern University of Science and Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China+1

AI for Science Document Information Extraction LLM Agent Skill Learning LLM Agent Reliability

Open paper AskPDF arXiv

Scientific discovery workflows rely heavily on lab notes, where researchers record observations, interpret uncertain results, and plan follow-up experiments. Unlike polished publications, lab notes preserve evolving scientific reasoning, tacit scientific knowledge, and author uncertainty, giving AI agents access to the process of science. However, most prior work on scientific text focuses on papers, protocols, or structured databases, leaving informal laboratory notes underexplored as inputs to AI agents for science. This gap matters because tacit knowledge in lab notes is not written as a clean set of instructions: validated observations, tentative judgments, and possible experimental next steps often appear in the same passage. If these signals are conflated, an AI agent may mistake uncertain scientific judgments for confirmed conclusions or executable actions. To this end, we present Notes2Skills, a two-stage framework for turning lab notebooks into verifiable skills for scientific AI agents while preserving the author's certainty. Across three wet-lab sessions and seven original downstream configurations, Notes2Skills is the only tested configuration that avoids both observed regime-level failures: over-acting on uncertain notes and missing firm directives. We show that certainty preservation is a key missing piece between lab notebooks and reliable agent skills, opening a path toward safer AI co-scientist systems.

Abstract

LLM Agent Harnesses

Explore similar work

Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills

LabAgent: Customize Any Research Hubs for Scientific Discoveries Using AI Agents

Reliable AI Needs to Externalize Implicit Knowledge: A Human-AI Collaboration Perspective