cs.AIOct 27, 2026

VerifiedLab: Harnessing Knowns for Unknowns

Authors: Yu Yao, Jiayi Dong, Yang Yang, Shuiwang Ji, Ju Li

Organizations: Massachusetts Institute of Technology, Cambridge, MA, USA. · University of California, Los Angeles, CA, USA. · Texas A&M University, College Station, TX, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Across human history, knowledge has undergone several transitions, from orality to writing and then to digital archives. At each transition, humans established new conventions to preserve knowledge for the new era. Nowadays, as research agents are increasingly deployed for laboratory experiments, knowledge is crossing another boundary: from records written for humans to those for machines. In this work, we introduce VerifiedLab, a multi-agent harness framework for this human-to-machine transition of knowledge. VerifiedLab transforms research output from passive artifacts into active systems that accelerate downstream use, adoption and prediction of future trends. In VerifiedLab, a team of AI scientists with distinct roles reasons over records verified field by field: a principal investigator plans, a librarian computes the evidence directly from the records, specialists draft, and a gate and a critic check each claim against the measured reliability of its data before the harness decides whether to release or withhold it. We demonstrate the efficacy of VerifiedLab through extensive experiments on perovskite solar cells: VerifiedLab recovers known physical trends, anticipates devices reported a year later, and lets AI scientists inherit the literature together with a measure of how far it can be trusted. By scaling the inference-time verification compute, VerifiedLab introduces a new pathway for knowledge dissemination and machine harness for AI co-scientists.

Figures & tables

Fig. 1: Scientific knowledge has crossed a boundary each time it changed medium. Left, the media that carried knowledge; right, the rule that made each trustworthy. The apex is the boundary being crossed now, where AI scientists read papers and this work supplies the rule. Icons: Heroicons (MIT), Font Awesome Free (CC BY 4.0).
Fig. 2: Overview of the verified-literature harness. a , Two coupled agent loops. In the harness loop (blue), a generator agent fills one record per paper under the annotation Skill AαA^{\alpha} , a verifier agent labels every field under the verification Skill VβV^{\beta} , and the audit re-versions both Skills. In the discovery loop (purple), a PI agent, a librarian and specialist agents reason over the verified database, and a critic rejects claims without provenance (dashed red). Published proposals re-enter the harness as papers (grey). b , The objects that a human laboratory, an LLM reading PDFs and the Verified Lab use for input, policy, evidence, checking and provenance. c , A recipe refined over three iterations with specialist feedback; nn and the share above 22% are computed over the matching verified records. d , A real record reconstructed without and with the annotation Skill (red, errors; green, released record), with the verifier’s feedback. e , Verifier decisions on candidate–reference pairs and the rule applied. f , A claim before and after the critic. Icons: Heroicons (MIT), Font Awesome Free (CC BY 4.0).
Fig. 3: Research agents reconstruct experimental records. a , Field accuracy of 18 generators on Collection A (100 papers, 9,700 field decisions each) with (filled) and without (open) the annotation Skill; colours, model families; dashed line, empty-record baseline. b , Accuracy by subfield for three generators. c , Error composition for the five strongest generators, by subfield (left) and by the competence a field tests (right; localized scalars, 24 fields; relational structures, 30; conditions, 43). Omission, wrong value and over-prediction are defined in Methods. Decisions are those of the o3 verifier with the verification Skill.
Fig. 4: Written verification rules bring verifiers to expert agreement. a , Agreement with expert decisions for three proprietary verifiers, with and without the verification Skill, on Collection B ( nn = 1,460 decisions) and Collection A ( nn = 1,940). b , Agreement of ten open-weight verifiers with the audited o3 decisions (9,700 decisions) without (open) and with (filled) the Skill; Δ\Delta , paired difference in percentage points. c , Acceptance of equivalent pairs ( nn = 4,620) against rejection of non-equivalent pairs ( nn = 548), without (open) and with (filled) the Skill; inset, dashed region enlarged. d , Within each family, the largest minus the smallest checkpoint without rules (grey) and the smallest with rules minus the largest without (blue). Error bars, 95% record-clustered bootstrap intervals (2,000 resamples).
Fig. 5: Reliable fields rediscover known trends and predict devices reported a year later. a – c , The 10,258 champion devices read from the high-reliability categories: a , reported PCE (dashed line, median); b , PCE against bandgap with the Shockley–Queisser limit (dashed) and the 1.5–1.7 eV window (shaded); c , PCE against cell area with the maximum, mean and median per logarithmic bin. d – g , Forecasting 2025 devices from recipe fields alone: d , reported PCE by publication year and the temporal split (box, interquartile range; whiskers, 1.5 ×\times ); e – g , the fine-tuned T5 encoder on the 858 test devices, with predicted against reported PCE (band, ±\pm 2 points), the cumulative absolute error and the ROC curve for PCE >> 20%; the expert-descriptor LightGBM and the training-mean predictor are shown for reference. h , Benchmark on the same split: MAE, the ratio of the mean absolute deviation of the targets to the MAE (MAD:MAE), R2R^{2} and the AUC for PCE >> 20%; mean ±\pm s.d. over three seeds; bold, best per metric.
Fig. 6: A team of AI scientists reasoning over verified records. a , Architecture, after the Virtual Lab 70 . A human sets the goal; a PI agent recruits specialists and a critic; every agent reaches the literature through the librarian, which returns records tagged by category reliability and provenance, and a forecaster trained on records to 2023 scores candidate devices. b , Six phases of a design question. Dialogue is illustrative and abridged; all numbers are computed from the released records and the forecaster. The recipes in Phase 5 are held-out 2025 devices chosen for illustration. c , One design round: the device at each iteration, the agent acting on it, the record the librarian returns with its verifier labels (TP, true positive), and the claim after the critic strikes the clause that rests on an unreliable category. nn and the share above 22% are computed over the matching verified records (Methods).
More in this topic

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 389 papers

Explore similar work

CardsList
Jun 10, 2026

Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills

Scientific discovery workflows rely heavily on lab notes, where researchers record observations, interpret uncertain results, and plan follow-up experiments. Unlike polished publications, lab notes preserve evolving scientific reasoning, tacit scientific knowledge, and author uncertainty, giving AI agents access to the process of science. However, most prior work on scientific text focuses on papers, protocols, or structured databases, leaving informal laboratory notes underexplored as inputs to AI agents for science. This gap matters because tacit knowledge in lab notes is not written as a clean set of instructions: validated observations, tentative judgments, and possible experimental next steps often appear in the same passage. If these signals are conflated, an AI agent may mistake uncertain scientific judgments for confirmed conclusions or executable actions. To this end, we present Notes2Skills, a two-stage framework for turning lab notebooks into verifiable skills for scientific AI agents while preserving the author's certainty. Across three wet-lab sessions and seven original downstream configurations, Notes2Skills is the only tested configuration that avoids both observed regime-level failures: over-acting on uncertain notes and missing firm directives. We show that certainty preservation is a key missing piece between lab notebooks and reliable agent skills, opening a path toward safer AI co-scientist systems.

Shi Liu, Jiayao Chen, Chengwei Qin +3

Southern University of Science and Technology, Shenzhen, China · Shenzhen Loop Area Institute, Shenzhen, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China +1

AI for ScienceDocument Information ExtractionLLM Agent Skill LearningLLM Agent Reliability
AskPDF
Sep 11, 2026

LabAgent: Customize Any Research Hubs for Scientific Discoveries Using AI Agents

Scientific research is a continuous process that emphasizes inheritance. Methods developed by predecessors are often expanded upon by new researchers to explore more novel and in-depth scientific questions. However, the change of lab staff, such as student graduation, leads to a lack of personnel capable of replicating methods. Methods that have been developed with significant effort and resources cannot be continued. To address these limitations, we propose LabAgent, a reproduce and discovery harness tailored for a lab's continuous work. LabAgent employs two mechanisms to guarantee that all skills can be executed and verified and to record the corrective methods and experiences, allowing for direct correction or avoidance of similar errors. We applied LabAgent to drug property prediction, biomedical problem analysis, protein variant effect prediction, and statistical genetics in life science domains. LabAgent ranks first over commercial generalist agents in every domain, and demonstrates accurate reproduction of a published figure. Overall, these results demonstrate that LabAgent can effectively integrate and reasonably expand laboratory knowledge.

Lei Liu, Yikun Zhang, Jialin Chen +7

Department of Biostatistics, Yale University · Department of Computer Science, Northeastern University · EWSC Center, Broad Institute of MIT and Harvard +5

LLM Agent HarnessesScientific ReproducibilityAutonomous Scientific Discovery
AskPDF
May 3, 2026

Reliable AI Needs to Externalize Implicit Knowledge: A Human-AI Collaboration Perspective

This position paper argues that reliable AI requires infrastructure for human validation of implicit knowledge. AI learns from both explicit knowledge (papers, documentation, structured databases) and implicit knowledge (reasoning patterns, debugging processes, intermediate steps). Implicit knowledge remains unexternalized because documentation cost exceeds perceived value -- yet AI learns from it indiscriminately, acquiring both beneficial patterns and harmful biases. Current reliability methods can only verify explicit knowledge against sources, creating a fundamental gap: the most valuable AI capabilities (reasoning, judgment, intuition) are precisely those we cannot verify. We propose Knowledge Objects (KOs) -- structured artifacts that externalize implicit knowledge into forms humans can inspect, verify, and endorse. KOs transform verification economics: what was previously too costly to verify becomes feasible, enabling accumulated human validation to improve reliability over time.

Hengyu Liu, Tianyi Li, Zhihong Cui +5

Department of Computer Science, Aalborg University, Aalborg, Denmark · Department of Informatics, University of Oslo, Oslo, Norway · School of Computing, Macquarie University, Sydney, Australia

AI Agent ReliabilityHuman-in-the-Loop AIHuman-AI Collaboration
AskPDF