cs.CYMar 4, 2026
SaveMeasuring AI R&D Automation
Organizations: GovAI · University of Oxford
Abstract
The automation of AI R&D (AIRDA) could have significant implications, but its extent and ultimate effects remain uncertain. We need empirical data to resolve these uncertainties, but existing data (primarily capability benchmarks) may not reflect real-world automation or capture its broader consequences, such as whether AIRDA accelerates capabilities more than safety progress or whether our ability to oversee AI R&D can keep pace with its acceleration. To address these gaps, this work proposes metrics to track the extent of AIRDA and its effects on AI progress and oversight. The metrics span dimensions such as capital share of AI R&D spending, researcher time allocation, and AI subversion incidents, and could help decision makers understand the potential consequences of AIRDA, implement appropriate safety measures, and maintain awareness of the pace of AI development. We recommend that companies and third parties (e.g. non-profit research organisations) start to track these metrics, and that governments support these efforts.
Figures & tables
Figure 1: The extent of AI R&D automation could affect both AI progress and the oversight gap : the difference between how much oversight is needed (“ oversight demand ”) and how much oversight is actually achieved. Oversight capacity is the ability to achieve oversight, encompassing both the ability to understand the R&D process (e.g., having sufficient expertise) and the resources available for exercising control (e.g., human labour, monitoring tools). AI progress could also affect the oversight gap, such as by increasing the stakes of R&D decisions. This work proposes metrics to track all of these quantities.
| Actor | Recommendation | Metrics |
|---|---|---|
| Companies | Track differential progress in automating safety vs. capabilities research | Metric #1 (AI R&D evaluations) and Metric #6 (staff surveys) are most straightforward. Metric #2 (human–AI comparison RCTs) can also track differential progress, but requires significantly more effort. |
| Companies | Track how AIRDA affects oversight | Metric #7 (AI use in high-stakes decisions), Metric #8 (researcher time allocation), and Metric #14 (AI permission lists) are the most feasible starting points. Metric #10 (AI subversion incidents) requires somewhat more effort, while Metric #9 (oversight effectiveness retrospectives) requires the most effort. |
| Companies | Track the actual extent of AIRDA | Metric #11 (researcher headcount), Metric #12 (compute distribution), and Metric #13 (capital share) are relatively straightforward. Metric #8 (researcher time allocation) would also be informative, but requires somewhat more implementation effort. |
| Governments | Develop systems for confidential reporting, potentially in the form of industry-wide aggregates | Promising candidates are Metric #7 (AI use in high-stakes decisions), Metric #10 (AI subversion incidents), Metric #12 (compute distribution), and Metric #13 (capital share). |
| Third parties | Estimate metrics using public sources | Feasible places to start are Metric #12 (distribution of compute usage; You, 2025 ) and Metric #11 (researcher headcount). |
| Third parties | Create tooling and design surveys | Time-tracking software for Metric #8 (researcher time allocation), and surveys for Metric #6 (self-reported productivity boosts) and Metric #7 (AI use in high-stakes decisions). |
Table 1: Recommendations for companies, governments, and third parties (e.g. non-profit research organisations).
Figure 2: Positive and negative implications of AI R&D automation for AI progress.
Figure 3: AI R&D automation could decrease or increase the oversight gap by changing oversight capacity, the level of oversight achieved (not depicted for simplicity), or oversight demand. Note that oversight capacity may not directly translate into oversight achieved (e.g., companies decide to forgo oversight due to financial or time costs).
| Metric | Description | Extent of AIRDA | AI Progress | Oversight Capacity | Oversight Achieved | Oversight Demand | What remains to be done |
|---|---|---|---|---|---|---|---|
| Metric #1 : AI performance on AI R&D evaluations | AI performance on evaluations of capabilities directly relevant to AI R&D work, such as replicating ML papers and picking research ideas to test. | ✓ Leading indicator of AI systems’ potential to perform AI R&D. | ✓ Can compare improvements on safety research tasks to capabilities research tasks. | Developing more evaluations for non-software-engineering tasks and safety and security research tasks. | |||
| Metric #2 : AI performance on AI R&D evaluations compared to humans and human-AI teams (“AI R&D Performance RCTs”) | Performance differences between human-AI teams, human-only teams, and AI-only teams on AI R&D tasks. | ✓ Superior AI-only performance suggests greater AIRDA potential. | ✓ Can compare improvements on safety research tasks to capabilities research tasks. | ✓ Depending on the interaction protocol, superior human-AI performance can suggest non-negligible oversight demand. | Broadening coverage of AI R&D tasks and running more comparisons between AI systems and human-AI teams. | ||
| Metric #3 : Oversight red-teaming experiments | Evaluations for whether oversight systems used in production would catch an AI system that is directed to subvert the R&D process, such as by attempting to sabotage experiments. | ✓ Measures ability to catch attempts to subvert the AI R&D process. | Developing and running evaluations specific to internal infrastructure. | ||||
| Metric #4 : Misalignment evaluations | Evaluations measuring propensities and capabilities relevant to misaligned behaviour, including alignment faking, sycophancy, sabotage, and reward hacking. | ✓ Greater misalignment increases oversight demand. | Developing evaluations that are more representative of realistic AI R&D workflows. | ||||
| Metric #5 : Compute efficiency improvements | The year-over-year percentage reduction in the amount of compute needed to achieve a given level of performance on capability (including AI R&D) evaluations. | ✓ Can measure efficiency improvements on AI R&D tasks. | ✓ Directly tracks AI progress. | Calculating efficiency improvements from existing data. Potentially running additional evaluations. |
Table 2: Experimental metrics for measuring AI R&D automation.
| Metric | Description | Extent of AIRDA | AI Progress | Oversight Capacity | Oversight Achieved | Oversight Demand | What remains to be done |
|---|---|---|---|---|---|---|---|
| Metric #6 : Staff views on AI use and productivity boosts | Researchers’ self-reported AI use patterns and productivity gains across different AI R&D task categories. | ✓ Elicits self-reported AI usefulness across R&D tasks. | ✓ Can compare productivity boosts across teams (e.g. safety vs. pre-training teams). | ✓ Can ask how helpful AI is for oversight tasks. | Designing questions and establishing a regular cadence for running surveys. | ||
| Metric #7 : Extent of AI use in high-stakes decisions | Survey of how AI is involved in high-stakes R&D decisions, such as research agenda prioritization, training runs, and deployment decisions. | ✓ Self-reported signal of AI involvement in key decisions. | ✓ Can ask about how much oversight is exercised over AI involvement in high-stakes decisions. | ✓ Significant AI involvement suggests greater oversight demand. | Identifying high-stakes R&D decisions. Designing surveys and running them each time such decisions are made. |
Table 3: Survey-based metrics for measuring AI R&D automation.
| Metric | Description | Extent of AIRDA | AI Progress | Oversight Capacity | Oversight Achieved | Oversight Demand | What remains to be done |
|---|---|---|---|---|---|---|---|
| Metric #8 : Researcher time allocation (“AI-powered Toggl”) | How human researchers allocate time across different R&D activities, measured via automated time tracking software. | ✓ Captures potential impacts of AIRDA on researcher time allocation. | ✓ Can measure time spent on oversight activities. | Developing and deploying time-tracking software, managing privacy concerns. | |||
| Metric #9 : Oversight effectiveness retrospectives | The proportion of AI-generated R&D outputs (e.g. production code, experimental analyses) that have defects, stratified by level of oversight. | ✓ Tracks oversight activities on outputs. | ✓ Tracks oversight gap directly, and oversight demand indirectly if level of oversight is held constant. | Building a tracking/tagging pipeline and establishing a regular data analysis schedule. | |||
| Metric #10 : AI subversion incidents | The number and severity of incidents in which AI systems attempted to subvert real AI R&D processes, such as by sabotaging experiments or hiding misbehaviour. | ✓ An increasing rate of subversion incidents indicates growing oversight demand. | Defining what counts as a subversion incident and building detection infrastructure. |
Table 4: Operational metrics for measuring AI R&D automation.
| Metric | Description | Extent of AIRDA | AI Progress | Oversight Capacity | Oversight Achieved | Oversight Demand | What remains to be done |
|---|---|---|---|---|---|---|---|
| Metric #11 : Headcount of AI researchers and the distribution of their performance | The number and seniority of AI researchers at frontier AI companies, along with the distribution of their performance, as estimated, for example, by salaries or compute budgets. | ✓ Decreased headcount suggests increased AIRDA. | ✓ Useful AI tools could lead to a widening performance distribution. | ✓ Decreased headcount or performance could reduce oversight capacity. | ✓ Decreased headcount could reduce how much oversight is performed. | Aggregating and presenting existing data. | |
| Metric #12 : Distribution of compute usage | How compute is distributed across various AI R&D activities. | ✓ AIRDA could shift compute use towards internal inference. | Developing systems to classify compute use. | ||||
| Metric #13 : Capital share of AI R&D spending | Compute expenditure as a percentage of total AI R&D spending. | ✓ Rising capital-to-labour ratio could signal increasing automation. | Calculating and reporting the metric. | ||||
| Metric #14 : AI permission lists | A list of which actions AI systems are authorized to take with different levels of human approval or review, including where none is required. | ✓ Automation is potentially higher if AI systems are permitted to take significant actions autonomously. | ✓ Oversight demand may be higher if AI systems are permitted to take significant actions without human review. | Defining actions of interest, collecting permissions across teams, and updating the list regularly. |
Table 5: Organizational metrics for measuring AI R&D automation.
More in this topic
AI Risk Management
AI risk management studies how to identify, assess, reduce, and monitor harms from AI across development, deployment, and wider social systems.
See all 108 papers