cs.AIOct 29, 2026
SaveLetting AI Formulate the Research Question: Autonomous Research Agents
Organizations: Otsuma Women’s University, Faculty of Social Information Studies
Abstract
Systems such as The AI Scientist have shown that AI agents can carry out and write up research, but in those demonstrations humans chose the research area and selected the outputs to show. We study letting the AI formulate the research question itself, and we report every run. Our system gives an off-the-shelf coding agent an instruction document and two skills: one for formulating the question, in which the agent diverges from recent technologies into problems of other fields, screens its candidates for known failure modes and obtains a critique from a different model; and one for peer review, in which a reviewer from another model family reruns the study from its records before the agent revises the paper. Over 13 development runs, changes to the one or two sentences that set the research area coincided with large shifts in what the candidate questions and the chosen papers were about, which led us to the area of a new technology applied to a societal problem. We froze the system after a pilot run and report five consecutive runs: the pilot, counted by a disclosed amendment to the pre-registration, and four pre-registered runs. All five runs went from question formulation to a revised, re-computable paper, with local computation on a laptop and no research input from humans, and external AI reviewers reading only the papers judged them at or near workshop level. We release the system, the five papers and the records needed to reproduce them.
Figures & tables
Figure 1: Data flow in a run: a human and four agents (management agent, research agent, critic and reviewer). Numbers give the order within a run. The human deals only with the management agent; after the instructions, the research agent receives only fixed resumption messages, the critique it requested and the review. Steps 1 and 13 close the loop through which the instructions were revised between development runs (Section 4 ); during the evaluation runs the instructions were frozen.
| Run | Ver. | Model | Area setting | AI | Tools | Pract. | Sys. | Math | Soc. | Chosen study |
|---|---|---|---|---|---|---|---|---|---|---|
| 001 | v1.0 | gpt-6-sol | None: no field, no selection procedure | 0 | 0 | 0 | 0 | 2 | 0 | Non-uniform birthday problem (exact computation) |
| 002 | v1.0 | gpt-6.1-sol | Field: computer science (CS) | 0 | 0 | 0 | 2 | 0 | 0 | Admission control for LRU caches |
| 003 | v1.1 | Opus 5.5 | CS; at least 8 candidates; aim at questions that change someone’s judgment | 2 | 1 | 1 | 3 | 0 | 3 | Third-party tracking on Japanese local-government websites |
| 004 | v1.2 | Opus 5.5 | CS; “AI-era problems” welcomed and scored | 5 | 4 | 4 | 1 | 0 | 2 | Silent mis-application in coding-agent edit tools |
| 005 | v1.2.1 | GPT-6 | unchanged (research model changed) | 0 | 8 | 2 | 1 | 0 | 0 | Cost of structured-generation engines under JSON-Schema rewrites |
| 006 | v1.3 | Opus 5.5 | CS; readers are future autonomous research agents; AI-era axis removed | 9 | 2 | 7 | 1 | 0 | 0 | Do hypothesis-supporting results make LLM agents check less? |
Table 1: Development runs (Japanese instructions). “Area setting” is what the version of the instructions said about where to look; “unchanged” refers to this part only, since other parts of the instructions changed in every version. Candidates are counted by our classification of what each candidate studies: AI, the behaviour of AI models; Tools, tools and infrastructure for AI agents; Pract., research practice; Sys., computer systems; Math, mathematics; Soc., societal problems. The class of the chosen study is in bold.
| Run | Topic | Draft | Stops | Active | Margin | Resumes | Perm. | Advice | Review | Resp. | API |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 014 | 64 | 11:05 | 1:53 | 9:12 | 2:48 | 2 + 1 | 0 | 0 | 28 | 57 | 2.02 |
| 015 | 54 | 14:30 | 2:48 | 11:42 | 0:18 | 3 + 1 | 0 | 0 | 44 | 49 | 2.02 |
| 016 | 72 | 11:46 | 1:53 | 9:54 | 2:06 | 2 + 0 | 0 | 0 | 20 | 69 | 1.94 |
| 017 | 33 | 7:26 | 0:59 | 6:27 | 5:33 | 1 + 0 | 0 | 0 | 31 | 42 | 1.95–2.02 |
| 018 | 60 | 18:01 | 6:18 | 11:43 | 0:17 | 4 + 0 | 0 | 0 | 73 | 75 | 1.99 |
Table 2: Time and inputs. “Topic” is minutes from start to topic selection (excluding stops); “Draft”, “Stops” and “Active” are hours:minutes from start to the draft tag, of usage-limit stops, and their difference; “Margin” is the time left before the draft deadline (12 h plus stops). “Resumes” counts fixed resumption messages after usage limits + after other stops. “Perm.” are permission requests and “Advice” research-content inputs recorded in the run’s intervention log. “Review” and “Resp.” are minutes of the AI review and of the agent’s response. “API” is the cost of the critique in US dollars, as recorded by the agent; in run 017 the agent recorded a range because it could not determine how long cached input was retained. Times are rounded from seconds, so Active may differ from Draft minus Stops by one minute.
| Run | Version | Question | Main data |
|---|---|---|---|
| 014 | v1.10 | Screening one-year tree-cover change in the contiguous US with annual satellite embeddings | LCMAP, NLCD; AlphaEarth, TESSERA embeddings |
| 015 | v1.10.1 | Bounds on joint temperature-exceedance probabilities from published ensemble summaries | ECMWF ensemble forecasts |
| 016 | v1.10.2 | Re-sampling reference locations to estimate change in Swiss crop-group diversity | Swiss crop declarations; annual embeddings |
| 017 | v1.10.3 | Training-point allocation and recovery of small wetland patches | US National Wetlands Inventory; AlphaEarth, Sentinel-2 |
| 018 | v1.10.4 | Do satellite embedding distances improve matching of wetland water regimes? | US National Wetlands Inventory; AlphaEarth, TESSERA; terrain and surface water |
Table 3: The five runs and their questions (abridged from the paper titles). LCMAP: Land Change Monitoring, Assessment, and Projection; NLCD: National Land Cover Database; ECMWF: European Centre for Medium-Range Weather Forecasts.
| Run | Cand. | Tech. | Points | Adopted | On chosen |
|---|---|---|---|---|---|
| 014 | 15 | 6 | 26 | 22 | 9 |
| 015 | 15 | 6 | 26 | 21 | 7 |
| 016 | 15 | 6 | 23 | 13 | 7 |
| 017 | 15 | 6 | 23 | 19 | 6 |
| 018 | 17 | 7 | 21 | 14 | 6 |
Table 4: Candidate questions and the critique. “Cand.” is the number of candidates (3 obvious + 12–14 from the divergence), “Tech.” the number of technologies listed in the divergence; “Points” the critic’s points on the top five, “Adopted” those adopted or partly adopted by the agent, and “On chosen” the points on the candidate that was finally chosen. In no run did the critique change which candidate was chosen.
| Run | Verdict | Corr. | Verif. | Nov. | Gen. | Use. | Clar. | Crit. | Mod. | Minor | Response | Claims |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 014 | major | 4 | 5 | 3 | 2 | 3 | 2 | 0 | 6 | 2 | 5 + 3 | 22 (14) |
| 015 | minor | 4 | 5 | 2 | 2 | 3 | 2 | 0 | 4 | 3 | 5 + 2 | 20 (12) |
| 016 | major | 4 | 5 | 2 | 2 | 3 | 2 | 0 | 6 | 3 | 5 + 4 | 13 (8) |
| 017 | major | 3 | 5 | 3 | 2 | 3 | 2 | 0 | 4 | 2 | 6 + 0 | 11 (7) |
| 018 | major | 3 | 5 | 3 | 2 | 3 | 2 | 1 | 5 | 4 | 8 + 2 | 17 (13) |
Table 5: In-loop review (Claude Opus 5.5 with the run records) of the first drafts: verdict, scores (1–5) on correctness, verifiability, novelty, generality, usefulness and clarity, findings by severity, the agent’s response (adopted + partly adopted), and claims in the final paper (exploratory in parentheses).
| Run | Answer in the final paper | Main limitation stated by the paper |
|---|---|---|
| 014 | Reference and control corrections of annual-embedding change scores did not consistently improve the screening of one-year tree-cover gains and losses; a directly trained classifier scored higher, with unequal label resources. | Small one-year tests; exploratory changes of the data partition reversed all correction contrasts; performance on the native grid is unverified; multi-year restoration and ecological recovery are not addressed. |
| 015 | Published single-site exceedance probabilities leave wide bounds on the probability that at least of a set of sites exceed a threshold; one aggregate moment narrows them, and all pairwise co-exceedance probabilities nearly determine it. | The near-identification is conditional on exact moments of the 50-member empirical forecast law; no observed-weather skill, health benefit or realised economic value was measured. |
| 016 | Revisiting the same reference locations gave much smaller errors in the annual change of declared crop-group diversity than independent locations, but predicting no change was better still at every budget; assistance from satellite embeddings gave no or modest gains. | Declaration labels are treated as perfectly repeatable on a fixed frame; field-audit errors, costs and ecological outcomes were not measured. |
| 017 | With patch-first allocation of training points, annual embeddings recovered more small inventory wetland patches than seasonal optical imagery in the planned split, but the gain did not transfer reliably between projects. | Intervals exclude the uncertainty of split selection; the endpoint is agreement with the mapped inventory, not ecological completeness or savings in annotation cost. |
| 018 | An increment of embedding distances over terrain, surface-water and drainage covariates for matching wetland water regimes was not established: planned permutation controls gave changes comparable to the real distances. | Convenience windows; the endpoint is agreement with inventory labels, not field hydroperiod or benefit to practitioners. |
Table 6: What each final paper reports, summarised from its abstract: the answer to its question and the main limitation the paper itself states. Numbers are left out; they are in the papers.
| Reviewer | Run | Corr. | Verif. | Nov. | Gen. | Use. | Clar. | Overall | Verdict | WS | Main |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini 4 Argon | 014 | 4.0 | 4.0 | 3.0 | 2.0 | 3.0 | 2.0 | 5/5 | M/M | Y/Y | N/N |
| 015 | 5.0 | 5.0 | 3.0 | 3.0 | 3.0 | 3.0 | 6/6 | m/m | Y/Y | B/B | |
| 016 | 4.5 | 5.0 | 3.0 | 2.5 | 3.5 | 2.0 | 6/5 | M/M | Y/Y | B/N | |
| 017 | 4.0 | 4.0 | 3.0 | 3.0 | 3.0 | 2.5 | 6/5 | m/M | Y/Y | B/N | |
| 018 | 4.5 | 3.5 | 3.0 | 2.5 | 3.0 | 2.0 | 5/5 | M/M | Y/Y | N/N | |
| mean | 4.4 | 4.3 | 3.0 | 2.6 | 3.1 | 2.3 | 5.4 |
Table 7: External AI evaluation of the final papers (PDF, paper and web search only, two reviews per paper). Axis scores are means over the two reviews; “Overall” lists the two ICLR-scale scores; verdicts are m (minor revision), M (major revision) or R (reject); WS and Main are workshop-level and main-track-level judgments, Y (yes), B (borderline) or N (no).
More in this topic
Autonomous Research Agents
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
See all 326 papers