Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agents

Ankur Samanta; Yonathan Efroni; Paul Sajda; Kaveh Hassani; Anirudh Goyal

cs.AIOct 1, 2026

Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agents

Authors: Ankur Samanta, Yonathan Efroni, Paul Sajda, Kaveh Hassani, Anirudh Goyal

Organizations: Meta AI · Columbia University · Work done at Meta · Tel Aviv University

Abstract

Long-horizon research agents must decide both how to investigate and what to investigate next as evidence accumulates. This is hard to learn because such decisions are sparse in long execution traces, and their consequences may emerge several investigations later. We introduce Meta-reasoning for Iterative Research Agents (MIRA), a hierarchical architecture separating research allocation from execution. An outer-loop meta-reasoner curates context from a persistent research record, then writes a work order for the next investigation or ends the episode. A fresh inner-loop executor carries out each work order, making execution part of the transition between meta-reasoning actions. Without policy training, MIRA improves long-horizon inference and allocates additional compute more effectively in theorem proving and open-ended neural-architecture research. Its decision boundaries also provide natural units for credit assignment. At each boundary, we train a generative critic to forecast expected remaining return from partial states, outperforming token-level alternatives. Cross-environment pretraining improves forecasting and adaptation, yielding a transferable prior for valuing partial progress. We use this prior to initialize MIRA-AC, a generative actor-critic jointly trained to forecast remaining return and choose the next investigation, without a separate critic model. MIRA-AC concentrates policy optimization on meta-reasoning decisions, enabling efficient long-horizon reinforcement learning without directly optimizing the longer execution traces they initiate. Training MIRA-AC on the model's own proxy hill-climbing signals improves gold performance across four autoresearch environments; the actor transfers with cross-environment value initialization. Together, these results show that meta-reasoning can be learned as an explicit policy for directing long-horizon autonomous research.

Figures & tables

Figure 1: MIRA separates research allocation from execution. At each research-decision boundary, the outer loop reads the persistent research record

H_{t}

and curates evidence bearing on the most consequential unresolved uncertainty into an observation

o_{t}

. It then produces a meta-reasoning action

a_{t}

, a complete decision response comprising its reasoning followed by either a work order or a stopping decision. For a work order, a fresh executor carries out the investigation and writes its results back to the persistent record, producing

H_{t+1}

. The next outer-loop observation is then constructed from this updated record; a stopping decision instead terminates the research episode. Thus, each investigation, however long, is represented as one transition at the meta-reasoning level.

Component	Role	Trained?
Curator $C$	Constructs $o_{t}$	No
Policy $\pi_{\theta}$	Generates reasoning and work order / termination	Yes
Value critic	Forecasts remaining return	Yes
Executor $\mathcal{E}$	Implements investigation	No
Tools / evaluators	Environment interaction	No
Git substrate $H_{t}$	Persistent research state	—

Table 1: Optimization boundary in MIRA-AC.

Figure 2: Forecasting remaining research progress. The critic receives the same curated observation

o_{t}

as the policy and forecasts a distribution over remaining return. Its expectation is

V_{\phi}(o_{t})

Figure 3: Decoupled buffers allow MIRA-AC to learn asynchronously from variable-duration research episodes. (a) Synchronous cohorts leave rollout slots idle while waiting for slow episodes. (b) MIRA-AC restarts freed slots immediately and updates the shared model whenever either the actor or critic buffer fills; live episodes adopt new weights at the next meta-reasoning boundary. (c) The same language model serves as actor and critic, forecasting remaining return and generating the next meta-reasoning step.

Panel of Figure 4 — Figure 4: MIRA improves long-horizon theorem proving and neural-architecture hill climbing, outperforming baselines. (a) Theorem-proving performance at natural stopping and under output-token-matched continuation. (b) Best loss observed during the matched 24-hour architecture-research campaigns.

Figure 5: MIRA more effectively allocates additional inference across proof directions. On IMOProofBench Advanced Problem 23, GPT-5.5 and Codex pursue a small number of proof directions, while MIRA tests several directions, records refutations, and reaches the accepted proof.

Figure 6: MIRA with GPT-6 Astra improves the Loop Transformer by adding signed attention and a leaky recurrent update, outperforming Codex in architectural hill climbing. Across matched 48-hour campaigns, MIRA progresses from separating fast and slow states and retrieving prior iterations to testing an inhibitory attention branch, scaling the successful operator, and selecting the combined architecture. The resulting model achieves lower proxy and gold negative log-likelihood (NLL) than Codex and the reference implementation. Curves track the best proxy NLL observed over time; faint markers show all evaluations, and hollow markers show where each run ends. Callouts identify major changes in MIRA’s research direction. The stacked legend reports each system’s final proxy and gold NLL.

Figure 7: MIRA keeps its working context bounded and current as the repository grows. (a) Curation bounds the working state as the repository grows. (b) Its evidence mix changes with the active research direction. (c) Recent artifacts and the latest handover maintain continuity across context resets. The dashed line marks final candidate selection.

Figure 8: Decision-level generative forecasting and cross-environment pretraining provide a stronger prior for partial progress. (a) The decision-level generative critic achieves the lowest held-out RMSE, and (b) LOO pretraining achieves the lowest RMSE at every target-support size, improving zero-shot forecasting and adaptation to held-out environments. (95% CIs from a paired bootstrap over support/query split seeds.)

		Symbolic Regression		Physics Discovery		HillClimbBench-CPU		BNLearn
	Training variant	Solved	$R^{2}$	Solved	Law qual.	Wins	Arch. qual.	Solved	SHD gain
	MIRA (untrained)	50	0.9864	8	0.2118	—	0.7624	76	0.9672
Gold binary	Token regression	55	0.9957	9	0.2327	49	0.7594	79	0.9567
	Step generative	58	0.9993	9	0.2733	43	0.7518	82	0.9804
	MIRA-AC	65	0.9879	6	0.1846	49	0.7481	56	0.9261
Proxy marginal	Token regression	55	0.9860	6	0.2463	49	0.7618	63	0.9002

Table 2: Autoresearch gains from training MIRA’s meta-reasoning policy, measured by gold evaluation. The primary outcomes are Solved (environment-specific successes) and Wins (paired HillClimbBench-CPU wins over untrained MIRA); adjacent columns report secondary metrics.

R^{2}

is sealed-test coefficient of determination, Law qual. combines Physics trajectory fit and explanation quality, Arch. qual. is the mean task-specific HillClimbBench architecture score, and structural Hamming distance (SHD) gain is the normalized closure of the directed-SHD gap; higher is better and all are defined in App. B . Gold binary and Proxy marginal denote training feedback, not evaluation. Italics mark Gold-binary group bests, bold marks Proxy-marginal group bests, and boxes mark the best overall counts. The final row trains one MIRA-AC policy across all environments except BNLearn, which is held out from actor training but included in critic pretraining.

Figure 9: MIRA-AC’s meta-reasoning is more often judged to make the stronger contribution to gold outcomes and to target unresolved uncertainty. Results cover 400 matched pairs across four environments. (a) Which outer policy was judged to have contributed more to its own gold outcome; 66 comparisons were tied or unclear. (b) Share of trajectories assigned each behavior label. The first row reports reasonable exploratory bets that improved the gold result; the remaining rows report behaviors attributed at least partly to the outer policy. Each trajectory contributes at most once per row. Lower is preferable for local-search stagnation, defined as repeatedly trying minor variants of an unproductive idea.

Figure 10: MIRA-AC concentrates actor learning on research decisions. Across the four environment-specific training runs (400 episodes), complete meta-reasoning decision responses comprise 25.8% of accounted decision and execution output tokens and are the only generated tokens receiving actor loss. Left: one complete SRBench PO30 episode. Right: per-episode token shares across the full training set.

Figure 11: Curation keeps active contexts compact and deliberately focused as research histories grow. Across 400 recorded MIRA-AC trajectories, we compare accounted input-token use with projected replay of prior context either in full or with auto-compaction at 200k tokens. (a) Mean input tokens per episode. (b) Per-iteration input tokens for an example SRBench trajectory. Points show recorded MIRA-AC usage; bars span projected replay costs under different assumptions about how often retained history is reread. SR denotes SRBench, and BN denotes BNLearn.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: MIRA explores multiple proof directions and reaches the full-credit proof on PB-Advanced-023. Panels show (a) GPT-5.5, (b) Codex, and (c) MIRA; all conditions use GPT-5.5. Dashed red edges denote refutations.

Figure 13: MIRA develops and combines multiple proof branches into a complete classification on PB-Advanced-006. GPT-5.5 and Codex identify candidate functions but leave central steps unsupported or incomplete, whereas MIRA produces a complete proof receiving 7/7. Systems and notation follow Fig. 12 .

Figure 14: MIRA redirects from a refuted

L=4

approach and completes an

L=3

proof on PB-Advanced-018. GPT-5.5 remains within the unsuccessful

L=4

direction, while Codex pivots to

L=3

but leaves the proof incomplete. MIRA develops the redirected approach into a proof receiving 7/7. Systems and notation follow Fig. 12 .

Figure 15: MIRA explores competing combinatorial directions and synthesizes a correct proof on PB-Advanced-030. GPT-5.5 and Codex terminate in unsuccessful proof directions, whereas MIRA explores allocation, deletion, and induction arguments before reaching a complete 7/7 proof. Systems and notation follow Fig. 12 .

Figure 16: Architecture progression for the Loop Transformer study. A standard Transformer obtains depth from blocks with distinct parameters. Huginn instead repeatedly applies one weight-tied core while reinjecting the input representation. The MIRA-selected architecture retains this recurrent structure and adds what the campaign calls signed attention, which contrasts two attention reads over shared values. It also interpolates between the previous and proposed states at each recurrent iteration—a leaky recurrent update. Blue denotes weight-tied recurrence, dashed green denotes input reinjection, and orange denotes components introduced by MIRA.

Setting	Value
Base model	Qwen3.6-27B, revision 6a9e13bd6fc8 ; policy thinking enabled and critic thinking disabled
Initialization	Weights-only initialization from a proxy-marginal decision-level generative critic pretrained jointly across four environments
Actor-training distribution	SRBench, Physics Discovery, and HillClimbBench-CPU; 100 episodes per environment, one epoch, run seed 42
LoRA	Rank 32, scale 64, dropout 0; shared actor–critic adapter; online critic updates restricted to MLP projections
Research and sequence limits	20 research decisions; 262,144-token maximum sequence; 8,192-token maximum generation
Policy sampling	Temperature 1.0, top- $p$ 0.95, top- $k$ 20, minimum- $p$ 0, presence penalty 0, repetition penalty 1

Appendix

Table 3: General MIRA-AC training configuration. Settings are from the reported epoch-one, version-94 cross-environment run.

Figure 18: MIRA-AC achieves a lower best-observed loss than untrained MIRA on 33 of 50 HillClimbBench-CPU tasks. Each panel shows, for each policy, the better of two matched rollouts as measured by best-observed raw loss. Faint points show successful evaluations, staircases track the best loss observed so far, and the horizontal dashed line marks the reference implementation.

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 299 papers

Sep 29, 2026

Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning

As agents take on longer and more complex problems, controlling the execution becomes a task in its own right. Each step in the run brings new control choices, like which partial work to build on, whether to start fresh, or when to stop. We introduce agentic meta-reasoning, an inference-time harness that makes these choices an explicit and structured reasoning process. Workers carry out the task-level computation, while a controller consolidates what the run has established, explores next options, assesses what each option is worth under the remaining budget, and dispatches the chosen work with context drawn from persistent memory. Between decisions the controller carries only a compact account of the run rather than replaying its full history. Our baselines span production coding agents and research harnesses, together with a Direct Control Agent using the same workers and compute budget allowance. On ProgramBench, which tests long-horizon agentic capability through program reconstruction, meta-reasoning achieves 71.5% with GPT-5.5 against 58.0% for Codex; with Opus 4.8 it achieves 67.2% against 65.5% for Claude Code. On the other benchmarks, spanning abstract reasoning, multi-domain long-horizon reasoning, and proof generation, it gains between 3.6 and 4.2 points over direct control, averaged across three frontier models. It keeps improving over the tested budget ranges where direct control plateaus, though its overhead can hurt at small budgets. Artifact-graph analysis reveals more reuse of earlier work, higher coverage of correct solutions in most settings, and nonuniform gains in final selection. These results indicate that spending computation on structured control becomes more important as agents scale to longer runs.

Paras Dahal, Anton Bakhtin, Taco Cohen +9

Meta Superintelligence Labs

Agentic Reasoning Inference-Time Compute Allocation Long-Horizon LLM Agents LLM Agent Harnesses

Ask PDF

Mar 16, 2026

MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verification

We present MiroThinker-1.7, a new research agent designed for complex long-horizon reasoning tasks. Building on this foundation, we further introduce MiroThinker-H1, which extends the agent with heavy-duty reasoning capabilities for more reliable multi-step problem solving. In particular, MiroThinker-1.7 improves the reliability of each interaction step through an agentic mid-training stage that emphasizes structured planning, contextual reasoning, and tool interaction. This enables more effective multi-step interaction and sustained reasoning across complex tasks. MiroThinker-H1 further incorporates verification directly into the reasoning process at both local and global levels. Intermediate reasoning decisions can be evaluated and refined during inference, while the overall reasoning trajectory is audited to ensure that final answers are supported by coherent chains of evidence. Across benchmarks covering open-web research, scientific reasoning, and financial analysis, MiroThinker-H1 achieves state-of-the-art performance on deep research tasks while maintaining strong results on specialized domains. We also release MiroThinker-1.7 and MiroThinker-1.7-mini as open-source models, providing competitive research-agent capabilities with significantly improved efficiency.

MiroMind Team, S. Bai, L. Bing +41

Autonomous Research Agents Long-Horizon LLM Agents Agentic Reasoning LLM Agent Reliability Reasoning Verification

Ask PDF

Oct 13, 2026

Failure Recovery via Belief Updates and World Modeling for Long-Horizon Agents

Long-horizon agents often struggle to recover from failures whose causes lie several steps earlier, beyond what local reflection or backtracking can resolve. We introduce BELIEFWEAVER, a framework that combines persistent evidence across attempts with prospective reasoning over repair directions. It maintains a relational belief state over frontiers, attempts, observations, and checkpoints. Execution evidence updates related frontier priorities across attempts and rollback, while diagnostic evidence gates execution. An implicit world model (IWM) evaluates repair hypotheses, predicts progress, and identifies relations among frontiers to guide exploration. We further distill these recurring IWM reasoning components into a 9B student through supervised fine-tuning and on-policy distillation (OPD). Across three terminal-based benchmarks, BELIEFWEAVER improves aggregate final score by 5.1 percentage points over native Terminus-2 continuation, with positive trends in exploration efficiency. The distilled 9B student improves final score by 9.8 percentage points over the reported native baseline while reducing estimated teacher API cost by 54.8% relative to the matched teacher. These results show that persistent evidence and prospective reasoning can improve recovery in long-horizon agent tasks while substantially reducing reliance on teacher-model inference.

Xiyang Wu, Zhichao Liu, Shengxin zhang +4

University of Maryland · Google · MBZUAI

Long-Horizon LLM Agents Long-Horizon Reasoning Memory-Augmented Agents Belief-Space Planning LLM World Models

Ask PDF

Abstract

Autonomous Research Agents

Explore similar work

Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning

MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verification

Failure Recovery via Belief Updates and World Modeling for Long-Horizon Agents