This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search.
Figures & tables
Figure 1: RL-based agents (a) condition solely on the current episode, and episodes are independent , whereas meta-RL-based agents (b) leverage context accumulated across episodes. MR-Search performs sequential self-reflection over past episodes to guide exploration in subsequent episodes. In MR-Search , we have inner-episodes, each consisting of a maximum of T interactions steps with an answer. A sequence of N episodes forms a meta-episode.
Figure 2: An overview of our proposed MR-Search framework. Given a question, the agent first completes an initial episode by interleaving reasoning and tool calls. It then enters an iterative self-reflection loop, where previous episodes serve as experience to inform subsequent searches and answer revisions, enabling iterative improvement across episodes.
Figure 3: We evaluate MR-Search , Search-R1 with sequential reflection inference (Search-R1-S), and Search-R1 with parallel sampling (Search-R1-P), selecting the most frequent answer among the generated trajectories. Shaded regions show the standard deviation across 3 runs. We observe that MR-Search achieves the best performance. See § 4.3 for details
Method
Single-Hop QA
Multi-Hop QA
NQ
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Bamboogle
Avg.
Qwen2.5-3b
Direct Inference
10.6
28.8
10.8
14.9
24.4
2.0
2.4
13.4
Search-o1
23.8
47.2
26.2
22.1
21.8
5.4
32.0
25.5
ReSearch
42.7
59.7
43.0
30.5
27.2
7.4
11.5
30.4
Search-R1
46.2
62.2
45.6
32.6
31.0
7.7
17.6
34.7
PPRM
42.3
56.5
41.1
35.3
34.0
12.7
28.0
35.7
Table 1: Main accuracy (%) on search-based QA benchmarks. The best results are marked in boldface . We compare with baselines that rely on outcome rewards (ReSearch and Search-R1) and those that use process rewards with external models (PPRM and StepResearch).
Figure 4: Test performance, training curves of reward and search frequency on ASearcher, evaluated with Qwen2.5-7B-Base. Additional results are provided in Appendix A.2.1 .
Figure 5: MR-Search , Search-R1 with sequential reflection turns (Search-R1-S), and Search-R1 with parallel sampling (Search-R1-P), selecting the most frequent answer among them.
Method
Single-Hop QA
Multi-Hop QA
NQ
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Bamboogle
Avg.
ReSearch
36.6
60.5
39.1
37.8
38.6
16.6
37.6
38.1
Search-R1
46.4
64.1
44.8
43.0
42.5
19.5
44.0
43.5
MR-Search w. γ=0
49.3
63.9
45.6
44.3
41.7
19.3
42.6
43.8
MR-Search w. PPO
43.9
63.7
42.5
41.3
40.5
18.6
43.5
42.0
MR-Search w. MT-GRPO
46.1
64.5
44.5
44.7
44.2
20.7
45.1
44.3
Table 2: Ablation of discount factor and training algorithm evaluated on Qwen2.5-7B-Base.
Method
NQ
TriviaQA
PopQA
HotpotQA
2wiki
Musique
Bamboogle
ASearcher
Search-R1
45.9
63.2
44.9
43.9
38.7
18.1
40.0
36.9
MR-Search
50.2
66.6
47.2
46.8
43.6
22.1
45.2
41.3
MR-Search Exploration
48.3
65.1
46.4
44.7
39.4
21.8
44.0
43.2
MR-Search Step Level
48.6
64.6
45.7
42.3
41.4
16.3
41.6
38.4
MR-Search Short Context
48.1
65.9
45.2
44.6
41.0
19.3
47.2
40.5
Table 3: Comparison with variants of MR-Search : encouraging exploration, MR-Search at the step level and context management (keeping one preceding episode as context).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9Figure 10
Figure 6: The training dynamics of MR-Search and Search-R1 in terms of test accuracy, training accuracy, and the number of tool calls on Qwen2.5-3B-Base.
Figure 8: The idea of MR-Search can be extended beyond full trajectories to any semantically meaningful episode that can be explicitly segmented. For example, in agentic search, each tool-interaction step can naturally form an episode. We can decompose the one interaction process into multiple self-reflective trials, where each tool-interaction step contributes a candidate answer to the original query via prompting. During training, at each interaction step, we monitor changes in the agent’s accuracy with respect to the target and use this signal as a dense training objective to promote effective exploration.
Question : During a major geopolitical event in the early Cold War, Robert Menzies exploited rising tensions to portray the Labor Party as ’soft on Communism,’ leading to the defeat of a Prime Minister who attended a religious school. In the same period, which British maritime patrol aircraft, known for its role in anti-submarine warfare and derived from the world’s first jet airliner, was frequently deployed to a U.S. military base in the Atlantic to support anti-submarine operations during the latter stages of the Cold War?
Ground Truth : 2002
MR-Search :
<think> To determine the year in which the Victorian government department responsible for agriculture and energy policy was established, we need more information about the department and its merger with another department in 2013. </think>
<search> Query: Victorian government department responsible for agriculture and energy policy merged with another department administering conservation strategy involving biodiversity offsets in 2013. When was this department established? </search>
<information> Document (Title: Department of the Environment (Australia, 2013–16)) by way of an Administrative Arrangements Order issued on 18 September 2013. It absorbed the responsibilities of the former Department of Sustainability, Environment, Water, Population and Communities (DSWEPaC) and climate change from the former Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education. The department was dissolved in July 2016 and its functions, along with energy policy functions, were moved to the newly established Department of the Environment and Energy. From when it… </information>
<think> It seems the search did not directly provide the information. Let’s try another search with a specific focus on the department related to agriculture and energy policy: </think>
Appendix
Table 4: Case study 1: An early intermediate answer is incorrect due to ambiguous retrieval results, but is subsequently reconsidered through additional search and self-reflection, leading to the correct final answer.
<search> Query: When was the Department of Environment and Primary Industries, which collaborated with Field and Game Australia to establish the Game Council of Victoria in 2012, merged with another department in 2013? </search>
<information> Document (Title: Department of Primary Industries (Victoria)) Bureau of Animal Welfare branch Does this Department need investigation if Rosewarne had anything to do with it? DPI had primary responsibility for administering legislation governing agriculture, animal welfare, fisheries, hunting and energy and earth resources in Victoria. The department administered and responded to queries regarding 38 major Acts and is responsible for collecting fees under regulations for fishing licences, mining rights, and exploration licences. In 2012 DPI also created the Game Council of Victoria to with the Bureau of Animal Welfare and organisations such as Field and Game Australia to manage sustainable and responsible hunting in Victoria… </information>
<think> This information indicates that the Department of Environment and Primary Industries was merged with the Department of Sustainability and Environment to form the Department of Environment and Primary Industries in April 2013. However, we need to confirm if this is the department responsible for agriculture and energy policy. Given the context, it seems likely that the Department of Environment and Primary Industries was the one responsible for agriculture and energy policy before the merger. Therefore, the year in which the Victorian government department responsible for agriculture and energy policy was established is: </think>
<answer> 2002 </answer>
Reflect on current answer to the question and provide an improved answer by searching for additional external information using search engines.
<think> Let’s verify the year of establishment for the Department of Environment and Primary Industries, which was merged in 2013 and was responsible for agriculture and energy policy: </think>
<search> Query: When was the Department of Environment and Primary Industries established in Victoria? </search>
Appendix
Table 14
Question : During a major geopolitical event in the early Cold War, Robert Menzies exploited rising tensions to portray the Labor Party as ’soft on Communism,’ leading to the defeat of a Prime Minister who attended a religious school. In the same period, which British maritime patrol aircraft, known for its role in anti-submarine warfare and derived from the world’s first jet airliner, was frequently deployed to a U.S. military base in the Atlantic to support anti-submarine operations during the latter stages of the Cold War?
Ground Truth : Hawker Siddeley Nimrod
MR-Search :
<think> To determine the name of the British maritime patrol aircraft that was frequently deployed to a U.S. military base in the Atlantic to support anti-submarine operations during the latter stages of the Cold War, let’s first consider the time frame of the event involving Robert Menzies and the Prime Minister who attended a religious school. Then we can perform a search to find the information about the British maritime patrol aircraft. </think>
Appendix
Table 5: Case study 2: MR-Search subsequently revisits the problem through additional search and self-reflection, ultimately producing the correct final answer.
<search> Query: British maritime patrol aircraft derived from the world’s first jet airliner, frequently deployed to a U.S. military base in the Atlantic during the Cold War </search>
<information> Document (Title: Maritime patrol aircraft) also been located in these areas: U.S. Navy and Canadian aircraft based in Greenland, Iceland, and Newfoundland; British aircraft based in Scotland and Northern Ireland; and Norwegian, Dutch, and German aircraft based in their home countries. Since the end of the Cold War the threat of a large-scale submarine attack is a remote one, and many of the air forces and navies have been downsizing their fleets of … </information>
<think> Let’s try one more time with the exact model derived from the first jet airliner: </think>
<search> Query: Air aztros 2000 maritime patrol aircraft frequently deployed to a U.S. military base in the Atlantic during the Cold War </search>
<information> Document (Title: Kindley Air Force Base) P-3B aircraft from various bases in the eastern United States, as well as NATO/Allied support consisting of Royal Air Force Hawker Siddeley Nimrod MR2s, Canadian Armed Forces CP-140 Auroras and other similar maritime patrol and reconnaissance aircraft from other NATO nations. During one period in 1985 that was characterized by exceptionally heavy Soviet Navy submarine activity off the … </information>
<think> This information does not directly name the aircraft. However, the mention of P-3C aircraft being frequently deployed to a U.S. military base in the Atlantic suggests that we should focus on this model. Given that P-3C is a well-known maritime patrol aircraft and the question specifies a connection to the first jet airliner, let’s consider if P-3C has any such connection. The P-3C is derived from the Lockheed P-3 Orion, which is itself a derivative of the Lockheed Model 85 Electra airliner. The first jet airliner was the de Havilland Comet, not directly related to the Electra. Therefore, we might need to re-evaluate our query to ensure we are considering the correct aircraft. Given the context and the need for accuracy, let’s consider the aircraft derived from the de Havilland Comet, which is the de Havilland Sea Vixen. However, the P-3C is more directly related to the Lockheed 1049C Hercules transport aircraft, which is not derived from the Comet but rather the Lockheed Electra. Given the specificity of the question, it seems likely that the aircraft in question is the P-3C Orion, which is frequently deployed and fits the timeframe. </think>
<answer> P-3C Orion </answer>
Appendix
Table 16
Question : Who directed the 1909 Italian silent film in which the actor who played the castle castellan in the 1928 German silent film The Mysterious Mirror portrayed the titular character of Shakespeare’s Macbeth?
Ground Truth : Mario Caserini
MR-Search :
<think> To determine who directed the 1909 Italian silent film in which the actor who played the castle castellan in the 1928 German silent film ’The Mysterious Mirror’ portrayed the titular character of Shakespeare’s ’Macbeth’, let’s first identify the actor and then find the Italian silent film he appeared in during 1909. </think>
<search> Query: Italian silent film from 1909 in which the actor who played the castle castellan in the 1928 German silent film The Mysterious Mirror portrayed the titular character of Shakespeare’s ’Macbeth’ directed by whom? </search>
<information> Document (Title: Macbeth (1916 film)) Macbeth (1916 film) Macbeth is a silent, black-and-white 1916 film adaptation of the William Shakespeare play Macbeth. It was directed by John Emerson, assisted by Erich von Stroheim, and produced by D. W. Griffith, with cinematography by Victor Fleming. The film starred Herbert Beerbohm Tree and Constance Collier, both famous from the stage and for playing Shakespearean parts. Although released during the first decade of feature filmmaking, it was already the seventh version of Macbeth to be produced, one of eight during the silent film era. It is considered to be a lost film. In the companion book to his … </information>
<think> It appears that the search did not directly provide the name of the actor. Let’s try the search again with a focus on the actor who played the castle castellan in The Mysterious Mirror and then find the film he portrayed Macbeth in 1909. </think>
Appendix
Table 6: Case study 3: MR-Search subsequently revisits the problem through additional search and self-reflection, ultimately producing the correct final answer.
<information> Document (Title: Macbeth (1916 film)) Macbeth (1916 film) Macbeth is a silent, black-and-white 1916 film adaptation of the William Shakespeare play Macbeth. It was directed by John Emerson, assisted by Erich von Stroheim, and produced by D. W. Griffith, with cinematography by Victor Fleming. The film starred Herbert Beerbohm Tree and Constance Collier, both famous from the stage and for playing Shakespearean parts. Although released during the first decade of feature filmmaking, it was already the seventh version of Macbeth to be produced, one of eight during the silent film era. It is considered to be a lost film. In the companion book to his … </information>
<think> This information does not directly name the director. However, it mentions that the 1909 film Macbeth was directed by Maria Caserini’s husband. Given the previous context, let’s consider the most likely candidate based on the available information: </think>
<answer> Mario Caserini </answer>
Reflect on current answer to the question and provide an improved answer by searching for additional external information using search engines.
<think> Let’s re-evaluate the information we have. We know that the 1909 Italian silent film Macbeth was directed by Maria Caserini’s husband. Since the previous searches did not provide the name of her husband, let’s consider if we can infer this information from the context or search for it directly: </think>
<search> Query: Who was the husband of Maria Caserini who directed the 1909 Italian silent film Macbeth? </search>
<information> Document (Title: Maria Caserini) Maria Caserini Maria Caserini (née Gasperini; 24 July 1884 – 15 April 1969) was an Italian stage and film actress, as well as a pioneer of filmmaking during the early 20th-century. She often starred in adaptations of stage and film productions for the works of William Shakespeare. Born in Milan, she started acting in stage productions at a young age. Her first film was in 1906, in what is believed to be the earliest film adaptation of Shakespeare’s Othello, titled Otello, which was directed by her husband, Mario Caserini. She appeared in a film adaptation of Romeo and Juliet in … </information>
Appendix
Table 18
Question : In which country did Itamar Zorman make his first public appearance at a renowned festival in the early 2010s, and where is he scheduled to perform a concerto at a historic venue in a city with a rich Norman heritage and a famous opera house?
Ground Truth : Switzerland
MR-Search :
<think> To determine the country where Itamar Zorman made his first public appearance at a renowned festival in the early 2010s and where he is scheduled to perform a concerto at a historic venue in a city with a rich Norman heritage and a famous opera house, let’s first search for information about Itamar Zorman’s early public appearances and then find the details about his upcoming concert. </think>
<search> Query: Itamar Zorman first public appearance at a renowned festival in the early 2010s and upcoming concerto at a historic venue in a city with rich Norman heritage and famous opera house </search>
<information> Document (Title: Itamar Zorman) Hamburg and Kronberg Academy Festival in Frankfurt as well as with Yuri Bashmet. His first CD is planned to be released sometime in 2014 with his recitals the Louvre. In November of the same year he, Daniel Oren and Musicians from Marlboro group, will perform Ludwig van Beethoven’s concerto at the Teatro Massimo in Palermo. For the past five years he has traveled through Israel and North America during which he was a founder of the Israeli Chamber Project as well as a Lysander Piano Trio member. On March 9, 2014 he performed one of Johannes Brahms’ sonatas in Detroit … </information>
Appendix
Table 7: Case study 4: MR-Search subsequently revisits the problem through additional search and self-reflection, ultimately producing the correct final answer.
<think> Itamar Zorman made his first public appearance at the Verbier Festival in Switzerland in 2010. Now, let’s find out where he is scheduled to perform a concerto at a historic venue in a city with a rich Norman heritage and a famous opera house. </think>
<search> Query: Itamar Zorman concerto at a historic venue in a city with rich Norman heritage and famous opera house </search>
<information> Document (Title: Semperoper) Semperoper The Semperoper is the opera house of the Sächsische Staatsoper Dresden (Saxon State Opera) and the concert hall of the Staatskapelle Dresden (Saxon State Orchestra). It is also home to the Semperoper Ballett. The building is located near the Elbe River in the historic centre of Dresden, Germany. The opera house was originally built by the architect Gottfried Semper in 1841. After a devastating fire in 1869, the opera house was rebuilt, partly again by Semper, and completed in 1878. The opera house has a long history of premieres, including major works by Richard Wagner and Richard Strauss. The … </information>
<think> This information does not directly state the location of the upcoming concert. However, it mentions the Teatro Massimo in Palermo, which is in Italy. Let’s verify if this matches the criteria of a city with a rich Norman heritage and a famous opera house. </think>
<search> Query: Teatro Massimo in Palermo, Italy </search>
<information> Document (Title: Teatro Massimo) Teatro Massimo The Teatro Massimo Vittorio Emanuele is an opera house and opera company located on the Piazza Verdi in Palermo, Sicily. It was dedicated to King Victor Emanuel II. It is the biggest in Italy, and one of the largest of Europe (the third after the Opéra National de Paris and the K. K. Hof-Opernhaus in Vienna), renowned for its perfect acoustics. An international competition for the creation of the opera house was announced by the Palermo Council in 1864 at the instigation of the mayor, Antonio Starrabba di Rudinì. For many years there had been talk of building … </information>
<think> The Teatro Massimo in Palermo is indeed a famous opera house. Now, let’s confirm if Palermo has a rich Norman heritage: </think>
Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.
Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis +2
University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited
Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization.
Deep search has recently emerged as a promising paradigm for enabling agents to retrieve fine-grained historical information without heavy memory pre-managed. However, existing deep search agents for memory system repeat past error behaviors because they fail to learn from the prior high- and low-quality search trajectories. To address this limitation, we propose R^2-Mem, a reflective experience framework for memory search systems. In the offline stage, a Rubric-guided Evaluator scores low- and high-quality steps in historical trajectories, and a self-Reflection Learner distills the corresponding abstract experience. During the online inference, the retrieved experience will guide future search actions to avoid repeated mistakes and maintain high-quality behaviors. Extensive experiments demonstrate that R^2-Mem consistently improves both effectiveness and efficiency over strong baselines, improving F1 scores by up to 22.6%, while reducing token consumption by 12.9% and search iterations by 20.2%. These results verify that R^2-Mem provides a RL-free and low-cost solution for self-improving LLM agents.