cs.LGMar 11, 2026
SaveMeta-Reinforcement Learning with Self-Reflection for Agentic Search
Organizations: Allen Institute for AI · University of Washington · Independent
Abstract
This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search.
Figures & tables
Figure 1: RL-based agents (a) condition solely on the current episode, and episodes are independent , whereas meta-RL-based agents (b) leverage context accumulated across episodes. MR-Search performs sequential self-reflection over past episodes to guide exploration in subsequent episodes. In MR-Search , we have inner-episodes, each consisting of a maximum of interactions steps with an answer. A sequence of episodes forms a meta-episode.
Figure 2: An overview of our proposed MR-Search framework. Given a question, the agent first completes an initial episode by interleaving reasoning and tool calls. It then enters an iterative self-reflection loop, where previous episodes serve as experience to inform subsequent searches and answer revisions, enabling iterative improvement across episodes.
Figure 3: We evaluate MR-Search , Search-R1 with sequential reflection inference (Search-R1-S), and Search-R1 with parallel sampling (Search-R1-P), selecting the most frequent answer among the generated trajectories. Shaded regions show the standard deviation across 3 runs. We observe that MR-Search achieves the best performance. See § 4.3 for details
| Method | Single-Hop QA | Multi-Hop QA | |||||||
| NQ | TriviaQA | PopQA | HotpotQA | 2wiki | Musique | Bamboogle | Avg. | ||
| Qwen2.5-3b | Direct Inference | 10.6 | 28.8 | 10.8 | 14.9 | 24.4 | 2.0 | 2.4 | 13.4 |
| Search-o1 | 23.8 | 47.2 | 26.2 | 22.1 | 21.8 | 5.4 | 32.0 | 25.5 | |
| ReSearch | 42.7 | 59.7 | 43.0 | 30.5 | 27.2 | 7.4 | 11.5 | 30.4 | |
| Search-R1 | 46.2 | 62.2 | 45.6 | 32.6 | 31.0 | 7.7 | 17.6 | 34.7 | |
| PPRM | 42.3 | 56.5 | 41.1 | 35.3 | 34.0 | 12.7 | 28.0 | 35.7 | |
Table 1: Main accuracy (%) on search-based QA benchmarks. The best results are marked in boldface . We compare with baselines that rely on outcome rewards (ReSearch and Search-R1) and those that use process rewards with external models (PPRM and StepResearch).
Figure 4: Test performance, training curves of reward and search frequency on ASearcher, evaluated with Qwen2.5-7B-Base. Additional results are provided in Appendix A.2.1 .
Figure 5: MR-Search , Search-R1 with sequential reflection turns (Search-R1-S), and Search-R1 with parallel sampling (Search-R1-P), selecting the most frequent answer among them.
| Method | Single-Hop QA | Multi-Hop QA | ||||||
| NQ | TriviaQA | PopQA | HotpotQA | 2wiki | Musique | Bamboogle | Avg. | |
| ReSearch | 36.6 | 60.5 | 39.1 | 37.8 | 38.6 | 16.6 | 37.6 | 38.1 |
| Search-R1 | 46.4 | 64.1 | 44.8 | 43.0 | 42.5 | 19.5 | 44.0 | 43.5 |
| MR-Search w. | 49.3 | 63.9 | 45.6 | 44.3 | 41.7 | 19.3 | 42.6 | 43.8 |
| MR-Search w. PPO | 43.9 | 63.7 | 42.5 | 41.3 | 40.5 | 18.6 | 43.5 | 42.0 |
| MR-Search w. MT-GRPO | 46.1 | 64.5 | 44.5 | 44.7 | 44.2 | 20.7 | 45.1 | 44.3 |
Table 2: Ablation of discount factor and training algorithm evaluated on Qwen2.5-7B-Base.
| Method | NQ | TriviaQA | PopQA | HotpotQA | 2wiki | Musique | Bamboogle | ASearcher |
| Search-R1 | 45.9 | 63.2 | 44.9 | 43.9 | 38.7 | 18.1 | 40.0 | 36.9 |
| MR-Search | 50.2 | 66.6 | 47.2 | 46.8 | 43.6 | 22.1 | 45.2 | 41.3 |
| MR-Search Exploration | 48.3 | 65.1 | 46.4 | 44.7 | 39.4 | 21.8 | 44.0 | 43.2 |
| MR-Search Step Level | 48.6 | 64.6 | 45.7 | 42.3 | 41.4 | 16.3 | 41.6 | 38.4 |
| MR-Search Short Context | 48.1 | 65.9 | 45.2 | 44.6 | 41.0 | 19.3 | 47.2 | 40.5 |
Table 3: Comparison with variants of MR-Search : encouraging exploration, MR-Search at the step level and context management (keeping one preceding episode as context).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The training dynamics of MR-Search and Search-R1 in terms of test accuracy, training accuracy, and the number of tool calls on Qwen2.5-3B-Base.
Figure 8: The idea of MR-Search can be extended beyond full trajectories to any semantically meaningful episode that can be explicitly segmented. For example, in agentic search, each tool-interaction step can naturally form an episode. We can decompose the one interaction process into multiple self-reflective trials, where each tool-interaction step contributes a candidate answer to the original query via prompting. During training, at each interaction step, we monitor changes in the agent’s accuracy with respect to the target and use this signal as a dense training objective to promote effective exploration.
| Question : During a major geopolitical event in the early Cold War, Robert Menzies exploited rising tensions to portray the Labor Party as ’soft on Communism,’ leading to the defeat of a Prime Minister who attended a religious school. In the same period, which British maritime patrol aircraft, known for its role in anti-submarine warfare and derived from the world’s first jet airliner, was frequently deployed to a U.S. military base in the Atlantic to support anti-submarine operations during the latter stages of the Cold War? |
| Ground Truth : 2002 |
| MR-Search : |
| <think> To determine the year in which the Victorian government department responsible for agriculture and energy policy was established, we need more information about the department and its merger with another department in 2013. </think> |
| <search> Query: Victorian government department responsible for agriculture and energy policy merged with another department administering conservation strategy involving biodiversity offsets in 2013. When was this department established? </search> |
| <information> Document (Title: Department of the Environment (Australia, 2013–16)) by way of an Administrative Arrangements Order issued on 18 September 2013. It absorbed the responsibilities of the former Department of Sustainability, Environment, Water, Population and Communities (DSWEPaC) and climate change from the former Department of Industry, Innovation, Climate Change, Science, Research and Tertiary Education. The department was dissolved in July 2016 and its functions, along with energy policy functions, were moved to the newly established Department of the Environment and Energy. From when it… </information> |
| <think> It seems the search did not directly provide the information. Let’s try another search with a specific focus on the department related to agriculture and energy policy: </think> |
Table 4: Case study 1: An early intermediate answer is incorrect due to ambiguous retrieval results, but is subsequently reconsidered through additional search and self-reflection, leading to the correct final answer.
| <search> Query: When was the Department of Environment and Primary Industries, which collaborated with Field and Game Australia to establish the Game Council of Victoria in 2012, merged with another department in 2013? </search> |
| <information> Document (Title: Department of Primary Industries (Victoria)) Bureau of Animal Welfare branch Does this Department need investigation if Rosewarne had anything to do with it? DPI had primary responsibility for administering legislation governing agriculture, animal welfare, fisheries, hunting and energy and earth resources in Victoria. The department administered and responded to queries regarding 38 major Acts and is responsible for collecting fees under regulations for fishing licences, mining rights, and exploration licences. In 2012 DPI also created the Game Council of Victoria to with the Bureau of Animal Welfare and organisations such as Field and Game Australia to manage sustainable and responsible hunting in Victoria… </information> |
| <think> This information indicates that the Department of Environment and Primary Industries was merged with the Department of Sustainability and Environment to form the Department of Environment and Primary Industries in April 2013. However, we need to confirm if this is the department responsible for agriculture and energy policy. Given the context, it seems likely that the Department of Environment and Primary Industries was the one responsible for agriculture and energy policy before the merger. Therefore, the year in which the Victorian government department responsible for agriculture and energy policy was established is: </think> |
| <answer> 2002 </answer> |
| Reflect on current answer to the question and provide an improved answer by searching for additional external information using search engines. |
| <think> Let’s verify the year of establishment for the Department of Environment and Primary Industries, which was merged in 2013 and was responsible for agriculture and energy policy: </think> |
| <search> Query: When was the Department of Environment and Primary Industries established in Victoria? </search> |
| Question : During a major geopolitical event in the early Cold War, Robert Menzies exploited rising tensions to portray the Labor Party as ’soft on Communism,’ leading to the defeat of a Prime Minister who attended a religious school. In the same period, which British maritime patrol aircraft, known for its role in anti-submarine warfare and derived from the world’s first jet airliner, was frequently deployed to a U.S. military base in the Atlantic to support anti-submarine operations during the latter stages of the Cold War? |
| Ground Truth : Hawker Siddeley Nimrod |
| MR-Search : |
| <think> To determine the name of the British maritime patrol aircraft that was frequently deployed to a U.S. military base in the Atlantic to support anti-submarine operations during the latter stages of the Cold War, let’s first consider the time frame of the event involving Robert Menzies and the Prime Minister who attended a religious school. Then we can perform a search to find the information about the British maritime patrol aircraft. </think> |
Table 5: Case study 2: MR-Search subsequently revisits the problem through additional search and self-reflection, ultimately producing the correct final answer.
| <search> Query: British maritime patrol aircraft derived from the world’s first jet airliner, frequently deployed to a U.S. military base in the Atlantic during the Cold War </search> |
| <information> Document (Title: Maritime patrol aircraft) also been located in these areas: U.S. Navy and Canadian aircraft based in Greenland, Iceland, and Newfoundland; British aircraft based in Scotland and Northern Ireland; and Norwegian, Dutch, and German aircraft based in their home countries. Since the end of the Cold War the threat of a large-scale submarine attack is a remote one, and many of the air forces and navies have been downsizing their fleets of … </information> |
| <think> Let’s try one more time with the exact model derived from the first jet airliner: </think> |
| <search> Query: Air aztros 2000 maritime patrol aircraft frequently deployed to a U.S. military base in the Atlantic during the Cold War </search> |
| <information> Document (Title: Kindley Air Force Base) P-3B aircraft from various bases in the eastern United States, as well as NATO/Allied support consisting of Royal Air Force Hawker Siddeley Nimrod MR2s, Canadian Armed Forces CP-140 Auroras and other similar maritime patrol and reconnaissance aircraft from other NATO nations. During one period in 1985 that was characterized by exceptionally heavy Soviet Navy submarine activity off the … </information> |
| <think> This information does not directly name the aircraft. However, the mention of P-3C aircraft being frequently deployed to a U.S. military base in the Atlantic suggests that we should focus on this model. Given that P-3C is a well-known maritime patrol aircraft and the question specifies a connection to the first jet airliner, let’s consider if P-3C has any such connection. The P-3C is derived from the Lockheed P-3 Orion, which is itself a derivative of the Lockheed Model 85 Electra airliner. The first jet airliner was the de Havilland Comet, not directly related to the Electra. Therefore, we might need to re-evaluate our query to ensure we are considering the correct aircraft. Given the context and the need for accuracy, let’s consider the aircraft derived from the de Havilland Comet, which is the de Havilland Sea Vixen. However, the P-3C is more directly related to the Lockheed 1049C Hercules transport aircraft, which is not derived from the Comet but rather the Lockheed Electra. Given the specificity of the question, it seems likely that the aircraft in question is the P-3C Orion, which is frequently deployed and fits the timeframe. </think> |
| <answer> P-3C Orion </answer> |
| Question : Who directed the 1909 Italian silent film in which the actor who played the castle castellan in the 1928 German silent film The Mysterious Mirror portrayed the titular character of Shakespeare’s Macbeth? |
| Ground Truth : Mario Caserini |
| MR-Search : |
| <think> To determine who directed the 1909 Italian silent film in which the actor who played the castle castellan in the 1928 German silent film ’The Mysterious Mirror’ portrayed the titular character of Shakespeare’s ’Macbeth’, let’s first identify the actor and then find the Italian silent film he appeared in during 1909. </think> |
| <search> Query: Italian silent film from 1909 in which the actor who played the castle castellan in the 1928 German silent film The Mysterious Mirror portrayed the titular character of Shakespeare’s ’Macbeth’ directed by whom? </search> |
| <information> Document (Title: Macbeth (1916 film)) Macbeth (1916 film) Macbeth is a silent, black-and-white 1916 film adaptation of the William Shakespeare play Macbeth. It was directed by John Emerson, assisted by Erich von Stroheim, and produced by D. W. Griffith, with cinematography by Victor Fleming. The film starred Herbert Beerbohm Tree and Constance Collier, both famous from the stage and for playing Shakespearean parts. Although released during the first decade of feature filmmaking, it was already the seventh version of Macbeth to be produced, one of eight during the silent film era. It is considered to be a lost film. In the companion book to his … </information> |
| <think> It appears that the search did not directly provide the name of the actor. Let’s try the search again with a focus on the actor who played the castle castellan in The Mysterious Mirror and then find the film he portrayed Macbeth in 1909. </think> |
Table 6: Case study 3: MR-Search subsequently revisits the problem through additional search and self-reflection, ultimately producing the correct final answer.
| <information> Document (Title: Macbeth (1916 film)) Macbeth (1916 film) Macbeth is a silent, black-and-white 1916 film adaptation of the William Shakespeare play Macbeth. It was directed by John Emerson, assisted by Erich von Stroheim, and produced by D. W. Griffith, with cinematography by Victor Fleming. The film starred Herbert Beerbohm Tree and Constance Collier, both famous from the stage and for playing Shakespearean parts. Although released during the first decade of feature filmmaking, it was already the seventh version of Macbeth to be produced, one of eight during the silent film era. It is considered to be a lost film. In the companion book to his … </information> |
| <think> This information does not directly name the director. However, it mentions that the 1909 film Macbeth was directed by Maria Caserini’s husband. Given the previous context, let’s consider the most likely candidate based on the available information: </think> |
| <answer> Mario Caserini </answer> |
| Reflect on current answer to the question and provide an improved answer by searching for additional external information using search engines. |
| <think> Let’s re-evaluate the information we have. We know that the 1909 Italian silent film Macbeth was directed by Maria Caserini’s husband. Since the previous searches did not provide the name of her husband, let’s consider if we can infer this information from the context or search for it directly: </think> |
| <search> Query: Who was the husband of Maria Caserini who directed the 1909 Italian silent film Macbeth? </search> |
| <information> Document (Title: Maria Caserini) Maria Caserini Maria Caserini (née Gasperini; 24 July 1884 – 15 April 1969) was an Italian stage and film actress, as well as a pioneer of filmmaking during the early 20th-century. She often starred in adaptations of stage and film productions for the works of William Shakespeare. Born in Milan, she started acting in stage productions at a young age. Her first film was in 1906, in what is believed to be the earliest film adaptation of Shakespeare’s Othello, titled Otello, which was directed by her husband, Mario Caserini. She appeared in a film adaptation of Romeo and Juliet in … </information> |
| Question : In which country did Itamar Zorman make his first public appearance at a renowned festival in the early 2010s, and where is he scheduled to perform a concerto at a historic venue in a city with a rich Norman heritage and a famous opera house? |
| Ground Truth : Switzerland |
| MR-Search : |
| <think> To determine the country where Itamar Zorman made his first public appearance at a renowned festival in the early 2010s and where he is scheduled to perform a concerto at a historic venue in a city with a rich Norman heritage and a famous opera house, let’s first search for information about Itamar Zorman’s early public appearances and then find the details about his upcoming concert. </think> |
| <search> Query: Itamar Zorman first public appearance at a renowned festival in the early 2010s and upcoming concerto at a historic venue in a city with rich Norman heritage and famous opera house </search> |
| <information> Document (Title: Itamar Zorman) Hamburg and Kronberg Academy Festival in Frankfurt as well as with Yuri Bashmet. His first CD is planned to be released sometime in 2014 with his recitals the Louvre. In November of the same year he, Daniel Oren and Musicians from Marlboro group, will perform Ludwig van Beethoven’s concerto at the Teatro Massimo in Palermo. For the past five years he has traveled through Israel and North America during which he was a founder of the Israeli Chamber Project as well as a Lysander Piano Trio member. On March 9, 2014 he performed one of Johannes Brahms’ sonatas in Detroit … </information> |
Table 7: Case study 4: MR-Search subsequently revisits the problem through additional search and self-reflection, ultimately producing the correct final answer.
| <think> Itamar Zorman made his first public appearance at the Verbier Festival in Switzerland in 2010. Now, let’s find out where he is scheduled to perform a concerto at a historic venue in a city with a rich Norman heritage and a famous opera house. </think> |
| <search> Query: Itamar Zorman concerto at a historic venue in a city with rich Norman heritage and famous opera house </search> |
| <information> Document (Title: Semperoper) Semperoper The Semperoper is the opera house of the Sächsische Staatsoper Dresden (Saxon State Opera) and the concert hall of the Staatskapelle Dresden (Saxon State Orchestra). It is also home to the Semperoper Ballett. The building is located near the Elbe River in the historic centre of Dresden, Germany. The opera house was originally built by the architect Gottfried Semper in 1841. After a devastating fire in 1869, the opera house was rebuilt, partly again by Semper, and completed in 1878. The opera house has a long history of premieres, including major works by Richard Wagner and Richard Strauss. The … </information> |
| <think> This information does not directly state the location of the upcoming concert. However, it mentions the Teatro Massimo in Palermo, which is in Italy. Let’s verify if this matches the criteria of a city with a rich Norman heritage and a famous opera house. </think> |
| <search> Query: Teatro Massimo in Palermo, Italy </search> |
| <information> Document (Title: Teatro Massimo) Teatro Massimo The Teatro Massimo Vittorio Emanuele is an opera house and opera company located on the Piazza Verdi in Palermo, Sicily. It was dedicated to King Victor Emanuel II. It is the biggest in Italy, and one of the largest of Europe (the third after the Opéra National de Paris and the K. K. Hof-Opernhaus in Vienna), renowned for its perfect acoustics. An international competition for the creation of the opera house was announced by the Palermo Council in 1864 at the instigation of the mayor, Antonio Starrabba di Rudinì. For many years there had been talk of building … </information> |
| <think> The Teatro Massimo in Palermo is indeed a famous opera house. Now, let’s confirm if Palermo has a rich Norman heritage: </think> |