cs.CLApr 1, 2026
SaveORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget
Organizations: David R. Cheriton School of Computer Science University of Waterloo
Abstract
Search agents, which integrate language models (LMs) with web search, are becoming crucial for answering complex user queries. Constructing training datasets for deep research tasks, involving multi-step retrieval and reasoning, remains challenging due to expensive human annotation, or cumbersome prerequisites. In this work, we introduce ORBIT, a training dataset with 20K reasoning-intensive queries with short verifiable answers, generated using a frugal framework without relying on paid API services. The modular framework relies on four stages: seed creation, question-answer pair generation, and two stages of verification: self and external. ORBIT spans 15 domains and each training pair requires 4-5 reasoning steps, with external search verification required from the complete web. We train Qwen3-4B as the base model on ORBIT using GRPO and evaluate it on Wikipedia question answering tasks. Extensive experiment results demonstrate that ORBIT-4B achieves strong performance among sub-4B LLMs as search agents, proving the utility of synthetic datasets. Our framework, code and datasets are open-sourced and available publicly.
Figures & tables
| Dataset Name | Source | Train Pairs | Complexity | Pre-requisite / Constraint |
| NQ ( Kwiatkowski et al., 2019 ) | Wiki | 300K+ | Easy | None |
| HotpotQA ( Yang et al., 2018 ) | Wiki | 100K+ | Easy | None |
| WebWalkerQA ( Wu et al., 2025 ) | Web | 14.3K | Easy | Web Traversal |
| SimpleDeepS. ( Sun et al., 2025b ) | Wiki & Web | 871 | Easy | Web Search & HTML Extract |
| InfoDeepSeek ( Xi et al., 2025 ) | Web | 245 | Easy | Human Annotators |
| WebShaper ( Tao et al., 2026 ) | Wiki | 500 | Hard | Wikipedia Hyperlinks |
Table 1: Comparison of ORBIT against other training datasets for search agents. Prior datasets require prior constraints such as a knowledge graph (KG), lack structural depth or remain limited in scale. ORBIT is a large-scale multi-hop dataset, requiring no prior constraints and containing complex queries, that can be scaled easily with a low budget.
| Question: What was the exact runtime (minutes) of the 2017 animated feature set inside a smartphone’s messaging application , directed by a filmmaker previously known for sequels to popular children’s franchises , featuring a protagonist whose facial expression malfunctions , with voice casting that includes a lead actor from a critically acclaimed sitcom , and produced by a studio that later won an Oscar for Spider-Man animation ? |
| Answer: 86 minutes |
| Verification / Summary of Supporting Facts: |
| ✓ Animated Set : The Emoji Movie released in 2017, set inside a smartphone messaging app world called Textopolis. |
| ✓ Filmmaker: Tony Leondis , sequels to franchises: Lilo & Stitch 2, Kung Fu Panda: Secrets of the Masters . |
| ✓ Protagonist : Gene , the main character in The Emoji Movie, struggles with malfunctioning expressions. |
| ✓ Voice Cast : T.J. Miller , who was a lead actor from the HBO sitcom titled Silicon Valley. |
Table 2: A training data example showing the question, answer and verification (for the reader) from the TV Shows & Movies domain in the ORBIT dataset. Colored spans mark the distinct factual clues embedded in the question; the verification rows confirm each clue with key supporting evidence ( blue ). More dataset examples are provided in Appendix E .
Figure 1: An end-to-end data instance in the ORBIT dataset generated using our framework. The procedure involves the following stages: (1) seed creation , (2) question–answer pair generation , (3) self-verification with DeepSeek Chat , and (4) external verification with scraped web page URLs . Each stage is described in detail in Section 3.1 .
| Dataset Statistic | Value |
| # Question–Answer pairs | 20,147 |
| - Avg. # question tokens | 63.87 |
| - Avg. # answer tokens | 3.46 |
| - Avg. # reasoning steps | 4.42 |
| Avg. Verification URLs | 4.36 |
| - Non-Wikipedia URLs | 3.35 |
Table 3: ORBIT dataset statistics . URL statistics reflect the distinct sources required for answer verification.
| Model | Single-Hop QA | Multi-Hop QA | Avg. 7 | |||||
| NQ | TQA | PopQA | HQA | 2Wiki | MSQ | Bamb | ||
| Search Agents (Base: Qwen2.5-3B-Instruct) | ||||||||
| Search-o1-3B | 23.8 | 48.2 | 26.2 | 22.1 | 21.8 | 5.4 | 32.0 | 25.6 |
| Search-R1-3B | 40.8 | 59.1 | 42.8 | 30.8 | 31.1 | 8.4 | 13.0 | 32.3 |
| ZeroSearch-3B | 41.2 | 61.5 | 44.0 | 31.2 | 33.2 | 12.6 | 14.3 | 34.0 |
| AutoRefine-3B | 43.6 | 59.7 | 44.7 | 40.4 | 38.0 | 16.9 | 33.6 | 39.6 |
Table 4: Performance comparison on single-hop and multi-hop QA benchmarks of search agents 4B parameters. Best scores are highlighted in bold . ( ) denotes models that were trained in our work using Qwen3-4B as the base model. In addition, EM accuracy results using the 2018 Wikipedia corpus and E5-base-v2 retriever are provided in Table 8 .
Figure 4: Validation EM accuracy of Search-R1-4B, InfoSeeker-4B and ORBIT-4B on 160 training steps on Wikipedia datasets, each with 125 randomly sampled validation pairs. The accuracy drops observed during training primarily occur due to DDGS web search retriever, that can potentially downgrade search results when servers are busy (e.g., google bing ).
| Model | FRAMES |
| Search-R1-4B † | 14.7 |
| InfoSeeker-4B † | 18.5 |
| ORBIT-4B † | 24.2 |
Table 5: EM accuracy on FRAMES.
| ORBIT-4B | Single-Hop QA | Multi-Hop QA | Avg. 8 | ||||||
| NQ | TQA | PopQA | HQA | 2Wiki | MSQ | Bamb | FRAMES | ||
| Dataset Mixing Ratios (Dataset ratio of NQ: HotpotQA: ORBIT) | |||||||||
| Ratio (1: 1: 1) | 43.7 | 67.3 | 53.8 | 42.5 | 61.1 | 20.1 | 55.2 | 24.2 | 46.0 |
| Ratio (1: 2: 4) | 41.7 | 65.3 | 50.8 | 40.0 | 61.3 | 16.1 | 52.0 | 19.1 | 43.3 |
| Top #K Search Results (Snippet & title only during training) | |||||||||
| Top 5 | 43.7 | 67.3 | 53.8 | 42.5 | 61.1 | 20.1 | 55.2 | 24.2 | 46.0 |
Table 6: Ablations on dataset mixing ratios and top- search results with ORBIT-4B.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Module | Hyper-parameter | Value |
| Data | Max observation (search result) length | 1,024 |
| Max response length | 8,192 | |
| Max prompt length | 2,048 | |
| Max action length | 2,048 | |
| Retriever top- | 5 (using DDGS) | |
| Actor | Training batch size | 256 |
Table 7: Hyperparameters used to train ORBIT-4B and other search agents (Search-R1-4B and InfoSeeker-4B) with GRPO via the verl-tool framework ( Jiang et al., 2025 ) .
| Methods | Single-Hop QA | Multi-Hop QA | Avg. 7 | |||||
| NQ | TQA | PopQA | HQA | 2Wiki | MSQ | Bamb | ||
| Search Agents (Base: Qwen2.5-3B) | ||||||||
| Search-o1 ( Li et al., 2025b ) | 23.8 | 47.2 | 26.2 | 22.1 | 21.8 | 5.4 | 32.0 | 25.5 |
| Search-R1-Instruct ( Jin et al., 2025 ) | 39.7 | 56.5 | 39.1 | 33.1 | 31.0 | 12.4 | 23.2 | 33.6 |
| Search-R1-Base ( Jin et al., 2025 ) | 42.1 | 58.3 | 41.3 | 29.7 | 27.4 | 6.6 | 12.8 | 31.2 |
| ReSearch-Instruct ( Chen et al., 2025 ) | 36.5 | 57.1 | 39.5 | 35.1 | 27.2 | 9.5 | 26.6 | 33.1 |
Table 8: Accuracy comparison of search agents 4B parameters on single-hop and multi-hop QA benchmarks using the Search-R1 evaluation setup ( Jin et al., 2025 ) , with the 2018 Wikipedia dump ( Karpukhin et al., 2020 ) , E5-base-v2 retriever ( Wang et al., 2024 ) serving as the retrieval engine retrieving top-3 documents for every search query. Qwen2.5-3B results are taken from SAPO ( Li et al., 2026a ) . ( ) denotes search agents trained in our work using Qwen3-4B as the base model. Best scores are highlighted in bold .
| Answer Type | Definition |
| Temporal | Calendar date, year, month, decade, or explicitly time-anchored period. |
| Numeric / Quantitative | Numerical value, measurement, ratio, duration, or rate, with or without units. |
| Geographic Location | Physical place, region, landmark, or spatially identifiable location on Earth. |
| Person | Named individual (historical, contemporary, or fictional). |
| Organization / Institution | Named group, body, office, society, standardization entity, or collective actor. |
| Event / Historical Phenomenon | Named occurrence, movement, crisis, or episode that happens in time. |
Table 9: Definitions of answer types used for categorizing answers in the ORBIT dataset.
| Case Study on FRAMES ( Krishna et al., 2025 ) |
| Question: What compass direction (of the 4 cardinal and 4 ordinal directions) is the capital city of the state which houses the Jackson Hole Mountain Resort in relation to the centre of the state? |
| Ground Truth: [“Southeast”] |
| Model Output: ORBIT-4B |
| <think> The question requires three reasoning hops: (1) identify the state that contains Jackson Hole Mountain Resort, (2) find the capital of that state, and (3) determine the capital’s compass direction relative to the state’s geographic centre. I recall that Jackson Hole lies in Wyoming; the capital is likely Cheyenne, but I should confirm. </think> |
| <search> capital of Wyoming </search> |
| <information> Doc 1 (Title: “Wyoming State Capitol”): The Wyoming State Capitol is located in the city of Cheyenne . Cheyenne is the seat of government for the U.S. state of Wyoming … |
Table 10: Case study of ORBIT-4B evaluated on FRAMES ( Krishna et al., 2025 ) . The search agent must chain geographic facts across multiple hops; key evidence extracted from retrieved documents is bolded .
| Case Study on ORBIT during GRPO Training |
| Question: In which 1957 nature essay collection does a chapter describing a boy’s unexpected discovery of a sunfish in a city creek—illustrating how preconceptions blind observers to novel phenomena—appear, written by an anthropologist-naturalist who later became president of the American Institute of Human Paleontology? |
| Ground Truth: [“The Immense Journey”] |
| Model Output: ORBIT-4B |
| <think> I need a 1957 nature essay collection with a sunfish chapter, written by an anthropologist-naturalist who later led the American Institute of Human Paleontology. This is a multi-hop question: (1) identify the author via the institute clue, (2) find the 1957 collection they wrote, and (3) confirm the sunfish chapter. Let me first search by the essay’s contents. </think> |
| <search> 1957 nature essay collection sunfish city creek </search> |
| <information> Doc 1 (Title: “1957 — Wikipedia”): 1957 (MCMLVII) was a common year starting on Tuesday of the Gregorian calendar… |
Table 11: Case study of ORBIT-4B during GRPO training on ORBIT. Key evidence extracted from retrieved documents is bolded .
| Domain: (2) Science & Technology |
| Question: What term describes the mathematical concept first systematically developed in a 1939 physics textbook by a Nobel laureate who independently predicted antimatter , where angled brackets denote dual vector spaces and vertical bars separate state vectors , revolutionizing the formalism of quantum measurement theory ? |
| Answer: bra-ket notation |
| Verification / Summary of Supporting Facts: |
| ✓ Origin: Paul Dirac introduced bra-ket notation in his 1939 publication A New Notation for Quantum Mechanics . |
| ✓ Nobel Laureate: Dirac independently predicted antimatter (positrons) through his equation, confirmed experimentally in 1932. |
| ✓ Notation: Uses angled brackets (bras) and vertical bars (kets) to denote dual vector spaces and separate state vectors . |
Table 12: One training example from each of the 14 additional knowledge domains in ORBIT. Colored spans mark the distinct factual clues embedded in the question; the verification rows confirm each clue with key supporting evidence ( blue ).
| List of up to 100 Wikipedia Categories (generated using OpenAI’s Deep Research) |
|---|
| (1) TV Shows & Movies |
| American crime drama television series; British science fiction television series; Japanese drama television series; Anime television series; Sitcoms; Reality television series; Game shows; Miniseries; Documentary television series; Animated films; Action films; Comedy films; Horror films; Science fiction films; Documentary films; Silent films; Black-and-white films; Direct-to-video films; Short films; American independent films; Spaghetti Western films; Film noir; Kaiju; Blaxploitation films; Animated feature films; 1970s American television series; Hindi-language films; French New Wave; South Korean films; Musical films; Films about outer space; Films about immigration to the United States; Films about poverty; Films about educators; Films about viral outbreaks; Films about artificial intelligence; Films about time travel; Films based on actual events; Lost films; Unfinished films; Cancelled films; Directorial debut films; Film series; Film remakes; Hong Kong action films; Direct-to-video horror films; British comedy films; Indian horror films; French animated films; Anime films; Soap operas; Telenovelas; Netflix original programming; HBO original programming; Anthology television series; Serial killer films; Superhero films; Monster movies; Spy films; Biographical films; Fantasy films; War films; Disaster films; Adventure films; Thriller films; Psychological thriller films; Slasher films; Dystopian films; Post-apocalyptic films; Biblical films; Christmas films; Best Picture Academy Award winners; Film festivals; Television spin-offs; Unaired television pilots; Propaganda films; Crossover films; Video nasties; 3D films; IMAX films; Peplum films; Neo-noir films; Samurai films; Martial arts films; African films; Mexican films; Iranian films; Films based on video games; Television series based on films; Propaganda films; Crossover films; Film posters by genre; Remake television series; Unaired television pilots; Video nasties; 3D films; IMAX films. |
| (2) Science & Technology |
| Physics; Chemistry; Biology; Mathematics; Astronomy; Geology; Computer science; Engineering; Materials science; Nanotechnology; Robotics; Artificial intelligence; Machine learning; Quantum mechanics; Quantum field theory; General relativity; Thermodynamics; Particle physics; Nuclear physics; Optics; Astrophysics; Spaceflight; Satellites; Rocketry; Renewable energy; Climate change; Meteorology; Oceanography; Volcanology; Ecology; Evolutionary biology; Genetics; Bioinformatics; Neuroscience; Molecular biology; Virology; Biomedical engineering; Biotechnology; Genetic engineering; Cryonics; Telescopes; Microscopy; Chemical elements; Organic compounds; Polymers; Crystallography; Electronics; Semiconductors; Internet; Computer security; Operating systems; Programming languages; Open-source software; Computer networks; Supercomputers; Distributed computing; Cloud computing; Databases; Algorithms; Graph theory; Number theory; Combinatorics; Topology; Algebra; Geometry; Differential equations; Computational biology; Astrobiology; Exoplanets; Dark matter; String theory; Complex systems; Chaos theory; Computer vision; Natural language processing; Human–computer interaction; Embedded systems; Quantum computing; Virtual reality; Augmented reality; Blockchains; Mobile phones; Telecommunications; Software engineering; Programming paradigms; Data structures; Artificial life; Cryptography; Information theory; Game theory; Automation; Aerospace engineering; Mechanical engineering; Electrical engineering; Civil engineering; Chemical engineering; Computational chemistry; Systems biology; Computational complexity theory. |
| (3) Art |
| Painting; Sculpture; Photography; Architecture; Drawing; Printmaking; Ceramics; Textile arts; Glass art; Graphic design; Industrial design; Fashion design; Calligraphy; Comics; Theatre; Opera; Dance; Ballet; Circus; Magic (illusion); Performance art; Street art; Public art; Digital art; Contemporary art; Modern art; Renaissance art; Medieval art; Baroque art; Impressionism; Post-Impressionism; Surrealism; Cubism; Futurism; Dada; Abstract art; Abstract expressionism; Pop art; Art Nouveau; Gothic architecture; Baroque architecture; Modernist architecture; Skyscrapers; Video art; Installation art; Conceptual art; Murals; Portrait painting; Landscape art; Still life paintings; Mosaics; Poetry; Novels; Short stories; Plays; Science fiction novels; Fantasy novels; Detective novels; Children’s literature; African art; Islamic art; Pre-Columbian art; Chinese art; Indian art; Japanese art; Indigenous Australian art; Mexican art; Native American art; Art history; Byzantine art; Christian art; Stagecraft; Choreography; Typography; Heraldry; Vexillology; Cartography; History painting; Nude art; Religious art; Decorative arts; Folk art; Outsider art; Manga; Musicals; Costume design; Black-and-white photography; Portrait photography; Renaissance architecture; Neoclassical architecture; Landscape architecture; French art; American art; Latin American art; Mime; Puppetry; Vaudeville; Improvisational theatre. |
Table 13: Top 100 categories per each domain in the ORBIT dataset generated using OpenAI’s Deep Research tool.
More in this topic
Synthetic Data Generation for Language Models
Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.
See all 293 papers