ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget
Organizations: David R. Cheriton School of Computer Science University of Waterloo
Abstract
Search agents, which integrate language models (LMs) with web search, are becoming crucial for answering complex user queries. Constructing training datasets for deep research tasks, involving multi-step retrieval and reasoning, remains challenging due to expensive human annotation, or cumbersome prerequisites. In this work, we introduce ORBIT, a training dataset with 20K reasoning-intensive queries with short verifiable answers, generated using a frugal framework without relying on paid API services. The modular framework relies on four stages: seed creation, question-answer pair generation, and two stages of verification: self and external. ORBIT spans 15 domains and each training pair requires 4-5 reasoning steps, with external search verification required from the complete web. We train Qwen3-4B as the base model on ORBIT using GRPO and evaluate it on Wikipedia question answering tasks. Extensive experiment results demonstrate that ORBIT-4B achieves strong performance among sub-4B LLMs as search agents, proving the utility of synthetic datasets. Our framework, code and datasets are open-sourced and available publicly.
Figures & tables
| Dataset Name | Source | Train Pairs | Complexity | Pre-requisite / Constraint |
| NQ ( Kwiatkowski et al., 2019 ) | Wiki | 300K+ | Easy | None |
| HotpotQA ( Yang et al., 2018 ) | Wiki | 100K+ | Easy | None |
| WebWalkerQA ( Wu et al., 2025 ) | Web | 14.3K | Easy | Web Traversal |
| SimpleDeepS. ( Sun et al., 2025b ) | Wiki & Web | 871 | Easy | Web Search & HTML Extract |
| InfoDeepSeek ( Xi et al., 2025 ) | Web | 245 | Easy | Human Annotators |
| WebShaper ( Tao et al., 2026 ) | Wiki | 500 | Hard | Wikipedia Hyperlinks |
| Question: What was the exact runtime (minutes) of the 2017 animated feature set inside a smartphone’s messaging application , directed by a filmmaker previously known for sequels to popular children’s franchises , featuring a protagonist whose facial expression malfunctions , with voice casting that includes a lead actor from a critically acclaimed sitcom , and produced by a studio that later won an Oscar for Spider-Man animation ? |
| Answer: 86 minutes |
| Verification / Summary of Supporting Facts: |
| ✓ Animated Set : The Emoji Movie released in 2017, set inside a smartphone messaging app world called Textopolis. |
| ✓ Filmmaker: Tony Leondis , sequels to franchises: Lilo & Stitch 2, Kung Fu Panda: Secrets of the Masters . |
| ✓ Protagonist : Gene , the main character in The Emoji Movie, struggles with malfunctioning expressions. |
| ✓ Voice Cast : T.J. Miller , who was a lead actor from the HBO sitcom titled Silicon Valley. |
| Dataset Statistic | Value |
| # Question–Answer pairs | 20,147 |
| - Avg. # question tokens | 63.87 |
| - Avg. # answer tokens | 3.46 |
| - Avg. # reasoning steps | 4.42 |
| Avg. Verification URLs | 4.36 |
| - Non-Wikipedia URLs | 3.35 |
| Model | Single-Hop QA | Multi-Hop QA | Avg. 7 | |||||
| NQ | TQA | PopQA | HQA | 2Wiki | MSQ | Bamb | ||
| Search Agents (Base: Qwen2.5-3B-Instruct) | ||||||||
| Search-o1-3B | 23.8 | 48.2 | 26.2 | 22.1 | 21.8 | 5.4 | 32.0 | 25.6 |
| Search-R1-3B | 40.8 | 59.1 | 42.8 | 30.8 | 31.1 | 8.4 | 13.0 | 32.3 |
| ZeroSearch-3B | 41.2 | 61.5 | 44.0 | 31.2 | 33.2 | 12.6 | 14.3 | 34.0 |
| AutoRefine-3B | 43.6 | 59.7 | 44.7 | 40.4 | 38.0 | 16.9 | 33.6 | 39.6 |
| Model | FRAMES |
| Search-R1-4B † | 14.7 |
| InfoSeeker-4B † | 18.5 |
| ORBIT-4B † | 24.2 |
| ORBIT-4B | Single-Hop QA | Multi-Hop QA | Avg. 8 | ||||||
| NQ | TQA | PopQA | HQA | 2Wiki | MSQ | Bamb | FRAMES | ||
| Dataset Mixing Ratios (Dataset ratio of NQ: HotpotQA: ORBIT) | |||||||||
| Ratio (1: 1: 1) | 43.7 | 67.3 | 53.8 | 42.5 | 61.1 | 20.1 | 55.2 | 24.2 | 46.0 |
| Ratio (1: 2: 4) | 41.7 | 65.3 | 50.8 | 40.0 | 61.3 | 16.1 | 52.0 | 19.1 | 43.3 |
| Top #K Search Results (Snippet & title only during training) | |||||||||
| Top 5 | 43.7 | 67.3 | 53.8 | 42.5 | 61.1 | 20.1 | 55.2 | 24.2 | 46.0 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Module | Hyper-parameter | Value |
| Data | Max observation (search result) length | 1,024 |
| Max response length | 8,192 | |
| Max prompt length | 2,048 | |
| Max action length | 2,048 | |
| Retriever top- | 5 (using DDGS) | |
| Actor | Training batch size | 256 |
| Methods | Single-Hop QA | Multi-Hop QA | Avg. 7 | |||||
| NQ | TQA | PopQA | HQA | 2Wiki | MSQ | Bamb | ||
| Search Agents (Base: Qwen2.5-3B) | ||||||||
| Search-o1 ( Li et al., 2025b ) | 23.8 | 47.2 | 26.2 | 22.1 | 21.8 | 5.4 | 32.0 | 25.5 |
| Search-R1-Instruct ( Jin et al., 2025 ) | 39.7 | 56.5 | 39.1 | 33.1 | 31.0 | 12.4 | 23.2 | 33.6 |
| Search-R1-Base ( Jin et al., 2025 ) | 42.1 | 58.3 | 41.3 | 29.7 | 27.4 | 6.6 | 12.8 | 31.2 |
| ReSearch-Instruct ( Chen et al., 2025 ) | 36.5 | 57.1 | 39.5 | 35.1 | 27.2 | 9.5 | 26.6 | 33.1 |
| Answer Type | Definition |
| Temporal | Calendar date, year, month, decade, or explicitly time-anchored period. |
| Numeric / Quantitative | Numerical value, measurement, ratio, duration, or rate, with or without units. |
| Geographic Location | Physical place, region, landmark, or spatially identifiable location on Earth. |
| Person | Named individual (historical, contemporary, or fictional). |
| Organization / Institution | Named group, body, office, society, standardization entity, or collective actor. |
| Event / Historical Phenomenon | Named occurrence, movement, crisis, or episode that happens in time. |
| Case Study on FRAMES ( Krishna et al., 2025 ) |
| Question: What compass direction (of the 4 cardinal and 4 ordinal directions) is the capital city of the state which houses the Jackson Hole Mountain Resort in relation to the centre of the state? |
| Ground Truth: [“Southeast”] |
| Model Output: ORBIT-4B |
| <think> The question requires three reasoning hops: (1) identify the state that contains Jackson Hole Mountain Resort, (2) find the capital of that state, and (3) determine the capital’s compass direction relative to the state’s geographic centre. I recall that Jackson Hole lies in Wyoming; the capital is likely Cheyenne, but I should confirm. </think> |
| <search> capital of Wyoming </search> |
| <information> Doc 1 (Title: “Wyoming State Capitol”): The Wyoming State Capitol is located in the city of Cheyenne . Cheyenne is the seat of government for the U.S. state of Wyoming … |
| Case Study on ORBIT during GRPO Training |
| Question: In which 1957 nature essay collection does a chapter describing a boy’s unexpected discovery of a sunfish in a city creek—illustrating how preconceptions blind observers to novel phenomena—appear, written by an anthropologist-naturalist who later became president of the American Institute of Human Paleontology? |
| Ground Truth: [“The Immense Journey”] |
| Model Output: ORBIT-4B |
| <think> I need a 1957 nature essay collection with a sunfish chapter, written by an anthropologist-naturalist who later led the American Institute of Human Paleontology. This is a multi-hop question: (1) identify the author via the institute clue, (2) find the 1957 collection they wrote, and (3) confirm the sunfish chapter. Let me first search by the essay’s contents. </think> |
| <search> 1957 nature essay collection sunfish city creek </search> |
| <information> Doc 1 (Title: “1957 — Wikipedia”): 1957 (MCMLVII) was a common year starting on Tuesday of the Gregorian calendar… |
| Domain: (2) Science & Technology |
| Question: What term describes the mathematical concept first systematically developed in a 1939 physics textbook by a Nobel laureate who independently predicted antimatter , where angled brackets denote dual vector spaces and vertical bars separate state vectors , revolutionizing the formalism of quantum measurement theory ? |
| Answer: bra-ket notation |
| Verification / Summary of Supporting Facts: |
| ✓ Origin: Paul Dirac introduced bra-ket notation in his 1939 publication A New Notation for Quantum Mechanics . |
| ✓ Nobel Laureate: Dirac independently predicted antimatter (positrons) through his equation, confirmed experimentally in 1932. |
| ✓ Notation: Uses angled brackets (bras) and vertical bars (kets) to denote dual vector spaces and separate state vectors . |
| List of up to 100 Wikipedia Categories (generated using OpenAI’s Deep Research) |
|---|
| (1) TV Shows & Movies |
| American crime drama television series; British science fiction television series; Japanese drama television series; Anime television series; Sitcoms; Reality television series; Game shows; Miniseries; Documentary television series; Animated films; Action films; Comedy films; Horror films; Science fiction films; Documentary films; Silent films; Black-and-white films; Direct-to-video films; Short films; American independent films; Spaghetti Western films; Film noir; Kaiju; Blaxploitation films; Animated feature films; 1970s American television series; Hindi-language films; French New Wave; South Korean films; Musical films; Films about outer space; Films about immigration to the United States; Films about poverty; Films about educators; Films about viral outbreaks; Films about artificial intelligence; Films about time travel; Films based on actual events; Lost films; Unfinished films; Cancelled films; Directorial debut films; Film series; Film remakes; Hong Kong action films; Direct-to-video horror films; British comedy films; Indian horror films; French animated films; Anime films; Soap operas; Telenovelas; Netflix original programming; HBO original programming; Anthology television series; Serial killer films; Superhero films; Monster movies; Spy films; Biographical films; Fantasy films; War films; Disaster films; Adventure films; Thriller films; Psychological thriller films; Slasher films; Dystopian films; Post-apocalyptic films; Biblical films; Christmas films; Best Picture Academy Award winners; Film festivals; Television spin-offs; Unaired television pilots; Propaganda films; Crossover films; Video nasties; 3D films; IMAX films; Peplum films; Neo-noir films; Samurai films; Martial arts films; African films; Mexican films; Iranian films; Films based on video games; Television series based on films; Propaganda films; Crossover films; Film posters by genre; Remake television series; Unaired television pilots; Video nasties; 3D films; IMAX films. |
| (2) Science & Technology |
| Physics; Chemistry; Biology; Mathematics; Astronomy; Geology; Computer science; Engineering; Materials science; Nanotechnology; Robotics; Artificial intelligence; Machine learning; Quantum mechanics; Quantum field theory; General relativity; Thermodynamics; Particle physics; Nuclear physics; Optics; Astrophysics; Spaceflight; Satellites; Rocketry; Renewable energy; Climate change; Meteorology; Oceanography; Volcanology; Ecology; Evolutionary biology; Genetics; Bioinformatics; Neuroscience; Molecular biology; Virology; Biomedical engineering; Biotechnology; Genetic engineering; Cryonics; Telescopes; Microscopy; Chemical elements; Organic compounds; Polymers; Crystallography; Electronics; Semiconductors; Internet; Computer security; Operating systems; Programming languages; Open-source software; Computer networks; Supercomputers; Distributed computing; Cloud computing; Databases; Algorithms; Graph theory; Number theory; Combinatorics; Topology; Algebra; Geometry; Differential equations; Computational biology; Astrobiology; Exoplanets; Dark matter; String theory; Complex systems; Chaos theory; Computer vision; Natural language processing; Human–computer interaction; Embedded systems; Quantum computing; Virtual reality; Augmented reality; Blockchains; Mobile phones; Telecommunications; Software engineering; Programming paradigms; Data structures; Artificial life; Cryptography; Information theory; Game theory; Automation; Aerospace engineering; Mechanical engineering; Electrical engineering; Civil engineering; Chemical engineering; Computational chemistry; Systems biology; Computational complexity theory. |
| (3) Art |
| Painting; Sculpture; Photography; Architecture; Drawing; Printmaking; Ceramics; Textile arts; Glass art; Graphic design; Industrial design; Fashion design; Calligraphy; Comics; Theatre; Opera; Dance; Ballet; Circus; Magic (illusion); Performance art; Street art; Public art; Digital art; Contemporary art; Modern art; Renaissance art; Medieval art; Baroque art; Impressionism; Post-Impressionism; Surrealism; Cubism; Futurism; Dada; Abstract art; Abstract expressionism; Pop art; Art Nouveau; Gothic architecture; Baroque architecture; Modernist architecture; Skyscrapers; Video art; Installation art; Conceptual art; Murals; Portrait painting; Landscape art; Still life paintings; Mosaics; Poetry; Novels; Short stories; Plays; Science fiction novels; Fantasy novels; Detective novels; Children’s literature; African art; Islamic art; Pre-Columbian art; Chinese art; Indian art; Japanese art; Indigenous Australian art; Mexican art; Native American art; Art history; Byzantine art; Christian art; Stagecraft; Choreography; Typography; Heraldry; Vexillology; Cartography; History painting; Nude art; Religious art; Decorative arts; Folk art; Outsider art; Manga; Musicals; Costume design; Black-and-white photography; Portrait photography; Renaissance architecture; Neoclassical architecture; Landscape architecture; French art; American art; Latin American art; Mime; Puppetry; Vaudeville; Improvisational theatre. |
Synthetic Data Generation for Language Models
Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.
See all 293 papers