cs.AIMar 5, 2026
SaveKARL: Knowledge Agents via Reinforcement Learning
Organizations: Databricks AI Research
Abstract
We present a system for training enterprise search agents via reinforcement learning that achieves state-of-the-art performance across a diverse suite of hard-to-verify agentic search tasks. Our work makes four core contributions. First, we introduce KARLBench, a multi-capability evaluation suite spanning six distinct search regimes, including constraint-driven entity search, cross-document report synthesis, tabular numerical reasoning, exhaustive entity retrieval, procedural reasoning over technical documentation, and fact aggregation over internal enterprise notes. Second, we show that models trained across heterogeneous search behaviors generalize substantially better than those optimized for any single benchmark. Third, we develop an agentic synthesis pipeline that employs long-horizon reasoning and tool use to generate diverse, grounded, and high-quality training data, with iterative bootstrapping from increasingly capable models. Fourth, we propose a new post-training paradigm based on iterative large-batch off-policy RL that is sample efficient, robust to train-inference engine discrepancies, and naturally extends to multi-task training with out-of-distribution generalization. Compared to Claude 4.6 and GPT 5.2, KARL is Pareto-optimal on KARLBench across cost-quality and latency-quality trade-offs, including tasks that were out-of-distribution during training. With sufficient test-time compute, it surpasses the strongest closed models. These results show that tailored synthetic data in combination with multi-task reinforcement learning enables cost-efficient and high-performing knowledge agents for grounded reasoning.
Figures & tables
Figure 1: Performance of KARL, with and without test-time compute, compared to state-of-the-art agentic models on KARLBench. The cost–quality and latency–quality Pareto frontiers show that KARL achieves favorable trade-offs over existing models. All while being more cost and latency effective, KARL exceeds the quality of Sonnet 4.6 with three parallel rollouts and matches the best model, Opus 4.6, with ten parallel rollouts. See experiment details in Appendix B .
| Name | Capability | Example Question | Example Answer |
| BrowseComp-Plus (Chen, 2025 ) | Constraint-driven entity search. | Which Nobel physicist was born in the same city as the author of The Trial and later worked at the Institute for Advanced Study? | Albert Einstein |
| TREC-Biogen (Gupta, 2024 ) | Cross-document report synthesis. | What evidence supports the effectiveness of mRNA vaccines against emerging SARS-CoV-2 variants? | A report integrating findings from clinical studies, observational analyses, and variant-specific evaluations. |
| FinanceBench (Islam, 2023 ) | Long-document traversal with tabular numerical reasoning. | Based on Company X’s 2022 annual report, what was the percent change in operating income from 2021 to 2022? | Operating income increased by 12.4%, computed from 2.36B (2022). |
| QAMPARI (Amouyal, 2023 ) | Exhaustive entity search over encyclopedic text. | Which countries have won at least one FIFA World Cup? | Brazil; Germany; Italy; Argentina; France; Uruguay; England; Spain. |
| FreshStack (Thakur, 2025a ) | Procedural reasoning over technical documentation. | How can a ModuleNotFoundError be resolved when running a Python script inside a virtual environment? | Activate the correct environment, verify installation with pip list , install the missing package using pip install <package> , and ensure the interpreter path matches the environment. |
| PMBench | Exhaustive fact search over internal company notes. | What are the specific concerns raised regarding governance in production environments, and which customers raised them? | XYZ Corp and ABC Financial raised governance concerns around access controls for model updates, audit logging, and environment separation. |
Table 1: Illustrative examples of task capabilities. Each dataset isolates a distinct structural challenge.
| Dataset | #Q | Avg Q Tok | Relevant Chunks/Q | Answer Nuggets/Q | #D | Avg D Tok |
| BrowseComp-Plus | 830 | 123.2 | 100,195 | 480.9 | ||
| TREC-Biogen | 65 | 15.6 | 26,805,982 | 309.4 | ||
| FinanceBench | 150 | 35.3 | 53,399 | 717.9 | ||
| QAMPARI | 1,000 | 12.3 | 256,680 | 129.8 | ||
| FreshStack | 203 | 475.0 | 49,514 | 1098.5 | ||
| PMBench | 57 | 40.4 | 3,395 | 1518.4 |
Table 2: Dataset statistics: Number of questions (Q), indexed document chunks (D), and their average token counts. We also report the mean and standard deviation of ground truth relevant chunks and answer nuggets per question.
Figure 2: Stage I: The synthesis pipeline takes as input few-shot examples and the corpus for a task. Then, the Question-Answer Generator Agent explores the corpus via a vector search tool before proposing a possible synthetic question-answer pair that is grounded in the retrieved documents. To ensure no test data leakage, the Deduplication Agent filters out any exact or near-duplicates.
Figure 3: Stage II: Multiple instantiations of the Solver Agent independently generate solutions for the synthetic questions produced in Stage I. Generated data at either extremes of difficulty, those solved on nearly all or almost no attempts, are filtered out, retaining only question-answer pairs where learning signal is richest. The Quality Filter Agent screens the remaining data points for ambiguity and incorrect reference answers. Synthetic data that pass both filters serve as inputs to RL training.
Figure 4: Parallel thinking method: We first generate responses in the generation phase and then aggregate the rollouts. The solver agent and the aggregator agent here are the same model on which we apply TTC.
Figure 5: Value-Guided Search method: Performs tree search, using a value model at each step to score candidate continuations, selecting the highest scoring branch. The search process is repeated times followed by aggregation.
Figure 6: aroll harness architecture. The dispatcher feeds prompts to a strategy, which instantiates environment–agent pairs. The agent owns an LLM client and generates actions at each step; the environment executes tool calls, evaluates rewards, and manages context via lifecycle plugins. The interaction loop between agent and environment yields finished rollouts.
| KARL Iter. 1 | KARL Iter. 2 | |
| Dataset / Synthesis Model | (GLM 4.5 Air) | (KARL Iter. 1) |
| BrowseComp-Plus | 1,218 | 1,336 |
| TREC-Biogen | 6,270 | 11,371 |
Table 3: Number of training prompts per dataset and synthesis model across two KARL iterations. To balance training data, we keep the prompt ratio in favor of TREC-Biogen, compensating for its shorter trajectories.
Figure 7: Distribution of trajectory lengths for KARL Iter. 1 (left) and KARL Iter. 2 across training data for BrowseComp-Plus (top) and TREC-Biogen (bottom). For BrowseComp-Plus, Iter. 2 trajectories are substantially shorter than Iter. 1 (median 20 vs 50 steps), indicating that the model learns to search more efficiently. For TREC-Biogen, Iter. 2 trajectories are slightly longer than Iter. 1 (median 6 vs 4 steps), reflecting increased exploration in the second iteration.
| In-Distribution | Out-of-Distribution | ||||||||
| Model | BrowseComp-Plus | TREC-Biogen | FreshStack | FinanceBench | QAMPARI | PMBench | In-Dist. | OOD | Total |
| GLM 4.5 Air | 44.7 | 66.0 | 52.9 | 72.7 | 45.9 | 33.4 | 55.4 | 51.2 | 52.6 |
| Qwen 3.5 397B A17B | 55.8 | 68.2 | 51.9 | 79.3 | 42.8 | 34.7 | 62.0 | 52.2 | 55.5 |
| Minimax m2.5 | 56.5 | 69.3 | 53.3 | 78.0 | 39.3 | 34.5 | 62.9 | 51.3 | 55.2 |
| GPT 5 | 68.3 | 68.2 | 55.6 | 86.7 | 44.4 | 37.5 | 68.3 | 56.1 | 60.1 |
| GPT 5.2 | 47.8 | 62.0 | 47.9 | 80.3 | 41.1 | 37.9 | 54.9 | 51.8 | 52.8 |
Table 4: Main Results: KARLBench results with the highest and second highest scores being bolded and underlined respectively. For both Claude and GPT models, we report the best values across low, medium, and high reasoning efforts. For all baselines, we use the recommended sampling parameters and report the best performance with or without compression as a context management tool. Our distinction of In-Distribution and Out-of-Distribution tasks is specifically with respect to our trained models and does not apply to our baselines. We additionally report single-task RL variants trained on individual in-distribution tasks (KARL-TREC and KARL-BCP) to isolate per-task training effects. VGS denotes value-guided search, a value model based test-time compute method applied at inference time with N candidate trajectories. Finally, par. stands for extra test-time compute with parallel thinking.
Figure 8: Multi-Task Distillation vs Reinforcement Learning: RL yields better out-of-distribution performance than the SFT distilled model, leading to a more general search agent. While SFT distillation benefits from test-time compute in-distribution, it shows negligible out-of-distribution scaling (59.4 59.6), whereas KARL improves consistently across both regimes.
Figure 9: Multi-Iteration Training: (left) We show how our agent continues to learn with each iteration of training on the in-distribution task. (center, right) We also track the out-of-distribution performance of our TREC-Biogen expert and see consistent improvements across iterations on FreshStack and QAMPARI as well.
Figure 10: Test-time Compute Scaling: (left) Training continues to improve the Max@K rather than only improving the Max@1. (right) Scaling continues to improve TTC strategies like Parallel Thinking beyond the base model.
Figure 11: BrowseComp-Plus Data Flow: (left) Alluvial flow chart tracking a training prompt’s Pass@16 performance from GLM 4.5 Air to KARL-BCP. (right) Transition matrix detailing the shift in prompts. The matrix is row-normalized and shows the majority of the movement goes to a more solved state with minimal degradation.
| Ablation | Setting | BrowseComp-Plus Score | BrowseComp-Plus Recall |
| Compression | With | 0.570 | 0.681 |
| Without | 0.389 | 0.503 | |
| Retrieval | Qwen3-Embedding-8B | 0.570 | 0.681 |
| Vector Search (GTE-large + hybrid) | 0.568 | 0.698 |
Table 5: Search Environment Ablations: We ablate both the compression tool and change the embedding model used by the vector search tool. The design choices used during training are bolded. We see that our model shows degradation without compression, while being robust to comparable embedding model choices in the tool.
Figure 12: Search Variations: (left) Varying search horizon from 10 steps to 400 steps. (right) Changing the number of documents returned by a vector search call. With 40 documents retrieved, we flood most of the available context with a single search.
| Compression Model | ||
| Search Model | GLM 4.5 Air | KARL-BCP |
| GLM 4.5 Air | 0.44 | 0.54 |
| KARL-BCP | 0.46 | 0.57 |
Table 6: Search and compression model choice. Rows represent the model used to conduct the search and the columns denote the compression model. We observe that our trained model, when used as a compression model, improves the performance of the GLM 4.5 Air model, suggesting context management has improved through training.
Figure 13: Parallel Thinking: Results spanning all of KARLBench as we scale up from 5 to 20. We compare KARL against GLM 4.5 Air to isolate the gains from reinforcement learning training. The shaded region highlights that KARL consistently outperforms GLM 4.5 Air across all benchmarks and all values of , with gains at ranging from (FinanceBench) to (TREC-Biogen). Notably, the out-of-distribution improvements observed in the single rollout setting are preserved as we scale parallel thinking, indicating that the generalization benefits from RL training are complementary to test-time compute scaling.
| BrowseComp-Plus | TREC-Biogen | FreshStack | FinanceBench | QAMPARI | PMBench | |
| LLM Turns | 3.7 | 1.5 | 1.3 | 1.6 | 2.0 | 2.1 |
| Rollout Token Length | 32156 | 9641 | 14678 | 15105 | 8128 | 20444 |
Table 7: Number of LLM Turns and Token Count During Parallel Thinking with . The final aggregation rollout does not require many extra steps, using at most an average of 3.7 turns to aggregate answers.
Figure 14: Value-Guided Search: Scaling performance of value-guided search on BrowseComp-Plus using various aggregation methods. Both weighted majority voting and best-of-N use the value model as an outcome reward signal for the final aggregation.
Figure 15: Alluvial flow diagrams for search trajectory lengths when grouped by Pass@16 pass-rates, namely, Unsolved , Partial , and Solved . Specifically, we bin each trajectory into 4 bins according to step counts where the lighter the blue the shorter the bins. For each pass rate category, we see a shortening of the trajectories, with the most dramatic decrease observed for the Solved category (51.1 36.3). The average trajectory length is shown above each bar.
Figure 16: Alluvial flow diagrams for search trajectory lengths when grouped by trajectory lengths. For each trajectory length category, training shifts more search rollouts towards Partial or Solved.
Figure 17: Document exploration diversity across BrowseComp-Plus and TREC-Biogen. The cumulative unique documents retrieved over successive vector search queries for GLM 4.5 Air, KARL Iter. 1, and KARL Iter. 2. The bottom panel shows the relative increase over GLM 4.5 Air for KARL iterations. The KARL model improves its search diversity across RL training iterations for both BrowseComp-Plus (+37%) and TREC-Biogen (+8%).
Figure 18: BrowseComp-Plus accuracy conditioned on document retrieval status. Comparison of BrowseComp-Plus accuracy of GLM 4.5 Air, KARL Iter. 1, and KARL Iter. 2 for the subsets of questions for which all, some, or none of the ground truth documents needed to answer the question are retrieved (raw counts in parentheses). With RL training, the models’ answer accuracy improves over the base GLM 4.5 Air model across all the retrieval conditions.
Figure 19: Impact of improved reasoning on search efficiency, measured across 87 BrowseComp-Plus instances where all three models, namely, GLM 4.5 Air, KARL Iter. 1, and KARL Iter. 2, achieved perfect recall of ground truth documents across two rollouts (174 rollouts per model). The faded bottom segment represents the mean number of searches to first retrieve all required documents (exploration/retrieval phase), while the solid top segment shows additional searches conducted after all necessary information was already available (synthesis phase). Percentages indicate the proportion of rollouts yielding correct vs. incorrect answers. Across iterations, not only does the model accuracy improve (53% → 64% → 71%), but the search efficiency also dramatically improves (91 → 52 → 32), with the largest reductions in unnecessary post-retrieval searches on incorrect instances (134.0 95.6 56.5).
Figure 20: Categorization of Various Search Behaviors: Each trajectory is classified into one of six behavioral categories based on LLM-extracted features and rule-based heuristics. Sonnet 4.5 and KARL show similar profiles dominated by Explore then Commit, while GLM 4.5 Air exhibits substantially more Exhaustive Search without convergence and context truncation, suggesting it struggles more to resolve queries within its context budget.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 37: Annotated data processing pipeline for BrowseComp-Plus (top two) and TREC-Biogen (bottom two) across two training iterations of KARL. Synthetic question-answer pairs are first deduplicated, then passed to a Solver Agent that generates 8 rollouts per question. Outputs are categorized as Solved , Partial , or Unsolved , depending on whether the 8 rollouts are all correct, partially correct, or all incorrect. The Partial correct rollouts are optionally filtered through a Quality Filter Agent (in Iter. 1 of BrowseComp-Plus), and then ultimately used as data for our RL pipeline.
| # | Model | Explanation | Answer |
|---|---|---|---|
| 1 | Sonnet 4.5 (Step 25) | Despite extensive searching, I was unable to locate documents that contain all the specific details required to answer this question with confidence. While I found that Liverpool had Phil Thompson as assistant manager in 2001, who was a former team captain in the 1970s, and that Barcelona was founded in 1899, I could not identify a team founded in the 1870s whose manager in 2001 met their spouse at age 20 as a youth player at the club. Additionally, I could not locate the specific match details, including the lineup with the away team’s third substitute, or verify the combined 35 wins statistic for the season. The available documents did not provide sufficient information to conclusively identify the match and the requested player. | Unable to determine with available information |
| 2 | GLM 4.5 Air (Step 200) | Based on extensive searches, I have identified the match as the 2001 Championship (Division One) play-off final between Bolton Wanderers and Preston North End. Bolton Wanderers, founded in 1874 as Christ Church F.C., was managed by Sam Allardyce in 2001. Preston North End was managed by David Moyes, who narrowly missed lifting Preston to the Premier League after losing this play-off final to Bolton. However, despite multiple targeted searches for match details including venue, lineups, substitutes, and away team designation, the available documents do not contain the specific information needed to identify which team was designated as the away team or the name of that team’s third substitute. Play-off finals are typically held at neutral venues (likely the Millennium Stadium in Cardiff for this era), making the “away” designation arbitrary. Without access to the official match report, lineup sheets, or detailed match statistics, I cannot provide the requested information about the away team’s third substitute. | Unable to determine from available sources |
| 3 | KARL (Step 155) | The research identified the match as Bolton vs Arsenal on 29 April 2002. Arsenal was the away team and won 2-0. Their substitutes were Lee Dixon (67th minute), Nwankwo Kanu (70th minute), and Sol Campbell (90th minute). This makes Sol Campbell the third substitute for the away team. The constraints were verified: Pat Rice was Arsenal’s assistant manager and former captain in the 1970s, and the team was founded in the late 19th century (close to the 1870s requirement). | Sol Campbell |
Table 11: Model comparison on BrowseComp-Plus, Query ID 280, question information given above. KARL is able to successfully locate the key details needed and identify the correct answer, without premature termination.
| # | Model | Explanation | Answer |
|---|---|---|---|
| 1 | Sonnet 4.5 (Step 25) | Based on the search results, the person described in the question is Wallah bin Wallah, a renowned Swahili author. He was born in 1956 in Mwanza, Tanzania, and originally had the Luo name Wallah Ndeda. He changed his Luo name from Ndeda to the Arabic-sounding “Wallah bin Wallah.” He hawked fish for three years between Mwanza and Kisumu and also sold peanuts and vegetables to make ends meet while studying. He converted to Islam, which influenced his name change and his advancement in Swahili language studies. Wallah bin Wallah later became one of Kenya’s most successful Swahili authors. His first book was “Malenga wa Ziwa Kuu,” written while teaching at Mbita High School. The book performed well and encouraged him to pursue writing as a career. Given the context that this was his first creative work before producing educational materials such as the “Kiswahili Mufti” series, it would have been a novel or fiction work, as many Swahili authors begin with creative fiction before moving into educational publishing. | Novel/Fiction |
| 2 | GLM 4.5 Air (Step 69) | Based on my research, I identified two potential candidates who match the characteristics described in the question: Wallah bin Wallah and Ken Walibora. Both authors had humble beginnings selling fish and nuts, changed their ethnic names to avoid discrimination, and became renowned Swahili authors. However, their first books are in different genres. Wallah bin Wallah’s first book “Malenga wa Ziwa Kuu” is described as Swahili poetry with the subject “Swahili poetry; Swahili language – Questions, answers, etc.” Ken Walibora’s first book “Siku Njema” is consistently described as an epic Swahili novel and a tale of triumph over adversity. Since the question specifically asks for the genre of the first book, and the search results clearly indicate that Ken Walibora’s “Siku Njema” is fiction (a novel), while Wallah bin Wallah’s first work is poetry, the genre is fiction/novel. | Fiction (novel) |
| 3 | KARL (Step 7) | Based on the summarized search results, the person identified is Wallah bin Wallah, a Swahili author from Kenya (born in Tanzania) who had experience hawking fish and nuts. His first book was “Malenga wa Ziwa Kuu,” published in 1994 by East African Educational Publishers. The descriptions indicate that the book deals with Swahili poetry and Swahili language questions and answers. Since the primary subject matter is explicitly listed as Swahili poetry, this indicates that the genre of his first book is poetry. | Poetry |
Table 12: Model comparison on BrowseComp-Plus, Query ID 61, question information given above. KARL is able to correctly identify the candidate and determine the genre, while other models either assume the genre or confuse the candidate.
| # | Phase | Vector Search Query | Intent Assignment by Claude Opus 4.5 |
|---|---|---|---|
| 1 | Exploration | second most important writer in their country | Starting with a distinctive constraint to narrow down candidates |
| 2 | Exploration | literary work person taken captive three times age | Searching for the plot constraint (captive by someone 3x their age) |
| 3 | Exploration | writer second most important country article January | Combining writer status with January article mention |
| 4 | Exploration | Tristana Benito Pérez Galdós review 2014 | Found the author (Galdós) and work (Tristana)! Now searching for the 2014 review |
| 5 | Exploration | Benito Pérez Galdós family siblings parents strong-willed | Verifying Galdós matches family constraints before committing |
| 6 | Exploration | Benito Pérez Galdós January article underestimated talents | Checking the “underestimated talents” constraint from January article |
Table 13: Analysis of vector searches made by our model for BrowseComp-Plus, Query ID 472.
| # | Model | Explanation | Answer |
|---|---|---|---|
| 1 | GLM 4.5 Air (Step 200) | The model identified “The Freddie Stories” by Lynda Barry and “Sasquatch Books” as the publisher by query #4 out of 161 total queries. It verified 4 of 5 constraints but could not verify that the author is known by a name matching a Star Wars character. The model then entered a loop, alternating between re-verifying the publisher and searching for the Star Wars connection: Q#4: ‘‘The Freddie Stories’’ Lynda Barry original publisher 1990s Sasquatch Books Q#11: Sasquatch Books founded 1980s history establishment Q#13: Lynda Barry pen name Star Wars character Q#26: ‘‘The Freddie Stories’’ Lynda Barry Sasquatch Books Q#52: Lynda Barry pen name Star Wars character Q#73: ‘‘ Sasquatch Books ’’ founded 1986 Q#101: ‘‘The Freddie Stories’’ Lynda Barry publisher Sasquatch Books Q#131: The Freddie Stories Lynda Barry publisher Sasquatch Books Q#161: Lynda Barry pen name Star Wars character “Sasquatch Books” appeared in 29 of the model’s own queries, yet it never committed to an answer, still issuing a vector_search call at step 201. | (No answer—still searching at cutoff) |
| 2 | GLM 4.5 Air (Step 200) | The second attempt initially pursued “Drawn & Quarterly” as the publisher, spending 80+ queries trying to verify it was founded in the 1980s (it was founded in 1990). At query #89 the model pivoted to Sasquatch Books: Q#3: Drawn & Quarterly founded 1980s publisher Q#45: ‘‘Drawn & Quarterly’’ founded 1980s publisher history Q#75: Drawn & Quarterly founded 1990 establishment Q#89: Sasquatch Books founded 1980s publisher history Q#104: ‘‘The Freddie Stories’’ Lynda Barry publisher Drawn Quarterly Sasquatch Books Q#125: ‘‘ Sasquatch Books ’’ founded 1988 history Q#150: ‘‘Lynda Barry’’ pen name ‘‘Star Wars’’ character name Q#160: Lynda Barry author different name Star Wars character As in the first attempt, the model confirmed Sasquatch Books (founded 1986) but could not verify the Star Wars name constraint. It exhausted all 201 steps without producing an answer. | (No answer—still searching at cutoff) |
| 3 | KARL (Step 57) | The model followed a similar initial search path but with far fewer queries: Q#1: graphic novel 1990s boy fire accusation woman dies Q#2: ‘‘seriously terrible’’ childhood graphic novel Q#8: Sasquatch Books founded 1980s publisher history Q#9: Lynda Barry pseudonym other name Star Wars character Chewbacca Q#26: ‘‘The Freddie Stories’’ Lynda Barry publisher Sasquatch Books 1999 Like GLM 4.5 Air, it searched repeatedly for the Star Wars name connection without finding definitive evidence. At step 56, the model explicitly reasoned: “While the research did not find specific evidence of Lynda Barry having a pen name or alternate name that matches a Star Wars character’s name, given the perfect match on all other highly specific constraints, this appears to be the correct novel and publisher.” | Sasquatch Books |
| 4 | KARL (Step 7) | On the second attempt, the model resolved the question in 4 search queries and 7 steps: Q#1: graphic novel 1990s boy childhood misadventures fire house woman dies Q#2: ‘‘The Freddie Stories’’ Lynda Barry 1990s original publisher Q#3: Sasquatch Books founded 1980s publisher history Q#4: Lynda Barry other name pen name Star Wars character It then immediately committed without requiring exhaustive verification. | Sasquatch Books |
Table 14: Comparing the outputs of GLM 4.5 Air and KARL on a BrowseComp-Plus question, Query ID 257, for which the former fails to commit to an answer it has already found.
| # | Phase | Vector Search Query | Intent / Reasoning by Model |
|---|---|---|---|
| 1 | Exploration | cricket match winning team batsman more sixes than fours took two wickets losing team won toss | Starting with distinctive constraints: highest scorer hit more 6s than 4s AND took 2 wickets |
| 2 | Exploration | cricket match run rate difference more than 6 losing team won toss both teams batted all overs | Focusing on run rate difference and toss constraint; results were “general cricket discussions” |
| 3 | Exploration | cricket batsman more sixes than fours two wickets losing team toss | Refining search for allrounder stats; “still not getting specific match information” |
| 4 | Exploration | cricket match 75 80 runs total winning team losing team highest scorer 35 45 | Trying specific run totals from constraints; results “mostly general cricket discussions” |
| 5 | Exploration | cricket match batsman more sixes than fours two wickets last ball | Adding last ball wicket constraint; recognizes this is “quite specific” |
| 6 | Exploration | cricket match 75 80 runs winning team batsmen losing team highest scorer 35 45 | Slight variation on run totals; “searches haven’t been fruitful” |
Table 15: Failure case analysis for BrowseComp-Plus Query ID 1185. KARL exhibits a failure to perform numerical reasoning, and is unable to find the correct answer.
| # | Model | Explanation | Score |
|---|---|---|---|
| 1 | GLM 4.5 Air (Step 5) | The model performed 4 searches exploring standard pathological fracture causes: Q#1: acute fractures both hands no injury trauma causes Q#2: atraumatic fractures hands pathological stress osteoporosis Q#3: bilateral hand fractures metabolic bone disease endocrine Q#4: rheumatoid arthritis bilateral hand fractures atraumatic The final response (2,752 chars) covered five categories: metabolic bone diseases, stress fractures, malignancy, rheumatologic disorders, and endocrine disorders—but missed seizures, medication-induced fragility, renal disease, and genetic disorders: “The main categories include: 1. Metabolic Bone Diseases: Osteoporosis… hyperparathyroidism… Cushing’s syndrome. 2. Stress and Insufficiency Fractures… 3. Malignant Conditions: Multiple myeloma… 4. Rheumatologic and Inflammatory Disorders… 5. Other Endocrine Disorders.” Missed: N2 (genetic disorders), N4 (medications), N5 (seizures), N6 (renal failure). ✓ N1 × N2 ✓ N3 × N4 × N5 × N6 ✓ N7 ✓ N8 ✓ N9 | 5/9 nuggets (0.56) |
| 2 | GLM 4.5 Air (Step 5) | With one fewer search, the first attempt covered a different subset—finding genetic disorders (N2) and renal disease (N6) but missing endocrine causes (N3) and malignancy (N8): Q#1: acute fractures both hands no injury trauma causes Q#2: atraumatic fractures hands pathological stress osteoporosis Q#3: bilateral hand fractures metabolic bone disease endocrine Q#4: rheumatoid arthritis bilateral hand fractures atraumatic Across both attempts, GLM 4.5 Air never searched for seizure-related fractures, medication-induced bone loss, or genetic bone disorders by name. Missed: N3 (endocrine), N4 (medications), N5 (seizures), N7 (stress), N8 (malignancy). ✓ N1 ✓ N2 × N3 × N4 × N5 ✓ N6 × N7 × N8 ✓ N9 | 4/9 nuggets (0.44) |
| 3 | KARL (Step 8) | The model performed 7 searches, with the first two similar to GLM 4.5 Air. Crucially, searches 3–7 targeted specific non-obvious causes : Q#1: acute bilateral hand fractures without injury trauma causes Q#2: pathological stress fractures bilateral hands no trauma causes Q#3: osteoporosis metabolic bone disease bilateral hand fractures no trauma Q#4: seizure disorder bilateral fractures hands no trauma mechanism Q#5: endocrine disorders vitamin deficiency bilateral hand atraumatic fractures Q#6: medication induced bone fragility bisphosphonate bilateral hand fractures Q#7: malignancy cancer pathological fractures bilateral hands no trauma After discovering seizure-related fractures in the initial results, the model reasoned: “I can see that seizures can cause bilateral fractures… The mechanism is related to violent involuntary muscle contractions” and followed up with a dedicated search (Q#4). Similarly, it explicitly searched for medication-induced bone loss (Q#6). The final response (5,476 chars) included an entire section absent from the base model’s answer: “ Seizure-Related Mechanisms: Generalized tonic-clonic seizures can cause bilateral fractures due to violent muscle contractions [20544651, 26307650]… Drug-Related Bone Loss: Long-term bisphosphonate therapy associated with atypical fragility fractures [36769684]… chronic corticosteroid use causing glucocorticoid-induced osteoporosis [17664365].” ✓ N1 ✓ N2 ✓ N3 ✓ N4 ✓ N5 ✓ N6 ✓ N7 ✓ N8 ✓ N9 | 9/9 nuggets (1.00) |
| 4 | KARL (Step 7) | In a second attempt with 6 searches, the model again searched for non-obvious causes early: Q#1: acute fractures both hands without injury non-traumatic causes Q#2: pathological fractures bilateral hands stress fractures osteoporosis Q#3: seizure fractures bilateral hands osteomalacia metabolic bone disease Q#4: bisphosphonate atypical fractures bilateral hands rheumatoid arthritis Q#5: reflex sympathetic dystrophy complex regional pain syndrome hand fractures Q#6: spontaneous hand fractures metabolic endocrine causes hyperparathyroidism This attempt covered 7/9 nuggets, missing only genetic disorders (N2) and malignancy (N8). ✓ N1 × N2 ✓ N3 ✓ N4 ✓ N5 ✓ N6 ✓ N7 × N8 ✓ N9 | 7/9 nuggets (0.78) |
Table 16: Comparing GLM 4.5 Air and KARL on a TREC-Biogen question about non-traumatic hand fractures.
Figure 40: Accuracy of aggregation rollout versus candidate rollouts : The aggregation rollout is able to effectively combine the correct parts of each candidate rollout, to produce a final, cohesive answer for an open-ended response question from PMBench , asking which companies care about a certain topic, and why.
| Category | Description | Rule Criteria |
| Running Out of Context | Trajectory truncated by the context limit while still actively searching. | Truncated flag is set, or context usage with search activity in final steps. |
| Exhaustive Search, No Convergence | Extensive search effort but fails to commit to a definitive answer. | Not truncated, 10 search actions, and agent either does not attempt an answer or expresses uncertainty with low confidence. |
| Giving Up Early | Agent stops searching before exhausting context, with low effort. | Context usage , no search activity at end, 8 searches, and trajectory ended early or unclearly. |
| Confidently Wrong Early | Agent locks onto an incorrect answer early without adequate exploration. | Incorrect answer proposed in first half of steps, high/medium confidence, and answer unchanged. |
| Explore then Verify | Broad search phase, candidate answer proposed, then explicit cross-checking. | 2 verification searches after first answer, with answer proposed before 70% of steps. |
| Explore then Commit | Broad search phase followed by a committed answer, with no verification. | Final answer present and 2 unique queries in the first third of the trajectory. |
Table 17: Behavioral categories for search traces. Each rollout is first processed by an LLM to extract structured behavioral features (e.g., number of search actions, when an answer was first proposed, whether post-answer searches were verification or continued exploration). These features are then passed through the rule-based classifier above, applied in priority order where the first matching rule determines the category. A second judge pass independently classifies each rollout; the two labels are merged by defaulting to the rule-based result unless the LLM disagrees with 95% confidence or the rules return no match.
Figure 41: Distribution of context compression steps per question on BrowseComp-Plus ( = 230). Each point represents the average number of compression events across 2 rollouts for a single sample. The Kernel Density Estimation (KDE) on top reveals a strongly right-skewed distribution: the majority of questions require few compressions (median = 6), while a long tail of harder questions drives the mean to 10.2. Orange points highlight the 47 samples (20%) requiring 20 or more compression steps.
Figure 42: Testing Suite: A screenshot of our testing app. This app is deployed to gather preferences on real-world queries that users may ask and keep a running scoreboard of preferred models.
Figure 43: Qualitative Analysis Tooling: Light-weight tooling helps us understand behavior patterns and differentiated search strategies between candidate models. For example, we showed in Table 13 that models can spend many steps on verification. Our viewer allows us to compare behavior between models, per question, and gain a deeper understanding of time spent grounding, exploring, verifying, etc.