LongCapHarness: Capability-Guided Auto-Harnessing for Long-Context Reasoning
Organizations: Microsoft Research Asia
Abstract
Existing methods enhancing an LLM's long-context reasoning capability often rely on data synthesis and follow-up LLM post-training. However, the data synthesis strategy and the post-training recipes are empirically driven and computationally intensive. In this work, we propose a new framework, dubbed LongCapHarness, to instead harness the underlying capabilities for long-context reasoning. The harnessing process recursively revises and updates capability primitive implementation, capability state and control logic through recursive self-improvement on the development set. With neither data synthesis nor LLM post-training, LongCapHarness is applied to Llama3-8B and transferred to Qwen2.5-3B. LongCapHarness substantially outperforms recent data-synthesis-based baselines on HELMET and RULER benchmarks; its estimated development cost is less than 50% of the estimated training costs of NExtLong and PolicyLong. We hope these results motivate further research into harnessing the latent capabilities of LLMs to solve tasks beyond their explicit training.
Figures & tables
| Model | Harness | Context | Avg | H-Avg | Recall | RAG | ICL | Rank | LongQA | RULER |
| Zero-shot | ||||||||||
| Llama3-8B | – | 8K | 15.14 | 12.48 | 18.00 | 12.68 | 13.60 | 7.74 | 10.38 | 17.80 |
| Qwen2.5-3B | – | 32K | 61.91 | 49.01 | – | – | – | – | – | 74.81 |
| Data Synthesis + LLM Post-training | ||||||||||
| Llama3-8B-NExtLong-128K | – | 128K | 70.15 | 58.80 | 82.56 | 60.91 | 81.76 | 31.47 | 37.30 | 81.50 |
| Llama3-8B-EntropyLong-128K | – | 128K | – | – | – | – | – | – | – | 87.37 |
| Method | Medium | Long |
| Base | 21.50 | 20.60 |
| NExtLong | 23.70 | 26.90 |
| EntropyLong | 24.20 | 28.70 |
| PolicyLong | 27.10 | 31.50 |
| LongCapHarness | 34.47 | 27.85 |
| Version | 8K | 16K | 32K | 64K | 128K | Mean |
| Initial code-based implementation (both models) | ||||||
| V0: BM25 + sort | 53.26 | 45.44 | 39.52 | 34.66 | 28.85 | 40.35 |
| Llama3-8B | ||||||
| V1: LLM scoring + sort | 65.51 | 51.88 | 36.50 | 22.95 | 17.89 | 38.95 |
| V2: Comparison + merge | 57.79 | 47.48 | 36.70 | 30.85 | 25.51 | 39.67 |
| Qwen2.5-3B | ||||||
| Method | H100 GPU-hours | Cost ($) | |
| Model Training | |||
| NExtLong ( Gao et al., 2025b ) | 960 | 3,696.00 | |
| PolicyLong ( Jia et al., 2026b ) | 482.81 | 1,858.82 | |
| Method | Input (M) | Output (M) | Cost ($) |
| Development (GPT-5.6 Sol) | |||
| RRSI ( Xia et al., 2026 ) | 52.42 | 3.87 | 331.56 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Development | Test/evaluation | Total |
| HELMET | 2,080 | 27,035 | 29,115 |
| RULER | 112 | 32,388 | 32,500 |
| LongBench-v2 | 50 | 403 | 453 |
| Total | 2,242 | 59,826 | 62,068 |
| Category | Development | Test | Total |
| Recall | 70 | 1,930 | 2,000 |
| RAG | 945 | 9,555 | 10,500 |
| ICL | 75 | 12,425 | 12,500 |
| Re-rank | 210 | 405 | 615 |
| LongQA | 450 | 1,050 | 1,500 |
| Citation | 30 | 970 | 1,000 |
| Input length | Development | Test | Total |
| 8K (8,192) | 458 | 11,865 | 12,323 |
| 16K (16,384) | 431 | 11,892 | 12,323 |
| 32K (32,768) | 438 | 11,885 | 12,323 |
| 64K (65,536) | 433 | 11,890 | 12,323 |
| 128K (131,072) | 432 | 11,891 | 12,323 |
| Total | 2,192 | 59,423 | 61,615 |
| Category | Subtask | Dev. | Test | Total |
| Recall | json_kv | 54 | 446 | 500 |
| Recall | niah_multikey_2 | 5 | 495 | 500 |
| Recall | niah_multikey_3 | 5 | 495 | 500 |
| Recall | niah_multivalue | 6 | 494 | 500 |
| RAG | rag_hotpotqa | 45 | 1,455 | 1,500 |
| RAG | rag_natural_questions | 90 | 2,910 | 3,000 |
| Domain | Development | Test | Total |
| Single-Document QA | 10 | 165 | 175 |
| Multi-Document QA | 10 | 115 | 125 |
| Long In-context Learning | 10 | 71 | 81 |
| Long-dialogue History Understanding | 10 | 29 | 39 |
| Long Structured Data Understanding | 10 | 23 | 33 |
| Total | 50 | 403 | 453 |
Long-Context Inference
Long-context inference studies how language models retrieve, retain, and reason over information spanning very large inputs.
See all 54 papers