ODSA: Orchestrated Discovery and Self-Aggregation
Organizations: Columbia Engineering Columbia University · School of Electrical Engineering and Computer Science KTH Royal Institute of Technology
Abstract
Test-time scaling improves the accuracy of a language model's answers by spending more computation at inference time. Established methods do so either sequentially, revising a single solution over several passes, or in parallel, generating several independent solutions and selecting among them. We present ODSA (Orchestrated Discovery and Self-Aggregation), an LLM-directed scaling framework in which an orchestrator LLM is given existing test-time scaling methods as actions --- discovery of a new independent solution, comparison of existing solutions to build a better one, and acceptance of a proposed answer, which triggers its verification --- and directs stateless workers to carry them out. The orchestrator does not solve the problem itself: it allocates computation, feeds the results of earlier actions into later ones, and takes every action from a single context, so that each decision is informed by everything tried before. On Humanity's Last Exam, ODSA improves accuracy by 4.7 to 18.0 points across three models over a single direct call to the model with no test-time scaling. On LiveCodeBench, where a single direct call already solves most problems, it improves accuracy by 0.6 to 14.0 points.
Figures & tables
| Gemma 4 31B-IT | GPT-OSS-120B | DeepSeek V4 Flash | ||||
| Method | Accuracy | $ / problem | Accuracy | $ / problem | Accuracy | $ / problem |
| HLE | ||||||
| Baseline (single sample) | 42.7 4.2% | 0.003 | 45.3 3.1% | 0.004 | 62.0 4.0% | 0.007 |
| Self-Refine (1 round) | 44.7 3.1% | 0.005 | 39.3 3.1% | 0.007 | 60.7 1.2% | 0.017 |
| Self-Refine (iterated) | 54.0 3.5% | 0.050 | 44.0 5.3% | 0.049 | 64.0 2.0% | 0.162 |
| Majority voting | 44.7 5.0% | 0.049 | 49.3 5.0% | 0.045 | 65.3 1.2% | 0.156 |
| Gemma 4 31B-IT | GPT-OSS-120B | DeepSeek V4 Flash | ||||
|---|---|---|---|---|---|---|
| Method | Accuracy | $ / problem | Accuracy | $ / problem | Accuracy | $ / problem |
| ODSA | 60.7 2.3% | 0.046 | 50.0 3.5% | 0.044 | 75.3 2.3% | 0.157 |
| without verification | 64.0 8.0% | 0.022 | 46.0 5.3% | 0.026 | 72.0 5.3% | 0.117 |
| without comparison | 60.0 7.2% | 0.047 | 48.0 2.0% | 0.045 | 74.7 6.1% | 0.164 |
| without directives | 60.7 6.4% | 0.052 | 57.3 4.6% | 0.043 | 72.0 2.0% | 0.152 |
| Method | Gemma 4 31B-IT | GPT-OSS-120B | DeepSeek V4 Flash |
|---|---|---|---|
| Majority voting | 53.1 4.7% | 65.3 10.3% | 68.5 1.8% |
| Best-of- | 45.9 6.3% | 73.8 10.0% | 79.5 2.8% |
| GSA | 74.7 6.8% | 72.8 4.7% | 73.8 3.1% |
| ODSA | 80.6 3.8% | 80.6 1.0% | 81.9 2.5% |
Test-Time Scaling
Test-time scaling studies how extra inference-time computation can improve model answers, decisions, and generated outputs.
See all 243 papers