Octopus: One Brain, Many Branches for Latent-KV Multi-Agent Reinforcement Learning
Organizations: Stony Brook University · University of Wisconsin–Madison · University of Minnesota
Abstract
Deep search and long-horizon reasoning unfold as branching trajectories that pursue parallel sub-queries, abandon dead-end detours, and cross-check sibling findings. Yet modern serving engines, harnesses, and RL trainers execute episodes as flat, append-only sequences: stretching a context window leaves failed detours that dilute attention, in-place edits break KV caching and recurrent states, and all-layer cache splicing overwrites hybrid recurrent working memory and triggers hijacking. We introduce Octopus, a co-designed serving runtime, memory manager, and RL framework executing episodes as a GPU-resident DAG over radix cursors (89.6% KV hit rate). Agents coordinate via five primitives: fork shares parent KV prefixes in with isolated 32K budgets, rewind restores recurrent checkpoints, read_thread mounts sibling KV pages in zero copy, and wait and done synchronize results. Octopus splices sibling KV pages only into full-attention layers under monotonic position offsets while recurrent layers skip mounted spans, retaining 92.2% of full-text answer information and cutting hijacking () with zero new KV pages. During RL, our packed DAG kernel and DAG-GRPO objective deduplicate shared prefixes via loss masks while keeping mounted KV edges differentiable, cutting rollout-training KL divergence and routing reader gradients into producer KV projections. Trained on Qwen3.5-4B with binary rewards, Octopus self-organizes a commander-worker hierarchy and latent KV mesh, lifting BrowseComp-Plus accuracy under BM25 from 26.0% (32K ReAct) and 32.1% (143K GRPO) to 49.6% (+17.5 pts; 51.2% on OpenResearcher), surpassing 9B context-editing RL (42.5%), matching o3 (49.3%), and outperforming dense-retrieval agents up to larger: Tongyi-DeepResearch (44.5%), DeepSeek-R1 (16.4%), Kimi-K2 (35.4%), GPT-4.1 (36.4%), and Claude 4 Opus (36.8%).
Figures & tables
| System, Memory & RL Dimension | 1. Long Window (Qwen-1M / Single-Agent RL) | 2. Edit / Fold (CLM+SCR / Context-Folding) | 3. Multi-Agent (AutoGen / CAMEL / APR) | 4. Prior KV Sharing (Blend / Hogwild! / KVComm) | 5. Sequence Agent RL (Search-R1 / RAGEN + Leaf-Split Trees) | Octopus (Ours) (Cursor + Attn Mount + DAG-GRPO) |
| 1. Execution Topology & Isolation | ✗ Flat Linear Detour pollution dilutes attention | Sequential Fold Depth- serial; no path speedup | ✓ Parallel Chats Isolated sessions, no shared KV | ✓ Parallel Tree Inference-only KV block sharing | ✗ Flat Sequences Single-chain rollout or leaf-split tree | ✓ Parallel CoW Tree + DAG Unlimited depth ( /branch) + lateral peer read_thread |
| 2. Per-Turn Wire & Token Protocol | ✗ Resend Full history resend & template drift | ✗ Re-Tokenize Re-renders prompt after mid-edits | ✗ Resend Repeated text serialization | Block Sync External position & block tracker | ✗ Stitch Re-tokenizes flat turns every step | ✓ Server Cursor Immutable radix cursor + startup template probe |
| 3. Prefix KV & Hybrid State Reuse | Append-Only Invalidated upon any backtrack | ✗ Breaks Recurrence Mid-edit / fold invalidates state | ✗ Isolated cross-agent KV cache reuse | ✗ Attn-Only / All-Layer Corrupts hybrid recurrent state | ✗ Recomputes Repeats shared trunk per leaf | ✓ Shared GPU Radix Tree ref-counted KV prefix + cloned recurrent state |
| 4. Cross-Branch KV Sharing & Hijack Safety | — Single-thread execution only | ✗ Lossy Text Return Drops IDs; text pollutes memory | ✗ Text Copy Raw transcript hijacks reader | ✗ Copy / Unshifted RoPE copy pages or longer-sibling flip | — No cross-branch state reading | ✓ -Copy Attn KV Mount Recurrent layers skip mount + monotonic pos. offset |
| 5. Backtrack Recovery & Tree Closure | ✗ No Backtrack Dead-end docs remain in window | KV Truncate Only No recurrent snap or lateral read_thread | ✗ No State Rewind Cannot roll back branch KV or state | ✗ No State Snap Cannot roll back recurrent state | ✗ No Branch Rewind Append-only turn rollout buffer | ✓ Exact State Checkpoint Reclaims branch budget + termination prompt guard |
| 6. RL Kernel & Rollout Alignment | ✗ Detour Noise Failed detours dilute credit | Cut at Segments No cross-branch KV mount edges | ✗ No Shared-KV RL Independent chats or text-summary joins | ✗ Inference-Only No packed tree training kernel | ✗ Drops KV Mounts Sequence pass breaks rollout alignment | ✓ Prefix-Tree Packed Kernel Prefix-deduplicating mask + differentiable KV mounts |
| Primitive | Signature | Cursor & KV-Cache Semantics | Control-Flow & Budget Semantics |
| fork | fork(branches= {id: prompt}) | Clones parent cursor node in ; shares radix KV prefix; clones recurrent state . | Spawns parallel child branches, each granted a fresh token growth budget. |
| read_thread | read_thread(branch, steps=[a, b]) | Zero-copy lends branch ’s KV pages to attention layers w/ monotonic pos. ( Equation 2 ); recurrent layers skip span. | Injects a compact metadata stub ( tokens) into reader ’s prompt without copying ’s text. |
| wait | wait(branches=[...]) | Suspends caller cursor until target child branches reach terminal COMPLETED state. | Returns brief branch status summaries and step manifests available for read_thread . |
| rewind | rewind(step=k, note) | Restores cursor and exact recurrent state snapshot at step ; appends diagnostic note . | Reclaims dead-end token budget ( ) while retaining the pruned branch in the training DAG. |
| done | done(answer, evidence) | Marks branch or root cursor COMPLETED and pins its KV pages in GPU radix pool for parent/sibling read_thread . | Enforced by a answer reserve and loss-masked termination prompt; returns verified answer and evidence. |
| OR Val | BrowseComp-Plus Official Test Benchmark (830 Queries) | |||||||||
| Method / [-2.0pt] Agent System | Backbone / [-2.0pt] Scale | Tools | PF/Q [-2.0pt] | Val [-2.0pt] Acc. | Test Acc. [-2.0pt] BM25 / Dense † | Search [-2.0pt] Rec. | Open [-2.0pt] Rec. | Gold [-2.0pt] Rec. | Context [-2.0pt] Used / Cap | Crit. [-2.0pt] / Total |
| Proprietary Frontier Models on BrowseComp-Plus [ 2 , 3 ] | ||||||||||
| Claude Sonnet 4 [ 2 ] | Proprietary | s-only ∗ | — | — | ( ) | — | — | |||
| Gemini 2.5 Pro [ 2 , 3 ] | Proprietary | s+o+f | — | — | ( ) | — | — | |||
| GPT-4.1 [ 2 , 3 ] ⋄ | Proprietary | s+o+f | — | — | ( ) | — | — | |||
| Claude 4 Opus [ 2 , 3 ] | Proprietary | s+o+f | — | — | ( ) | — | — | |||
| Context Management [-2.0pt] & Serving Architecture | Explored Context [-2.0pt] (Active / Win. Cap) | Critical-Path [-2.0pt] Token Load | Gold Open Rec. [-2.0pt] / BCP Acc. | GPU Prefix KV [-2.0pt] Cache Reuse | 128K Assembly [-2.0pt] & Wire Cost |
| Single-Agent ReAct (Stateless ) | ( of cap) | tok ( ) | / | Append-only ( resend) | / |
| Single-Agent GRPO (Flat Window) | ( of cap) | tok ( bloat) | / | Polluted window ( pruned) | / |
| In-Place Context Edit (CLM+SCR, ) [ 15 ] | Multi-edit ( active cap) | serial edits | — / ( PF/Q) | edit re-prefill (breaks ) | Suffix copy & re-rotate |
| Octopus (Ours, Branches) [-1.5pt] Server-Resident Radix Cursor | expl. / net [-1.5pt] ( active; hard) | compressed [-1.5pt]( tok) | / [-1.5pt] ( gold; ) | GPU KV Hit [-1.5pt]( prefix re-prefill) | ( ) [-1.5pt]( , ) |
| Cross-Branch Context-Sharing [-2.0pt] Mechanism | Extra KV [-2.0pt] Memory | Full-Text Info [-2.0pt] Retained | Exact Match [-2.0pt] All (Longer Sib. ) | Policy Drift [-2.0pt] NLL | Off-Task [-2.0pt] Hijack Rate |
| No Sibling Read ( none ) | Pages | ( rec.) | ( ) | ||
| Transcript Copy (AutoGen [ 5 ] ) | ( ) | ( rec.) | ( ) | ||
| Full-Layer KV Splice (CacheBlend [ 20 ] ) | ( ) | ( rec.) | ( ) | ||
| Unshifted Zero-Copy KV | Pages ( -Copy) | ( rec.) | ( ) | ||
| RoPE-Relocated KV Copy (KVCOMM [ 22 , 21 ] ) | ( ) | ( rec.) | ( ) | ||
| Octopus Zero-Copy Mount (Ours) [-1.5pt] Attn-Only Monotonic Offset ( Equation 2 ) | Pages ( -Copy) | Info [-1.5pt]( – rec.) | ( ) [-1.5pt] (0-shot: ) | [-1.5pt]( vs. Text) | ( 0-shot) [-1.5pt] ( vs. Text) |
| Multi-Agent Topology [-2.0pt] & Training Regime | Max Threads [-2.0pt] (Depth, Fanout) | Root [-2.0pt] Fork Rate | Workers / Ep [-2.0pt] (Child Ratio ) | Deep ( ) [-2.0pt] / Full 10-Br. | Rewind Activity [-2.0pt] (Root / Worker) | OpenResearcher [-2.0pt] Acc. | BCP Test [-2.0pt] Acc. |
| Single-Agent ReAct ( Window) | ( ) | ( ) | / | — / — | (Val) | ||
| Single-Agent GRPO ( Flat) | ( ) | ( ) | / | — / — | (Val) | ||
| Multi-Agent Fork (0-Shot Base) | ( ) | ( Val) | ( Val) | / | rw/ep ( st) | (Val) | ( ) |
| Octopus RL (Parallel Team, , Step 35) | ( ) | ( ) | / | /ep / /br | (Val) | (+10.5) | |
| Octopus RL (Parallel Team, , Step 55) | ( ) | ( ) | / | /ep / /br | 51.2% (Val) | — | |
| Recursive 0-Shot ( on Policy) | ( ) | ( ) | / — | — / — | ( ) | — |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Multi-Agent Reinforcement Learning
Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.
See all 402 papers