MoE-Context: Show the Model Only What It Needs
Organizations: UpThink EduTech Services Pvt. Ltd.
Abstract
AI agents read their whole history at every step. People do not work like that: we keep what the moment needs and hold the rest as a rough note we can recall. MoE-Context applies this to language models. The context is split into blocks. Before each step a fast yes or no pass decides which blocks the step needs; the rest stay as one-line notes that can be brought back, and nothing is deleted. In the version we build, one model makes this decision and then answers, over a single key-value cache where hidden blocks are masked rather than removed. On a laptop with Qwen3-1.7B and a synthetic lookup task, showing only the needed block raises accuracy from 93 to 119 of 120, and a masked block is as hidden as a deleted one. The model's own picking is the weak part. A supervised linear probe written into spare output rows lifts its accuracy from 80 to 110 of 120, but simple keyword search picks perfectly on this task, and the probe does not transfer to a second task. We report these results openly, along with two cheap usage patterns that block switching enables: undoing a failed attempt and running subagents without a new context.
Figures & tables
| Keeps the | Can bring | Checks every | One model | |
|---|---|---|---|---|
| original | it back | block, every step | picks and answers | |
| Cut or summarize [ 22 , 8 , 9 ] | ||||
| Model manages itself [ 13 , 10 , 17 ] | ||||
| Fixed rules [ 12 , 15 ] | ||||
| Learned controller, untested [ 3 ] | ||||
| Decision model, once [ 20 ] |
LLM Context Management
Large language model context management studies how to select, compress, retrieve, and organize information within bounded model inputs.
See all 265 papers