Microfitting: Continual Learning via Cached Test-Time Training
Organizations: Interval
Abstract
Continual learning is a promising approach to specialize language models to an organization's private data. However, current methods exhibit a number of drawbacks including catastrophic forgetting, where new training updates overwrite prior knowledge. We present microfitting, which trains and caches a small adapter for each document. Each adapter is trained once, offline, on synthetic questions targeted at demonstrated base-model shortcomings and reused whenever that document is queried. This method borrows from test-time learning literature and sidesteps the problem of continual learning through parameter isolation, allowing the system to specialize on new data without overwriting prior knowledge. The process that generates adapter training data also improves over time, revising its recipe from previous outcomes and fresh examples of the target workload. On DeepSeek V4 Flash, microfitting substantially improves accuracy over the base model. In Enterprise Document Operations (EDO), an open-book Q&A environment over real-world enterprise documents, our method achieves a 38.2% relative improvement over base-model performance. Microfitting makes continual specialization practical and cost-efficient: a frozen model can adapt to an evolving private corpus overnight on local hardware, without modifying the base model or previously cached adapters.
Figures & tables
| base model | DeepSeek V4 Flash, frozen |
|---|---|
| adapter | LoRA rank 16, shared-expert MLP |
| trainable parameters | 12.7M per adapter |
| documents per batch | 10 |
| questions per document | 40 |
| rows per step | 8, one question each |
| steps per epoch | 50 |
| iteration | policy | baseline Q&A adapter | gain (points) | relative improvement |
|---|---|---|---|---|
| 1 | ||||
| 2 | ||||
| 3 |
| questions per document | accuracy | average time per document |
|---|---|---|
| 12 | .632 | 0.44 min |
| 24 | .696 | 0.90 min |
| 40 | .735 | 1.48 min |
| 85 | .725 | 3.17 min |
| training condition | 1 | 2 | 3 | 10 | 10 + general |
|---|---|---|---|---|---|
| evaluation accuracy | .697 | .738 | .775 | .835 | .835 |
| cumulative configuration | warm s/step | 50-step time | speedup |
|---|---|---|---|
| baseline (RAM offload) | 134 | 112 min | |
| cache + 2 q./row | 57 | 52 min | |
| windowing + shared prefix | 36 | 32 min | |
| 4 q./row + sharing | 18.3 | 16.8 min |
| question | base model | gold |
|---|---|---|
| In the Home Office immigration and nationality fees: 9 April 2025 schedule, a sponsor applies in the UK for a worker holding a four-year certificate of sponsorship for a job on the Immigration Salary List that is not a Health and Care Visa job. Using section 7 (Work – applications made in the UK), how much more would that application cost from 9 April 2025 if the job were not on the Immigration Salary List? | £115 | £591 |
| In the press release ‘BlackLine Announces Third Quarter 2025 Financial Results’, take the individual operating expense line of the Condensed Consolidated Statements of Operations with the smallest amount in the nine months ended September 30, 2025 (excluding the total line). By what percentage did it grow against the same period of 2024? Give it to one decimal place. | 6.1% | 398.3% |
| In ‘Addus HomeCare Announces Third Quarter 2025 Financial Results’, suppose adjusted EBITDA for the third quarter of 2025 had grown year over year at the rate the Overview gives for net service revenues rather than at its own rate. What would adjusted EBITDA for the quarter have been, to the nearest $0.1 million? | $43.1 million | $42.9 million |
| In the Eurostat release ‘Industrial production down by 0.5% in the euro area and by 0.3% in the EU’, take the grouping with the smallest increase in the euro area bullet list for August 2026 compared with July 2026. In the euro area block of the ‘compared with same month of the previous year’ table, by how many percentage points is its August 2026 figure above its March 2026 figure? | 0.4 percentage points | 5.5 percentage points |
| In the Child Nutrition Programs: Income Eligibility Guidelines (July 1, 2025–June 30, 2026), for a household of two, which of the chart’s three area blocks gives the lowest Reduced Price Meals–185% annual figure, and how much lower than the next-lowest block’s figure is it? | Hawaii; $4,096 lower | 48 Contiguous States, $5,864 lower |
| items | base | microfit | corrected | regressed | |
|---|---|---|---|---|---|
| 1–3 | 51 | .42 | .80 | .70 | .05 |
| 4–6 | 61 | .43 | .64 | .57 | .27 |
| 7–10 | 55 | .39 | .55 | .46 | .30 |
| 11–15 | 31 | .32 | .42 | .33 | .40 |
Continual Learning for LLMs
Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.
See all 253 papers