cs.CLOct 22, 2026
SaveMostly Reading: What Open-Book Scores Measure in Document-Level LoRA Adapters
Organizations: Bosch Center for Artificial Intelligence, Germany · LMU Munich, Germany · Munich Center for Machine Learning, Germany
Abstract
Single-document LoRA adapters have been proposed as units of knowledge to be merged and retrieved; yet open-book scoring, with the document in the prompt, can credit a document's own adapter (its specialist) for reading that document as readily as for holding it. Training one adapter for each of 150 Wikipedia articles on Llama-3.1-8B-Instruct, we find that open-book scores mostly measure a reading behaviour every adapter shares: a random adapter trained on another document recovers 82 % of the specialist's open-book advantage over the base model but only 2 % of its advantage on a cloze probe that asks for the same document's facts without providing the source text, a share we cannot distinguish from zero. Content does reach the weights, since the specialists beat the base model in cloze recall by 0.094, and the geometry of their updates follows the documents they read; yet, in a test of 30 queries, adapters chosen by weight-space proximity or text embedding are not distinguishable from random ones on either probe, although that test cannot exclude an advantage of about four fifths of what a query's own adapter gains over a random one. At 16 documents or more, correctly scaled merges recover 87 % to 89 % of the specialists' open-book advantage but only 12 % to 16 % of their cloze advantage, with intervals that run from below zero to at most 32 %, against 94 % for joint training. We also report four silent errors, each of which produced a clean, wrong result.
Figures & tables
| Mean | |||
| Condition | OB | CB | cloze recall |
| Base | — | ||
| Natural | |||
| Style-normalized | |||
| Synthetic | |||
| No context | |||
Table 1: Token over all documents (OB and CB) and cloze recall on each document’s three factual items as a paired difference from the base model. All CIs exclude zero except for synthetic and swapped context on source; each OB condition reads the text listed in section C.1 , with swapped context on source reading its donor’s text (which holds none of its answers).
Figure 1: Share of the specialists’ advantage over the base model that each method recovers, OB on all four held-out items against cloze on the factual items, at . Each point is a ratio of means with its confidence interval from bootstrap resamples over documents; the dotted lines mark full recovery on each probe, and the dashed diagonal indicates equal recovery on both. The shares of no context and swapped context are divided by the natural adapter’s advantage over the base model on the documents each is scored on — all for no context, and its donor documents for swapped context; the no-context adapter’s cloze confidence interval, -276\text{,}\mathrm{%}-74\text{,}\mathrm{%} , runs below the axis.
| Adapter applied | OB | Cloze recall |
| None | ||
| Weight space | ||
| Text embedding | ||
| Random (bank) | ||
| Random (domain) | ||
| Query’s own |
Table 2: OB token on all four held-out items and cloze recall on the three factual items. One adapter is selected for each of query documents from a bank of that never holds the query’s own. The random rows average five draws, from the whole bank and from the query’s domain respectively.
| Reg. | Lerp | UM | Ties | Rand. | |
| Cross | |||||
| With. |
Table 3: OB retention at rank in the nine cells with . Each document’s under the method is divided by that of its own specialist, averaged over documents, and then over the cell’s sets; section E.2 gives the scaling error, the controls, and every paired confidence interval. ‘Reg.’ stands for regime, ‘UM’ for update merging, ‘rand.’ for random, and ‘with.’ for within; random averages five out-of-collection adapters per set.
Figure 2: Advantage over the random adapter in cloze retention (mean of ratios). Values are shown with confidence intervals, reported in the five cells with and pooled over all collections; section E.4 gives every cell.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Condition | Open-book | Closed-book | cloze recall |
| Base model | [ , ] | [ , ] | — |
| Natural | [ , ] | [ , ] | [ , ] |
| Style-normalized | [ , ] | [ , ] | [ , ] |
| Synthetic | [ , ] | [ , ] | [ , ] |
| No context | [ , ] | [ , ] | [ , ] |
| Swapped, on source | [ , ] | [ , ] | [ , ] |
Table 4: The training-text conditions with every CI: OB and CB token on all four held-out items and cloze recall on the three factual items as a paired difference from the base model (on the donor document for the donor row). Values are computed over all documents, with percentile bootstrap CIs over documents from resamples. OB and CB use the -bit base with greedy decoding, a -token window, no repetition penalty and the training template; cloze uses the -bit base with each adapter unmerged and the chat wrapper of section A.3 .
| (all layers) | (late layers) | NMI | -NN purity | Late layers | |||||
| Architecture | Late | Late | All | Kept | |||||
| Llama-3.1-8B | |||||||||
| Llama-3.2-3B | |||||||||
| Qwen3.5-2B | |||||||||
| Qwen3.5-9B | |||||||||
Table 5: Geometry before and after the truncation fix. Spearman correlations are given at both training windows; NMI and -NN purity are reported for the corrected adapters only, NMI over the selected late layers and purity both over those layers and over all layers; purity is computed as the fraction of an adapter’s six nearest neighbours sharing its domain, for which chance is ; the last two columns give the number of late layers selected after the fix and, of those selected before it, how many are selected again.
| Standard deviation | Draws | |||||
| Regime | Mean | Within | Between | Min | Max | |
| Within | ||||||
Table 6: Mean and spread of the random baseline’s OB retention. ‘Within’ is the mean standard deviation across the five draws made for a collection, ‘between’ is the standard deviation of the collection means, and the last two columns the most extreme single draws.
| Contrast | Factual | With true/false |
| Top 1 | ||
| Weight space random (bank) | [ , ] | [ , ] |
| Weight space text embedding | [ , ] | [ , ] |
| Text embedding random (bank) | [ , ] | [ , ] |
| Weight space random (domain) | [ , ] | [ , ] |
| Centred weight space random (bank) | [ , ] | [ , ] |
Table 7: The declared cloze contrasts of the retrieval run, paired over the query documents, with CIs from resamples; each query’s adapter is selected from a bank of that never holds the query’s own. ‘Top 1’ applies the selected adapter and ‘top 3’ merges the three highest-ranked with lerp ; ‘factual’ scores the three factual items and ‘with true/false’ adds the fourth; the random arms average five draws. The primary contrast is weight space against a random adapter from the bank at top 1 on the factual items, and its decision criterion was fixed before it was computed; the centred contrasts were added afterwards, outside that criterion.
| Regime | Scaling error | Lerp | Rank-8 update | Ties | Random | Joint | Full-rank update | Spectral share | |
| Cross | |||||||||
| Within |
Table 8: OB retention at matched rank in the nine cells with . Each document’s under the method is divided by that of its own specialist, every column by the same specialists, averaged over documents and then over the cell’s sets (five per cell, six at within-domain ). The scaling error is the factor average with both factors scaled by ; the random experiment averages five out-of-collection adapters per set, which at within-domain necessarily come from other domains; joint training is a retraining reference; full-rank update keeps the averaged update at rank , a ceiling rather than a merge; the spectral share is the fraction of the averaged update’s squared singular values in its top directions. Joint training at cross-domain averages four of the five sets, the fifth having no OB evaluation of its joint adapter.
| Regime | Lerp scaling error | Rank-8 update lerp | Rank-8 full-rank update | |
| Cross | 4 | [ , ] | [ , ] | [ , ] |
| 8 | [ , ] | [ , ] | [ , ] | |
| 16 | [ , ] | [ , ] | [ , ] | |
| 25 | [ , ] | [ , ] | [ , ] | |
| 32 | [ , ] | [ , ] | [ , ] | |
| Within | 4 | [ , ] | [ , ] | [ , ] |
Table 9: Paired OB contrasts that isolate scale, cross terms, and truncation. All values are shown with CIs, computed from resamples over sets. Lerp minus the scaling error isolates scale, rank- update minus lerp isolates cross terms at equal own-term weight, and rank- update minus full-rank update isolates the cost of truncation. Each contrast is computed from unrounded retentions, so it can differ in its last digit from the difference of the displayed values.
| Regime | Lerp random | Rank-8 update random | Ties rank-8 update | |
| Cross | 4 | [ , ] | [ , ] | [ , ] |
| 8 | [ , ] | [ , ] | [ , ] | |
| 16 | [ , ] | [ , ] | [ , ] | |
| 25 | [ , ] | [ , ] | [ , ] | |
| 32 | [ , ] | [ , ] | [ , ] | |
| Within | 4 | [ , ] | [ , ] | [ , ] |
Table 10: Paired OB contrasts of the correctly scaled merges with the random adapter and of Ties with the rank- update. Values are shown with CIs from resamples over sets. Each contrast is computed from unrounded retentions, so it can differ in its last digit from the difference of the displayed values.
| Joint lerp | Joint rank-8 update | Joint ties | ||
| Cross | 16 | [ , ] | [ , ] | [ , ] |
| 25 | [ , ] | [ , ] | [ , ] | |
| 32 | [ , ] | [ , ] | [ , ] | |
| Within | 16 | [ , ] | [ , ] | [ , ] |
| 25 | [ , ] | [ , ] | [ , ] | |
| Pooled | — | [ , ] | [ , ] | [ , ] |
Table 11: Paired contrasts in retention of joint training against each rank- merge at , OB. Values are computed at set level, with CIs from resamples. Joint training at cross-domain averages four of the five sets, the fifth having no OB evaluation of its joint adapter. Each contrast is computed from unrounded retentions, so it can differ in its last digit from the difference of the displayed values.
Figure 3: Open-book retention against collection size. CIs are omitted for legibility, so the paired contrasts of sections E.2 and E.3 , rather than the figure, carry the claims about significance; the marks parity with the per-document specialist each method replaces. Scaling error is the factor average at ; joint training at cross-domain averages four of the five sets, the fifth having no OB evaluation of its joint adapter.
| Random | Lerp | Rank-8 update | Ties | Joint | |||||||
| Regime | MR | AR | MR | AR | MR | AR | MR | AR | MR | AR | |
| Cross | |||||||||||
Table 12: Cloze retention relative to each document’s own specialist, on the -bit footing. ‘MR’ is the mean over documents of the ratio of recalls, without the seven documents whose specialist recalls none of the factual items; ‘AR’ is the ratio of mean recalls, with them; the pooled AR covers rows. The random, ties , and joint values come from an earlier run, whose footing is described in section E.4 .
| Truncated documents (%) | Spearman | Pearson | |||||||||||
| Definition | L | E | CS | H | M | B | All | ||||||
| First answer | ∗ | ∗∗ | |||||||||||
| Any string | ∗ | ∗∗ | |||||||||||
| Any answer | ∗∗ | ∗∗ | |||||||||||
| All answers | ∗ | ∗∗ | |||||||||||
| Mean share | ∗ | ∗∗ | |||||||||||
Table 14: Truncation at the -token default under five definitions, counted with the training tokenizer, and each definition’s correlation across the six domains with the change in the gap ( table 15 ). A document counts as truncated when: ( a ) the answer of its first training example lies wholly beyond the window; ( b ) any of its training strings exceeds the window; ( c ) at least one of its answers lies wholly beyond it; ( d ) all eight do, the definition of section 6 ; or ( e ) is the mean share of the eight answers that do. ‘L’ stands for law, ‘E’ for economics, ‘CS’ for computer science, ‘H’ for history, ‘M’ for medicine, and ‘B’ for biology. The -values are computed over the orderings; 0.05 , ${}^{**}p<$0.01 , 0.001$$ .
| Gap | |||
| Domain | at | at | Change |
| Law | [ , ] | [ , ] | [ , ] |
| Economics | [ , ] | [ , ] | [ , ] |
| C. science | [ , ] | [ , ] | [ , ] |
| History | [ , ] | [ , ] | [ , ] |
| Medicine | [ , ] | [ , ] | [ , ] |
Table 15: Open-book of the no-context adapters minus that of the natural adapters trained at each window. Rows are ordered by truncation rate, on one evaluation configuration ( -token window, no repetition penalty), with paired bootstrap CIs over documents. The change is the gap at minus the gap at , which equals, with its sign reversed, what the natural adapters lost to truncation; ‘c. science’ stands for computer science.
More in this topic
Low-Rank Adaptation
Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.
See all 542 papers