A learned representation is useful beyond its source task only if later tasks can make use of what the encoder has already learned. Source-task performance, however, gives no guarantee that the representation itself will remain useful once the original head is removed. We study representation reuse with respect to a specified family of future tasks. For such a family, we define the task-relevant state as the coarsest deterministic state sufficient for all of its tasks. A larger task family can require a finer state, and losing a required distinction makes at least one future task unrecoverable on a target support containing that conflict. Exact reuse also depends on how the retained information is presented to the downstream rule. We separate this into structural accessibility, which concerns whether the required distinctions remain recoverable, and coordinate consistency, which concerns whether those distinctions can be interpreted by the same downstream rule across samples. Both failure modes appear separately in the experiments. Reuse improves as task-relevant distinctions become more accessible. Reversible sample-dependent coordinate changes can sharply reduce reuse even when the representation remains recoverable, and inverting the applicable coordinate change restores performance. The same accessibility-reuse ordering appears in prospectively specified CIFAR-100 experiments with both ResNet-18 and DenseNet-121. Finally, preserving a candidate state that contains the distinctions needed by the anticipated task family improves reuse on compatible held-out tasks, with the clearest benefit when target supervision is limited. Representation reuse is therefore relative to the tasks that follow source learning: a useful representation must retain the distinctions those tasks need and make them available in a form that a shared downstream rule can use.
Figures & tables
Figure 1: Representation reuse. (1) Matched source performance need not imply matched reuse. (2) Future-task distinctions must remain structurally accessible. (3) Retained distinctions must admit a shared interpretation. (4) Structure-preserving source learning promotes recoverability of an anticipated candidate state.
Figure 2: Source-task performance does not determine reuse. (a) Paired reuse differences under exact source matching. (b) Mean reuse differences as target supervision increases; the dashed line denotes zero.
Figure 3: Accessibility and coordinate tests. (a) State recoverability and low-shot reuse for C0, C1, C4, Random auxiliary (C2), and Shuffled-state (C3). (b) Reuse under reversible coordinate interventions; increasingly heterogeneous sample-dependent conventions reduce blind reuse, while convention-aware inversion returns reuse to the Identity condition. Error bars show one standard deviation.
Figure 4: Structure preservation improves reuse primarily under limited target supervision. (a) Reuse across the predefined target-supervision conditions for C0, C1, and C4. (b) Paired seed-level gains over C0: C1 improves low-shot reuse by 0.1715 , whereas the full-support gain is 0.0241 and its confidence interval includes zero. C4 shows the same supervision-localized pattern. Panel (a) is descriptive; panel (b) reports the paired estimand with bootstrap 95% confidence intervals.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Name
Definition
C0
Source-only
Optimizes the source-task objective without structural supervision.
C1
SPR
Adds the structure-preserving objective in Eq. ( 21 ) using candidate-state labels.
C2
Random auxiliary
Uses random three-state auxiliary targets in place of the candidate state.
C3
Shuffled-state
Permutes candidate-state targets across observations, breaking their observation–state correspondence.
C4
Globally relabeled SPR
Uses the same structural supervision as C1 after one fixed, globally shared bijective relabeling of candidate-state labels.
Appendix
Table 1: Source-learning conditions. C2 and C3 are structural controls.
Held-out family
ANY_MATCH
ANY_MISMATCH
ANY_UNKNOWN
HOMOGENEOUS
MM
Included
Included
Included
Excluded
MD
Included
Included
Included
Included
MU
Included
Included
Included
Included
DD
Included
Included
Included
Excluded
DU
Included
Included
Included
Included
UU
Included
Included
Included
Excluded
Appendix
Table 2: Leave-one-composition-out oracle disposition of the 24 candidate family–rule cells. “Included” means that the fixed downstream readout extrapolates the replacement rule correctly from the five non-held-out oracle composition prototypes to the held-out prototype.
Level
Hcal(F∣C,Aq)
aq
Oracle fine-state Bayes accuracy
Q1
2.3219281
0.0000000
0.2
Q2
1.3509775
0.4181657
0.4
Q3
0.8000000
0.6554588
0.6
Q4
0.4000000
0.8277294
0.8
Q5
0.0000000
1.0000000
1.0
Appendix
Table 3: Conditional-accessibility calibration. The quantities are determined by the predefined auxiliary-target construction, not by learned representation performance.
Figure 5: Paired audit of source-matched reuse separation. All C0, C1, and C4 representations achieve identical source-task validation accuracy (1.0000 across all 15 trained models). (a) Paired reuse differences relative to the corresponding C0 representation for each independently trained source seed. Under limited target supervision, C1 and C4 show the same direction of separation from C0 across all five seeds, whereas the differences contract substantially under full target support. (b) Descriptive distributions of cell-level low-shot paired effects across the fixed 1004-cell admissible novel-rule registry. Thin curves show individual representation seeds and thick curves show pooled descriptive empirical cumulative distribution functions. Registry cells expose heterogeneity across replacement tasks but are not treated as independent statistical replicates; inference is performed at the paired representation-seed level.
Figure 6: Structural accessibility and low-shot reuse under matched controls. (a) Condition-level relationship between held-out candidate-state recoverability and low-shot frozen reuse across Source-only (C0), SPR (C1), Random auxiliary (C2), Shuffled-state (C3), and Globally relabeled SPR (C4). Markers show condition means and error bars show standard deviations across five independently trained source seeds. Improving candidate-state recoverability is accompanied by improved low-shot reuse; auxiliary training without task-aligned state structure (C2) or with broken observation–state correspondence (C3) does not reproduce the effect, whereas a globally shared relabeling of the preserved state space (C4) remains close to C1. (b) Paired C1–C0 gains for each source seed in held-out candidate-state recoverability and low-shot reuse. All five paired seeds lie in the upper-right quadrant; both gains are positive for every source seed.
Figure 7: Coordinate inconsistency degrades shared decoding and is exactly rescued by convention-aware inversion. (a) Frozen downstream reuse under increasingly heterogeneous reversible coordinate conventions. A single globally shared reparameterization leaves blind reuse essentially unchanged relative to the identity convention, whereas reuse decreases progressively as the number of unresolved conventions increases, reaching its minimum under sample-dependent conventions. Supplying the applicable inverse convention restores the identity-level reuse accuracy for every intervention condition; the frozen reconstruction audit reports zero residual relative to the common-coordinate representation. (b) Blind-decoder accuracy penalty relative to the identity convention for each independently trained source representation. For every source seed, the penalty increases from the shared-global control through the 2-, 4-, and 8-convention interventions to the sample-dependent condition.
Condition
ΔLS
Δfull
Γ
SPR
+0.1715
+0.0241
+0.1474
95% CI
[0.0986,0.2591]
[−0.0260,0.0992]
[0.1044,0.1905]
Globally relabeled SPR
+0.1771
+0.0300
+0.1471
95% CI
[0.1172,0.2593]
[−0.0201,0.0990]
[0.1193,0.1749]
Appendix
Table 4: Paired held-out-family reuse effects relative to Source-only. The inferential unit is the matched representation-training seed.
Figure 8: Held-out-family reuse gains are concentrated under limited target supervision. (a) Paired reuse effects relative to Source-only across target-supervision levels K∈{1,2,4,8,16} and the full-support condition. Solid circles denote SPR (C1) and open triangles denote Globally relabeled SPR (C4); thick curves show paired means across the five representation-training seeds, error bars show bootstrap 95% confidence intervals, and faint curves show the corresponding seed-level paired effects. For both conditions, paired reuse gains are larger at each predefined limited-supervision level than under full target support. (b) Descriptive low-shot and full-support paired effects for the 21 admissible held-out family–rule evaluations. Points above the diagonal have larger low-shot than full-support effects. The cell-level distribution shows heterogeneity across replacement tasks and is descriptive; inferential comparisons remain defined at the paired representation-training-seed level.
Encoder
Q1
Q2
Q3
Q4
Q5
Q5–Q1
ResNet-18
0.2670415
0.2916385
0.3000700
0.3075740
0.3212570
0.0542155
DenseNet-121
0.2810590
0.3001825
0.3080730
0.3216465
0.3260565
0.0449975
Appendix
Table 5: Condition-level mean held-out reuse in the conditional-accessibility replications. Q5–Q1 is computed from the unrounded condition means.
Figure 9: Source-task matching across Q-specific natural-image replications. (a) Mean source-task accuracy across the Q1–Q5 source-learning interventions for the ResNet-18 replication. Each Q level corresponds to a separately trained representation, and all 25 architecture–seed–Q source-training runs satisfy the pre-specified source-matching criterion. The dashed horizontal line denotes the mean source-task accuracy across the five Q conditions. (b) The corresponding source-task matching audit for the DenseNet-121 replication. All 25 Q-specific DenseNet-121 runs likewise satisfy the same pre-specified matching criterion. Together with Figure 10 , this audit separates the observed intervention–reuse relationship from uncontrolled failure of the source-task matching protocol; it does not require source-task accuracy to be numerically identical across Q conditions.
Figure 10: Conditional-accessibility intervention–reuse relationship across two encoder architectures. (a) Held-out frozen reuse across the predefined Q1–Q5 source-learning interventions for ResNet-18 and DenseNet-121. Solid circles denote ResNet-18 and dashed open triangles denote DenseNet-121. For each architecture, the legend reports the corresponding endpoint contrast Q5−Q1 . Condition means increase from Q1 to Q5 for both architectures. Q1–Q5 denote separately trained intervention-specific representations and should not be interpreted as progressively revealing additional side information to one fixed encoder. (b) Seed-level intervention–reuse slopes for the two architecture replications. Individual markers denote matched seed blocks, diamonds denote mean slopes, and vertical bars denote exhaustive seed-block bootstrap 95% intervals. All seed-level slopes are positive for both architectures. All 25 Q-specific source-training runs for each architecture satisfy the pre-specified source-matching criterion.
Which pretraining objective yields the best representation for downstream tasks under an information budget? We formalize this problem as a multitask information bottleneck, using a mutual information constraint to model incomplete information retention. For Gaussian inputs and tasks, we characterize optimal representations and show that a weakly task-aligned specialist can outperform a reconstruction generalist at low rates but lose at high rates. We also construct task families for which every fixed Gaussian prediction objective incurs arbitrarily large relative regret, driven by reversals in optimal allocation priorities. Conversely, the average task information matrix defines an objective with a uniform relative-regret bound controlled by maximum task SNR. CelebA and dSprites experiments illustrate that compression can change which pretraining objective performs best, and even task-aware training can reverse which downstream task is better preserved as rate increases.
Given a generalist model, learning a task-relevant specialist representation is fundamental for downstream applications. Identifiability, the asymptotic guarantee of recovering the ground-truth representation, is critical because it sets the ultimate limit of any model, even with infinite data and computation. We study this problem in a completely nonparametric setting, without relying on interventions, parametric forms, or structural constraints. We first prove that the structure between time steps and tasks is identifiable in a fully unsupervised manner, even when sequences lack strict temporal dependence and may exhibit disconnections, and task assignments can follow arbitrarily complex and interleaving structures. We then prove that, within each time step, the task-relevant latent representation can be disentangled from the irrelevant part under a simple sparsity regularization, without any additional information or parametric constraints. Together, these results establish a hierarchical foundation: task structure is identifiable across time steps, and task-relevant latent representations are identifiable within each step. To our knowledge, each result provides a first general nonparametric identifiability guarantee, and together they mark a step toward provably moving from generalist to specialist models.
Diffusion models are often trained in low-dimensional latent spaces, which are then reused for related but shifted datasets. In this work, we study when such latent reuse remains reliable under distribution shift. We consider a source-target setting in which both datasets are approximately low-dimensional but may lie near different subspaces. We show that freezing and reusing a source latent space induces a target-domain score error governed by two quantities: the principal-angle misalignment between the source and target subspaces, and the target ambient noise amplified by the diffusion time scale. Motivated by these limits, we further study mixed source-target training and characterize how the required shared latent dimension depends on the relative geometry of the two distributions. Our results provide theoretical guidance on when latent reuse is reliable and when learning a shared representation may be necessary.
Yifeng Yu, Lu Yu
Department of Mathematical Sciences, Tsinghua University · Department of Data Science, City University of Hong Kong