Support Operation Factorization: Compositional Readout of Frozen Vision Encoders under Controlled Interventions

Frozen vision foundation models are commonly evaluated through a single global image embedding, but this interface can conflate missing information with information lost at readout time. We study this distinction by keeping a pretrained vision encoder frozen and varying only the readout applied to its final patch tokens. We compare standard global readouts against a lightweight foveated readout, which attention-pools patch tokens using a learned or question-conditioned query, and against an oracle readout with access to the annotated target region. We evaluate these interfaces on three localized binding problems: a controlled synthetic color--shape binding task under clutter, a color-free crowded shape-detection variant, and a GQA-derived natural-image task where paired questions ask for the colors of different same-category objects in the same image. Global readouts perform near perfectly when the synthetic target appears alone, but collapse under clutter and counterfactual target edits, whereas the foveated readout recovers most of the oracle-accessible signal. On the GQA-derived task, question-independent global image vectors improve only modestly over question-only priors, while question-conditioned foveation substantially improves paired localized color accuracy. A counterfactual nuisance-to-signal ratio explains the synthetic failures: global pooling dilutes localized label-changing evidence while exposing the probe to nuisance variation from irrelevant objects. These results indicate that apparent spatial blindness in frozen vision models can arise from the global embedding interface rather than from an absence of spatial information in the frozen patch tokens.

A foundation encoder is pretrained once on a large image corpus and then reused with its weights frozen. Each new task is solved by training a small head on the features it produces. This setup is common in medical imaging, where labeled cases are scarce and a frozen encoder can be reused across findings. All downstream tasks then depend on the class separation present in that fixed feature space. Encoder selection usually uses the area under the receiver operating characteristic curve (AUROC) of a trained downstream head. AUROC measures the predictive information a head can extract from the features, but it does not measure class separation in the frozen feature space. A positive image that lies near a negative image in feature space has a bounded normalized margin under any Lipschitz head. Chance-calibrated neighborhood discordance (CANDOR) measures this feature-space separation without training a head. For a positive image, it compares the k nearest positive-label neighbors with the k nearest negative-label neighbors, among images acquired the same way. Discordance rate D is the share of positives whose opposite-label neighbors are nearer. Drawing the 2 candidate sets at equal size makes the labels interchangeable, so label-independent features have chance level D=1/2 without simulation. We apply CANDOR to 22 frozen encoders on 605,443 images from 20 public datasets, covering 8 binary tasks in 7 domains. On every task, the best encoder is below 1/2. A discordant image bounds the normalized margin of every Lipschitz head on that encoder. A selector that is shown the true label and chooses among 11 encoders reduces the miss rate from 0.359 to 0.028. Discordance is associated with occlusion retention and with none of pretraining objective, parameter count, release year, or finding size. The fixed chance level lets D be computed for a frozen encoder before any downstream head is trained.

This version reports a reproducibility audit of the frozen-encoder experiments presented in version 1. The numerical discrimination results are reproducible from the preserved artifacts, but their original attribution to interferometric pretraining is not supported. The released checkpoint contains a nested model state that loads without missing parameters, whereas loading the outer checkpoint dictionary leaves almost the entire EfficientNet-B0 feature stack uninitialized. Preserved embeddings labelled as interferometric have norms of order $10^{-12}$, matching freshly initialized EfficientNet-B0 networks and differing by more than twelve orders of magnitude from the preserved ImageNet embeddings. A second, separately preserved near-zero embedding set produces almost the same IMS 4th-test anomaly scores ($r=0.987$) and record-level discrimination (AUC $0.9812$ versus $0.9818$). We therefore withdraw the causal claim that IMS performance demonstrates a morphological prior transferred from gravitational-wave instrumentation. We reanalyse the controlled IMS splits at matched observed false-positive rates and add multivariate classical signal baselines. The near-zero representations retain strong tail separation, particularly in the 2nd and 4th IMS runs, but this is now interpreted as an exploratory architecture-and-initialization effect coupled to Mahalanobis scoring. A separate PRONOSTIA audit shows that the original large warning times were induced by a lifetime-fraction baseline; under fixed-time evaluation, a ten-feature classical baseline outperforms the preserved encoder scores. These results illustrate how checkpoint provenance, finite-sample calibration, architecture, and target-domain baselines can create an appearance of cross-domain transfer. They also define the controls required before assigning physical meaning to frozen-representation anomaly scores.