cs.CVOct 23, 2026
SaveSPOOK: Sparse Pixel Optimization for Off-Manifold Knowledge in Vision Encoders
Organizations: Department of Computer Science & Engineering, Michigan State University · Coco Robotics · Program in Evolution, Ecology, & Behavior, Michigan State University · Department of Physics & Astronomy, Michigan State University ·
Abstract
Modern vision encoders are widely used as semantic representations, but the relationship between embedding similarity and human-visible semantics can become unclear far from the natural-image manifold. We probe this regime in a black-box setting by applying sparse pixel edits to blank canvases with a simple pixel-level hill-climbing procedure. Across ImageNet and Flickr30k, we evaluate whether these edited blank canvases can retrieve chosen targets in multimodal contrastive encoders and self-supervised visual encoders. On OpenCLIP, inputs optimized toward class prompts achieve strong top-1 and top-5 targeted retrieval performance; on DINOv3, inputs optimized toward class prototypes achieve high target alignment and near-top prototype ranks. These results show that sparse edits on otherwise blank, visually non-semantic inputs can enter target semantic basins in both language-aligned and self-supervised visual representation spaces. Our findings identify a shared off-manifold failure mode of modern vision encoders and suggest sparse blank-canvas optimization as a simple probe of representation geometry.
Figures & tables
Figure 1: Sparse blank-canvas images can outrank natural examples under vision-encoder similarity. For the representative target “Irish water spaniel,” the top row shows natural ImageNet examples and the bottom row shows the white canvas and SPOOK images optimized for OpenCLIP and DINOv3. The optimized images remain visually non-semantic yet score above the natural examples under the corresponding encoder similarities. This case reflects the broader targeted-retrieval effect observed across targets; for more qualitative examples see Figures 8 – 10 .
| CLIP | DINOv3 | |||||
| Target | ImageNet | Flickr30k | ImageNet | |||
| Single caption | Caption prototype | Image prototype | ||||
| Metric | Mean | Median | Mean | Median | Mean | Median |
| Blank cosine | 0.1763 | 0.1762 | 0.1771 | 0.1766 | -0.0355 | -0.0338 |
| SPOOK cosine | 0.3314 | 0.3349 | 0.3510 | 0.3549 | 0.5365 | 0.6370 |
| Clean cosine | 0.2668 | 0.2723 | 0.3396 | 0.3413 | 0.5795 | 0.6099 |
Table 1: SPOOK images achieve targeted retrieval across CLIP and DINOv3. All results use white-canvas initialization, a retrieval bank of targets, and ViT-B/16 encoders. Bold cosine entries mark the highest target cosine among blank, SPOOK, and clean images for each setting.
Figure 2: SPOOK optimization dynamics. Across CLIP/ImageNet, CLIP/Flickr30k, and DINOv3/ImageNet, mean target cosine increases with query budget, while top- and top- targeted retrieval success rise quickly and saturate. Dashed lines in the top row mark mean reference scores for the same targets: the unedited white canvas ( Blank ) and the corresponding clean reference image or prototype ( Clean ). In the CLIP settings, SPOOK often exceeds the clean-reference score despite remaining visually non-semantic. The bottom row shows that these gains remain sparse: after candidate queries per target, SPOOK modifies less than of pixel-channels on average.
Figure 3: Interpolation score profiles. Target cosine similarity is measured along three image-space paths: white canvas to SPOOK image, SPOOK image to real target image, and white canvas to real target image. Columns show the three main settings: CLIP/ImageNet caption targets, CLIP/Flickr30k caption targets, and DINOv3/ImageNet prototype targets. SPOOK images achieve high target similarity through a sparse off-manifold route, while mixing SPOOK with the real image often causes a sharp score drop before the score recovers near the natural-image endpoint. This non-monotonic behavior suggests that sparse SPOOK images and real images can reach target alignment through distinct pixel-space routes.
| ImageNet | Flickr30k | ||||
| Backbone | Metric | Mean | Median | Mean | Median |
| ViT-B/16 | Blank cosine | 0.1763 | 0.1762 | 0.1771 | 0.1766 |
| SPOOK cosine | 0.3314 | 0.3349 | 0.3510 | 0.3549 | |
| SPOOK rank | 2.1 | 1.0 | 2.3 | 1.0 | |
| Pixel edits | 0.29% | 0.29% | 0.28% | 0.28% | |
| ASR@1 / ASR@5 | 94.1% / 98.5% | 87.6% / 95.8% | |||
Table 2: CLIP backbone ablation. We compare CLIP ViT-B/16 and ViT-B/32 using the same white-canvas initialization, caption objective, retrieval bank size, and query budget as Table 1 . These open-source checkpoints are included to test whether sparse spooking depends on a single patch size or checkpoint.
| DINOv3 ViT-S/16 | DINOv3 ViT-B/16 | DINOv3 ViT-L/16 | ||||
| Metric | Mean | Median | Mean | Median | Mean | Median |
| Blank cosine | -0.0337 | -0.0338 | -0.0355 | -0.0338 | -0.0142 | -0.0144 |
| SPOOK cosine | 0.6701 | 0.7214 | 0.5365 | 0.6370 | 0.5393 | 0.6890 |
| SPOOK rank | 2.0 | 1.0 | 8.5 | 1.0 | 23.6 | 1.0 |
| Pixel edits | 0.62% | 0.62% | 0.61% | 0.63% | 0.49% | 0.50% |
| ASR@1 / ASR@5 | 96.5% / 99.6% | 85.8% / 94.9% | 72.5% / 88.3% | |||
Table 3: DINOv3 scale ablation. We compare open-source DINOv3 ViT-S/16, ViT-B/16, and ViT-L/16 checkpoints using the same white-canvas initialization, image-prototype objective, retrieval bank size, and query budget as Table 1 . The effect persists across model scales, although success rates and cosine magnitudes vary.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Model | Total runtime | Runtime / query / target |
| ImageNet | OpenCLIP B/16 | 2.36 h | 0.1699 ms |
| Flickr30k | OpenCLIP B/16 | 2.29 h | 0.1649 ms |
| ImageNet | DINOv3 B/16 | 3.16 h | 0.2275 ms |
Table 4: Runtime for the three main SPOOK settings. Each run optimizes targets with a budget of candidate pixel-channel queries per target on a single NVIDIA H200 GPU.
Figure 4: Image-space interpolation from blank canvas to SPOOK image to clean image. Each curve follows a path from the blank canvas to the optimized SPOOK image and then to the corresponding clean image. The red dashed line marks the SPOOK image and the thin blue lines represent each one of the 1000 targets. Target similarity rises as sparse edits are added to the blank canvas, but drops sharply when the SPOOK image is mixed with the clean image, before recovering near the clean endpoint. This non-monotonic behavior indicates that sparse SPOOK images and clean images reach target alignment through distinct pixel-space routes.
Figure 5: Representative interpolation grid for CLIP/Flickr30k. The target is a Flickr30k caption, and scores are CLIP image–text cosine similarities. The first two rows interpolate from the white canvas to the optimized SPOOK image; the last two rows interpolate from the SPOOK image to the real target image. The SPOOK endpoint obtains a high target score while remaining visually sparse, but mixing it with the real image produces a lower-scoring intermediate state before the natural image route recovers target similarity.
Figure 6: Representative interpolation grid for CLIP/ImageNet. The target is the prompt “a photo of a Irish water spaniel,” and scores are CLIP image–text cosine similarities. Sparse edits from the white canvas produce a high-scoring SPOOK image without recognizable target content. When the SPOOK image is interpolated toward the natural ImageNet image, the target score drops sharply before recovering near the real-image endpoint, illustrating a distinct sparse route to target alignment.
Figure 7: Representative interpolation grid for DINOv3/ImageNet. The target is the ImageNet class “Irish water spaniel,” and scores are DINOv3 image–prototype cosine similarities. As in the CLIP setting, sparse edits on the white canvas produce a high-scoring SPOOK image despite the absence of recognizable target content. Interpolating from SPOOK to the natural image produces a non-monotonic score path, indicating that the sparse and natural inputs reach the class prototype through different pixel-space routes.
Figure 8: Qualitative CLIP/ImageNet examples. Each row shows a natural ImageNet image, the white canvas initialization, and the optimized SPOOK image for the same target caption. Scores are CLIP image–text cosine similarities to the row caption.
Figure 9: Qualitative CLIP/Flickr30k examples. Each row shows the real Flickr30k image, the white canvas initialization, and the optimized SPOOK image for the corresponding caption target. Scores are CLIP image–text cosine similarities to the row caption.
Figure 10: Qualitative DINOv3/ImageNet examples. Each row shows a natural ImageNet image, the white canvas initialization, and the optimized SPOOK image for the same ImageNet class target. Scores are DINOv3 image–prototype cosine similarities to the row ImageNet class.
Figure 11: Supplementary per-target diagnostics for SPOOK optimization. Columns correspond to the three main evaluation settings: CLIP/ImageNet, CLIP/Flickr30k, and DINOv3/ImageNet. Rows show, from top to bottom, target cosine over optimization, target retrieval rank over optimization, cumulative pixel-edit percentage over optimization, and the final distribution of edit counts across targets. These diagnostics complement Table 1 and Figure 2 by showing that SPOOK’s success is broad across the target bank: target scores increase consistently, ranks rapidly collapse toward the top of the retrieval bank, and the final images remain highly sparse.
| Dataset | Model | Method | SPOOK cosine | ASR@1 | ASR@5 | Rank |
| ImageNet | CLIP | SPOOK greedy | 0.3314 / 0.3349 | 94.1% | 98.5% | 2.1 / 1.0 |
| ImageNet | CLIP | Random sparse | 0.1384 / 0.1385 | 0.1% | 0.5% | 499.0 / 504.0 |
| Flickr30k | CLIP | SPOOK greedy | 0.3510 / 0.3549 | 87.6% | 95.8% | 2.3 / 1.0 |
| Flickr30k | CLIP | Random sparse | 0.1008 / 0.0999 | 0.1% | 0.5% | 502.5 / 496.0 |
| ImageNet | DINOv3 | SPOOK greedy | 0.5365 / 0.6370 | 85.8% | 94.9% | 8.5 / 1.0 |
| ImageNet | DINOv3 | Random sparse | -0.0196 / -0.0177 | 0.1% | 0.4% | 513.9 / 509.0 |
Table 5: Random sparse images do not explain SPOOK retrieval. For each target, the random sparse control matches SPOOK’s edit count but chooses pixel-channel locations and values randomly. All results use targets and ViT-B/16 encoders.
| CLIP ImageNet | CLIP Flickr30k | DINOv3 ImageNet | |||||
| Init | Metric | Mean | Median | Mean | Median | Mean | Median |
| White | Blank cosine | 0.1763 | 0.1762 | 0.1771 | 0.1766 | -0.0355 | -0.0338 |
| SPOOK cosine | 0.3314 | 0.3349 | 0.3510 | 0.3549 | 0.5365 | 0.6370 | |
| SPOOK rank | 2.1 | 1.0 | 2.3 | 1.0 | 8.5 | 1.0 | |
| Pixel edits | 0.29% | 0.29% | 0.28% | 0.28% | 0.61% | 0.63% | |
| ASR@1 / ASR@5 | 94.1% / 98.5% | 87.6% / 95.8% | 85.8% / 94.9% | ||||
Table 6: Initialization strongly controls sparse spooking. All results use ViT-B/16 encoders, a retrieval bank of targets, and the same target/objective settings as Table 1 . Only the initial canvas is changed.
| ImageNet | Flickr30k | ||||
| Objective | Metric | Mean | Median | Mean | Median |
| Caption | Blank cosine | 0.1763 | 0.1762 | 0.1771 | 0.1766 |
| SPOOK cosine | 0.3314 | 0.3349 | 0.3510 | 0.3549 | |
| SPOOK rank | 2.1 | 1.0 | 2.3 | 1.0 | |
| Pixel edits | 0.29% | 0.29% | 0.28% | 0.28% | |
| ASR@1 / ASR@5 | 94.1% / 98.5% | 87.6% / 95.8% | |||
Table 7: Objective-mode ablation for CLIP. All rows use CLIP ViT-B/16, white-canvas initialization, a retrieval bank of targets, and the same query budget as Table 1 . Only the optimized objective is changed.
| Source model | Evaluation model | ASR@1 | ASR@5 | Median rank |
| CLIP ViT-B/16 | CLIP ViT-B/16 | 93.9% | 98.4% | 1.0 |
| CLIP ViT-B/16 | DINOv3 ViT-B/16 | 0.6% | 2.1% | 386.0 |
| DINOv3 ViT-B/16 | DINOv3 ViT-B/16 | 85.5% | 94.5% | 1.0 |
| DINOv3 ViT-B/16 | CLIP ViT-B/16 | 0.2% | 0.8% | 449.0 |
| CLIP ViT-B/16 | CLIP ViT-B/32 | 1.1% | 3.9% | 176.0 |
| CLIP ViT-B/32 | CLIP ViT-B/16 | 0.4% | 2.0% | 321.0 |
Table 8: Sparse SPOOKs show little cross-model transfer. Rows indicate the source model used to optimize the SPOOK image; columns indicate the evaluation model. All results use ImageNet targets, white initialization, and -target retrieval.
| Condition | Transform | Score | Score | ASR@1 | ASR@1 | ASR@5 | ASR@5 | Med. Rank |
| CLIP / ImageNet | None | 0.3355 | 0.0000 | 93.8 | 0.0 | 98.4 | 0.0 | 1 |
| CLIP / ImageNet | JPEG quality 75 | 0.2296 | 0.1059 | 14.1 | 79.7 | 31.2 | 67.2 | 23 |
| CLIP / ImageNet | Center crop 90% | 0.2024 | 0.1331 | 4.7 | 89.1 | 14.1 | 84.3 | 119 |
| CLIP / ImageNet | Center crop 80% | 0.1922 | 0.1433 | 4.7 | 89.1 | 12.5 | 85.9 | 162 |
| CLIP / ImageNet | Gaussian blur | 0.2189 | 0.1166 | 18.8 | 75.0 | 34.4 | 64.0 | 21 |
| CLIP / ImageNet | Resize 112 then restore | 0.2149 | 0.1206 | 14.1 | 79.7 | 23.4 | 75.0 | 21 |
Table 9: Robustness smoke test for main-table SPOOK images under simple image transformations. Scores are target cosine similarities after transformation. ASR and rank are computed against the same 1000-target bank as the corresponding main-table setting. Each row uses a fixed random subset of 64 targets.