Can Circuit Alignment Predict OOD Generalization?

Ayan Banerjee; Abhra Chaudhuri; Josep Llados; Umapada Pal; Anjan Dutta

cs.LGSep 25, 2026

Can Circuit Alignment Predict OOD Generalization?

Authors: Ayan Banerjee, Abhra Chaudhuri, Josep Llados, Umapada Pal, Anjan Dutta

Organizations: Computer Vision Center, Universitat Autònoma de Barcelona · Fujitsu Research of Europe · Indian Statistical Institute, Kolkata · University of Surrey

Abstract

Can out-of-distribution (OOD) generalization be predicted from a trained model's weights alone, without any target-domain data? Existing representational similarity metrics (CKA, SVCCA, RSA) compare activations rather than forecast generalization. We show they are provably insensitive to structural rerouting in the computational graph, the very change distribution shift induces. We close this gap with the Circuit Alignment Score (CAS), which compares class-specific circuits across domains via graph kernels, decomposed into same-class coherence and cross-class confusion. Casting CAS as a Lebesgue integral over the domain distribution, we prove its Monte Carlo estimate recovers the ground-truth ranking of learners by OOD accuracy, with pairwise inversion error vanishing at rate $O(1/M)$ , where $M$ is the number of sampled domains. Across $48$ learners on PACS, CAS attains $0.88$ rank correlation with OOD accuracy, versus $0.58$ (CKA), $0.23$ (SVCCA), and $0.14$ (RSA), with similar trends on other benchmarks and even against data-dependent methods, making it the first provably consistent predictor of distributional robustness requiring neither target-domain data nor labels. The code is available at: https://github.com/ayanban011/ACE

Figures & tables

Figure 1: Circuit Evolution prediction under distribution shift with CAS: From weights alone, we extract class-specific circuits and compare them via graph kernels to score and rank learners based on OOD accuracy, without target data or labels.

Condition	CKA	SVCCA	RSA	CAS
P1: Structural sensitivity	✗	✗	✗	✓
P2: Class-conditional resolution	✗	✗	❍	✓
P3: Semantic consistency	✗	✗	✗	✓

Table 1: Necessary conditions for OOD-predictive metrics derived in this section. ✓ = satisfied by construction; ✗ = fails by impossibility theorem ( Appendix A ); ❍ = partially satisfied.

Figure 2: CAS vs OOD accuracy shows a monotonous trend (

\rho_{S}=0.93

), whereas CKA (

\rho_{S}=0.58

), SVCCA (

\rho_{S}=0.23

), and RSA (

\rho_{S}=0.14

) show a diffuse scatter, confirming that representational similarity fails to capture the circuit-level signal (Target class: cartoon of the PACS dataset).

Figure 3: Consistency of CAS rankings. Effect of the number of source domains

M

on ranking stability. Top: Pairwise inversion probability

P_{\mathrm{inv}}^{(M)}

vs.

M

(mean

\pm

std over 10 trials) with

O(1/M)

Chebyshev bound (dashed lines). Bottom: Spearman

\rho_{S}

between

M

-domain CAS ranking and true OOD accuracy. TK consistently achieves the highest

\rho_{S}

and all methods follow the

O(1/M)

trend.

Figure 4: Domain interpolation via weighted LoRA adapters: Images generated by SDXL with a fixed ArtPainting LoRA (weight 1.0) and a Photo LoRA whose weight increases from 0.0 to 2.0. At $w{=}[1.0,0.0]$ , the output is purely ArtPainting style; at $w{=}[1.0,2.0]$ , it is fully photorealistic. Semantic content (dog, pose, composition) is preserved throughout, while domain-specific appearance (brush strokes, texture, lighting) transitions smoothly, enabling controlled, continuous domain shift for circuit-level analysis.

Figure 5: Evolution of CAS under controlled domain interpolation via LoRA. CAS values are monotonously decreasing with the increasing domain perturbations.

Method	Tgt: Photo	Tgt: Art	Tgt: Cartoon	Tgt: Sketch	Mean
ATC Garg et al. (2022)	0.65	0.54	0.62	0.51	0.58
ProjNorm Yu et al. (2022)	0.72	0.61	0.68	0.55	0.64
ALine-D Baek et al. (2022)	0.79	0.70	0.78	0.61	0.72
CKA Kornblith et al. (2019)	0.81	0.48	0.58	0.45	0.58
SVCCA Raghu et al. (2017)	0.28	0.21	0.21	0.22	0.23
RSA Kriegeskorte et al. (2008)	0.12	0.14	0.19	0.11	0.14

Table 2: Per-target Spearman

\rho_{S}

on PACS. CAS consistently outperforms all baselines across all four target domains. The advantage is largest on Sketch (

+0.22

over next-best), the hardest target.

Method	Input modality	Needs target data?	Extra training?	$\rho_{S}$ $\uparrow$	MAE (%) $\downarrow$
ATC Garg et al. (2022)	Softmax outputs	Unlabeled	None	0.58	5.31
ProjNorm Yu et al. (2022)	Model weights	Unlabeled	$2\times$ retrain ‡	0.64	5.72
ALine-D Baek et al. (2022)	Predictions ( $\geq$ 3 models) †	Unlabeled	None	0.72	4.85
CKA (cross-domain) Kornblith et al. (2019)	Activations	Source only	None	0.58	7.18
SVCCA (cross-domain) Raghu et al. (2017)	Activations	Source only	None	0.23	8.04
RSA (cross-domain) Kriegeskorte et al. (2008)	Activations	Source only	None	0.14	7.63

Table 3: Comparison of OOD accuracy prediction methods on PACS. CAS achieves the highest

p_{S}

and lowest MAE across all four target-domain splits. † ALine-D requires the full 48-model pool; all other methods are single-model. ‡ ProjNorm requires two extra retraining runs per fold.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture	Adapter Sites	$d$	Insertion Logic	Attn Heads
VGG-19	features.{28,30,32}	512	After final three conv layers	—
ResNet-50	layer4.{0,1,2}	2048	After each block in layer4	—
MobileNetV2	features.{16,17,18}	1280	After last inverted-residual blocks	—
ViT-B/16	encoder.layer.{4,8,11}	768	After selected transformer blocks	✓
Deep MLP	layers.{-3,-2,-1}	$d_{h}$	After last three hidden layers	—

Appendix

Table 4: Adapter insertion points for each backbone. Layer ID refers to the module path in the PyTorch model.

d

is the adapter’s operating dimensionality. Attn heads indicates whether attention heads are additionally traced as circuit nodes.

Figure 6: Layerwise feature visualization with ViT-B/16 for the class "dog" for PACS dataset: the selected transformer encoder blocks capture the essence of class dog effectively.

Figure 7: Layerwise adaptive circuit extraction.

Architecture	MLP neurons $C$	Stage 1 (MLP)	Stage 1 (Attn)	Stage 2
VGG-19	512	$1{,}536$	—	$30$
ResNet-50	2048	$6{,}144$	—	$30$
MobileNetV2	1280	$3{,}840$	—	$30$
ViT-B/16	768	$2{,}304$	$36$	$60$

Appendix

Table 5: The number of forward passes for circuit extraction per class. For ViT, the attention column adds the cost of patching all

H{=}12

heads per layer.

Figure 8: Choice of kernels: Pairwise similarity matrices computed using the Treelet Kernel, Random Walk Kernel, and Optimal Transport methods for ArtPainting and Photo domain with MobileNetv2. Diagonal entries indicate self-similarity, while off-diagonal values capture cross-instance structural similarity. The Treelet Kernel shows strong self-similarity with lower inter-instance similarities, whereas Random Walk and Optimal Transport exhibit comparatively higher cross-instance similarity, reflecting different notions of structural correspondence.

Figure 9: Domain-wise feature distribution divergence on PACS: Pairwise domain overlap between Artpainting (A), Cartoon (C), Photo (P), and Sketch (S) is quantified using linear separability (↑ indicates less overlap), Bhattacharyya distance (↑ indicates greater distributional divergence), and kNN overlap (↑ indicates more overlap). Results show progressively increasing separability and distributional distance from A–P to P–S, with near-zero kNN overlap for P–S, indicating minimal shared neighbourhood structure and substantial cross-domain shift.

Domains	CKA ( $\uparrow$ )	SVCCA ( $\uparrow$ )	RSA ( $\uparrow$ )	CAS (TK) ( $\uparrow$ )
A - P	0.65	0.04	0.39	0.62
A - C	0.83	0.25	0.08	0.59
C - P	0.80	0.37	0.46	0.58
A - S	0.77	0.23	0.17	0.45
C - S	0.68	0.02	0.27	0.37
P - S	0.77	0.10	0.04	0.28

Appendix

Table 6: Pairwise domain shift distances on PACS. Higher values indicate more similar domains. Expected ordering based on visual similarity (see Fig. 9 ): P

\leftrightarrow

<

\leftrightarrow

<

\leftrightarrow

<

\leftrightarrow

<

\leftrightarrow

Panel of Figure 10 — Figure 10: CAS vs OOD accuracy for PACS dataset: For all the domains, Art Painting ( $\rho_{S}=0.91$ ), Photo ( $\rho_{S}=0.93$ ), and Sketch ( $\rho_{S}=0.88$ ), CAS maintains a monotonous trend with OOD accuracy. In contrast, representation similarity metrics always produce a diffuse scatter plot due to the unavailability of structural information.

Panel of Figure 11 — Figure 11: CAS vs OOD accuracy for OfficeHome dataset: CAS (TK) exhibits a strong near-monotone relationship with OOD accuracy across all four OfficeHome domains (Art, Clipart, Product, Real World), with ERM consistently in the low-CAS low-accuracy region and IRM in the high-CAS high-accuracy region, while CKA, SVCCA, and RSA show weak, noisy associations with no consistent method ordering.

Panel of Figure 12 — Figure 12: CAS vs OOD accuracy for DomainNet dataset: Across all six DomainNet domains, we observe a similar trend, confirming the superiority of CAS over the rest.

Table 7: Signal-noise analysis and correction on Office-Home. Despite a higher extraction noise floor than PACS, cross-domain signal remains dominant, noise-corrected CAS improves OOD ranking, and the same stability trends across objectives and architectures persist.

Table 10: Signal-noise analysis and noise correction ablations on PACS. Cross-domain circuit differences substantially exceed extraction noise, and circuit stability varies systematically across training objectives and architectures.

Table 13: Signal-noise analysis and correction on DomainNet. Although extraction noise increases substantially for the 345-class, six-domain setting, cross-domain circuit signal remains dominant, and noise correction yields the largest correlation gain among all benchmarks.

Table 16: Ablations on Office-Home. The optimal settings shift to

K{=}60

due to the larger label space, while the preferred bottleneck ratio (

r{=}4

) and edge threshold (

\epsilon{=}10^{-4}

) remain consistent with PACS.

Table 20: Ablations on DomainNet. The optimal TopK increases to

K{=}164

for the 345-class setting, while the preferred bottleneck ratio (

r{=}4

) and edge threshold (

\epsilon{=}10^{-4}

) remain consistent across benchmarks.

Table 24: Ablation studies for TopK neurons, bottleneck ratio, and edge threshold. CAS and OOD performance improve as invariant circuit structure emerges, with optimal correlation at

K=30

r=4

, and

\epsilon=10^{-4}

. (

\overline{\text{CAS}}

: Average of CAS across all pair of domains)

Figure 13: Circuits are K-Minimal: Up to TopK neurons, the performance of the circuits grows exponentially. Then, if we add more neurons, the performance grows very slowly until it matches the original network performance.

Explore similar work

Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models
Oct 7, 2026Sangyoon Bae, Sk Miraj Ahmed, Shinjae Yoo +1OOD Generalization Neural Network Robustness
Interdisciplinary Program in Artificial Intelligence Seoul National University Seoul, 08826, South Korea · Computational Science Initiative Brookhaven National Laboratory Upton, New York, 11973, USA · Department of Psychology Seoul National University Seoul, 08826, South Korea
STAMP: Predicting Out-of-Distribution Generalization without Target Data
Sep 26, 2026Md Kawsher Mahbub, Milon BiswasOOD Generalization
Department of Computer Science and Engineering NPI University of Bangladesh Manikganj, Bangladesh · Department of Computer & Information Sciences Towson University Towson, MD 21252
A Probabilistic Circuit-Induced Pseudo-Metric for Out-of-Distribution Detection
Aug 10, 2026Bhumika K, Vidhya S, Narayanan C KrishnanProbabilistic Circuits OOD Detection
Mehta Family School of Data Science and Artificial Intelligence Indian Institute of Technology Palakkad Palakkad, Kerala, India

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models

Sangyoon Bae, Sk Miraj Ahmed, Shinjae Yoo, Jiook Cha

Interdisciplinary Program in Artificial Intelligence Seoul National University Seoul, 08826, South Korea · Computational Science Initiative Brookhaven National Laboratory Upton, New York, 11973, USA · Department of Psychology Seoul National University Seoul, 08826, South Korea

OOD Generalization Neural Network Robustness

Open paper AskPDF arXiv

Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available? Existing diagnostics require source or target data, which rules them out before a target domain exists. Those that use the weights alone apply one statistic to every architecture, and do not separate robust models from fragile ones. We show the answer is encoded in the spectral structure of pretrained weights. Two forces shape that structure. Architecture determines how information is stored in weight matrices. Pretraining strategy determines what is rewarded. Together they set a spectral geometry that governs OOD robustness. We prove that the OOD accuracy gap is bounded by how tightly the source representations concentrate. A statistic computed from the pretrained weights alone serves as a proxy for that concentration. The direction of that proxy reverses between architecture families. We operationalize it: the direction is stable within one (architecture X strategy) combination, the finest grouping we test, which we call a cell. Pooled over 116 models spanning 7 modalities, a single statistic ranks OOD robustness weakly, because cells of opposite direction cancel. Within a cell, the statistic selected for it orders 92% of model pairs by OOD robustness in-sample. The selection does not leak the target: for each model family outside the matrix we logged the cell, metric and sign before running its OOD evaluation, and the predicted direction held in every case: EEG, genomic and protein. Acting on spectral concentration narrows the OOD gap by 24% at 87.5% ID retention. The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time, before data or compute is committed to a target domain.

Abstract

Explore similar work

Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models

STAMP: Predicting Out-of-Distribution Generalization without Target Data

A Probabilistic Circuit-Induced Pseudo-Metric for Out-of-Distribution Detection