When Graph-JEPA Learns the Wrong Thing: Diagnosing and Repairing Category-Conditional Collapse

Gollam Rabby; Sören Auer

cs.LGAug 20, 2026

When Graph-JEPA Learns the Wrong Thing: Diagnosing and Repairing Category-Conditional Collapse

Authors: Gollam Rabby, Sören Auer

Organizations: L3S Research Centre, Leibniz University Hannover, Hannover, Germany · TIB Leibniz Information Centre for Science and Technology, Hannover, Germany

Abstract

Joint-embedding predictive architectures are selected almost universally by linear probing and effective rank. We report a case where both read healthily while the representation carries zero usable instance information. We repair it, and a second failure appears: the repaired metric saturates on a target carrying no structural information. Our corpus is a scientific-reasoning graph over 57,903 articles, each a subgraph. A Graph-JEPA predicts one masked aspect from a subgraph's remaining aspects, attaining linear-probe accuracy 0.871 and effective rank 18-47, yet retrieval recovers 0.00 of 14.4 bits (MRR 1.9e-4 vs chance 1.99e-4, p=0.98). Three upper bounds on the same pool and code recover nearly everything (+14.28, +14.34, +14.22 bits), ruling out corpus, masking, pool, and metric as causes. We trace this to variance allocation - frozen inputs place 86.05% of variance on subgraph identity and 0.40% on aspect identity, while trained latents place 0.39% and 99.61%. This is a property of the objective's optimum: the degenerate solution is a global minimum of the coupled predictor/EMA-target objective, present already at init. A repaired configuration reaches 14.377 of 14.379 bits, above the 13.865-bit oracle; reverting the loss to regression drops it to 0.307 bits, confirming it. Yet the repair licenses nothing about reasoning: the target is reducible, since intra-subgraph edges are a deterministic function of node census. The oracle reaches 96.4% of the ceiling, and our largest effect is the learning-rate schedule, not architecture. Bits and a reasoning probe show no relation across ten cells. A data-derived target fails a quality gate - 25.96% of nodes are duplicate placeholders, and the rest is more generic than supporting evidence. Rank, probes, and metrics can all saturate on an unsupportive evaluation. We release a harness with a reducibility audit and target gate.

Figures & tables

Figure 1: Neither standard health check predicts retrieval, and the null is exact. (left) Pooled effective rank spans

1.2

–

300

across all trained configurations and the ceilings; every trained point lies on the chance line and the ceilings lie

{\sim}14

bits above it. (centre) Probe accuracy spans

0.57

–

0.97

with no corresponding movement; the vertical spread is sampling noise around chance. (right) Cumulative distribution of the gold rank: baseline, centred baseline, best encoder and best fix are indistinguishable from the uniform-random curve (dotted) over the entire range, while the ceiling reaches

{\approx}0.6

at rank

1

. Not a shifted distribution, the same distribution.

Figure 2: Pipeline and diagnosis. (1) Graph construction over $57{,}903$ papers with typed reasoning nodes and edges. (2) Patch extraction; method and result are exact singletons. (3) Frozen sentence features feed a heterogeneous GNN trained with a JEPA objective; we sweep target geometry, aggregator, loss, target frame, structural cue, and regularisation. (4) Both standard health checks pass while retrieval recovers zero bits. Three ceilings show the signal is present, the variance allocation shows where it goes, and Sec. 9 shows that repairing it saturates a reducible target.

Figure 3: From the global reasoning graph to a paper-local target patch. (a) The heterogeneous graph; an extraction window selects one paper and its incident reasoning nodes. (b) The extracted subgraph, a paper node joined by typed edges to its claim, method and result nodes. Note that every edge shown is present for every paper possessing the endpoint node types, which is the content of Prop. 5 .

#	Intervention	$r_{\mathrm{node}}$	$r_{\mathrm{cand}}$	$r_{\mathrm{query}}$	$\mathcal{B}$	Diagnosis
0	mean pool, cosine target (baseline)	$18.3$	$18.1$	$1.9$	$0.00$	$p=0.98$ : chance
1	$+$ target VICReg	—	—	—	$0.00$	no effect
2	infonce , in-batch negatives	—	—	—	$0.00$	no effect
3	sum / deepsets / attn pooling	—	—	—	$0.00$	no effect
4	$+$ input whitening (instance var. $\times 25$ )	—	—	—	$0.00$	rank $\uparrow$ , $\mathcal{B}$ flat
5	retrieval frame: centre / rm-top $k$ / ZCA	—	—	—	$0.4$	$0.4\text{ bits}$ on $14.4\text{ bits}$

Table 1: The diagnostic ladder (

57{,}903

set, Protocol R, full pool).

\mathcal{B}

is bits recovered of

\mathcal{B}^{\max}=14.379

(Eq. 4 );

0

is exactly chance.

r_{\mathrm{cand}},r_{\mathrm{query}}

are the effective ranks of the candidate bank and of the model’s predictions . Rung 7 is the decisive control; rungs 8–10 are the ceilings of Sec. 6 ; rung 11 is the repaired configuration of Sec. 9 , included here so the whole trajectory is in one place. Per-rung detail: App. A13 .

$K$	$\mathcal{B}^{\max}$	bm25	bm25-no-ov	oracle	p1
$2$	$0.862$	$+0.860$	$+0.850$	$+0.840$	$+0.830$
$10$	$2.295$	$+2.284$	$+2.276$	$+2.260$	$+2.240$
$100$	$5.262$	$+5.240$	$+5.213$	$+5.198$	$+5.167$
$1000$	$8.531$	$+8.501$	$+8.445$	$+8.455$	$+8.421$
full	$14.385$	$+14.335$	$+14.250$	$+14.281$	$+14.220$
full pool, % of $\mathcal{B}^{\max}$		$\mathbf{99.7\%}$	$99.1\%$	$99.3\%$	$98.9\%$

Table 2: Bits recovered against pool difficulty (

58{,}145

set). Hard pools of size

K

are the

K

nearest within-category neighbours of the gold item, so difficulty increases downward; the pool has

N=K{+}1

items and

\mathcal{B}^{\max}=\frac{1}{N}\log_{2}N!

(Eq. 4 ), which lies

1.4427

bits below the index entropy

\log_{2}N

. Vector systems use

4{,}000

queries, lexical systems a nested

2{,}000

subsample. All four systems are within

1.2\%

of the ceiling at every rung; the baseline Graph-JEPA recovers

0.0\%

and the repaired configuration of Sec. 9 recovers

100.0\%

Variance attributable to	frozen inputs (block B)	frozen inputs (block A)	trained latents (block A)	inputs, de-templated (block B)
aspect identity ( $\rho_{\mathrm{asp}}$ )	$0.40\%$	-	$\mathbf{99.61\%}$	$0.0000$
paper identity ( $\rho_{\mathrm{pap}}$ )	$\mathbf{86.05\%}$	-	$0.39\%$	—
bits recovered on this representation	$+14.281$ ( oracle )	$13.865$ ( oracle )	$0.00$ (trained)	$+14.28$
% of the matching $\mathcal{B}^{\max}$	$99.3\%$	$96.4\%$	$0.0\%$	$99.3\%$

Table 3: The variance allocation. Share of total variance attributable to aspect identity and to paper identity. Columns 1 and 4 are block B (MPNet-

768

); columns 2 and 3 are block A (MiniLM-

384

), i.e. the same encoder that produced the trained latents, so the central comparison is within one feature space and the cross-encoder pair is retained only for continuity with the three ceilings. The two groupings are separate one-way decompositions and need not sum to

100\%

aspect share $\rho$	$0.0040$	$0.9$	$0.99$	$\mathbf{0.9961}$	$0.999$	$0.9999$
MRR	$1.000$	$0.893$	$0.107$	$\mathbf{0.0171}$	$0.0043$	$0.0014$
$\mathcal{B}$	$+12.845$	$+12.200$	$+3.833$	$\mathbf{+1.80}$	$+0.843$	$+0.266$
bits lost	$0.000$	$0.645$	$9.012$	$\mathbf{11.045}$	$12.002$	$12.579$
% of $\mathcal{B}^{\max}$ lost	$0.0\%$	$5.0\%$	$70.2\%$	$\mathbf{86\%}$	$93.4\%$	$97.9\%$

Table 4: Phase analysis at the measured operating point , at

\hat{\kappa}=0.942

and

\hat{\varepsilon}=1.515

, bank size

20{,}000

\mathcal{B}^{\max}=12.845

(Eq. 4 ). The trained representation’s measured share (

99.61\%

) predicts the loss of

11.045

12.845

bits; the inputs’ share (

0.40\%

) predicts no loss, which is what the three controls observe. Bits lost is

12.845-\mathcal{B}

and is monotone in

\rho

by construction. Critical share

\rho^{\star}=0.99999

(

1-\rho^{\star}=5.1\times 10^{-6}

); removing

\hat{\kappa}

\hat{\varepsilon}

\rho=0.9961

moves MRR by

<0.004

step	$\rho_{\mathrm{pap}}$	$\rho_{\mathrm{asp}}$	$r_{\mathrm{query}}$	$\mathcal{B}$
$20$ k (final)	$0.39\%$	$99.61\%$	$1.9$	$0.00$

Table 5: Variance allocation over training (block A, within one encoder,

\mathcal{B}

against

\mathcal{B}^{\max}=14.379

). One forward pass per checkpoint;

\rho_{\mathrm{pap}}

and

\rho_{\mathrm{asp}}

are the same one-way decompositions as Table 3 , and

r_{\mathrm{query}}

is the effective rank of the model’s predictions. Step

0

is the randomly initialised encoder before any gradient step.

Config	loss	steps	$\mathcal{B}$	A1	Reading
raw-skip gate	nce	6k	$14.379$	$0.875$	gate $\to 0.3886$ : encoder used
center :1	nce	3k	$14.379$	$0.963$
center :0, main	nce	20k	$14.377\pm 0.002$	$0.753\pm 0.010$	$3$ seeds
center :0, aspect cue	nce	6k	$14.376$	$0.800$
center :0, inductive	nce	6k	$14.376$	$0.769$	transduction gap $+0.001$
center :0	nce	3k	$\mathbf{14.359}$	$0.862$

Table 6: Eleven cells at matched schedule (block C; all share the graph cache, seed and evaluation;

\mathcal{B}

against

\mathcal{B}^{\max}=14.379

; A1 is a held-out reasoning probe with a pre-registered void line at

0.70

, defined in App. A16 ). The reg row is the decisive single-variable result: only the loss differs from the center:0 / nce /

3

k row. Bits span

11.808-14.379

and A1 spans

0.711-0.985

at Spearman

\rho=-0.24

(

n=10

, not significant), so the two metrics show no reliable relationship—this paper’s thesis restated on the repaired model. We claim no trade-off and no direction.

Gate / condition	Measured	Threshold	Verdict
G1 exact-duplicate share of challenged_by	$25.96\%$	$\leq 20\%$	FAIL
G1b genericness gap (challenge $-$ support)	$+0.2377$	$\leq 0.15$	FAIL
G0 census, challenged_by edges	$121{,}516$	$\geq 2{,}000$	pass
implies duplicate share	$0.00\%$	$\leq 20\%$	pass
paper-identity leak (claim- vs paper-grouped)	$+0.0003$	—	absent
shuffled-label control	$0.4975$	$\approx 0.500$	pass

Table 7: The data-derived target and its pre-registered gates (block C,

3

seeds, paper-grouped splits). Upper: the gates, with thresholds fixed before measurement. Lower: the probe conditions, which we report even though the gates failed, because two of them are independently informative. Every threshold appears in App. A10 .

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Claim	Value	Source
Trained retrieval is indistinguishable from chance	$0.00$ bits, $p=0.98$	Table 1 , rung 0
Chance level	$1.99\times 10^{-4}$ (MRR), $0$ bits	Sec. 3 , Eq. 3
Recoverable ceiling, block A / B	$14.379$ / $14.385$ bits	Eq. 4
Linear probe succeeds	$0.871$	Table 28
Candidate bank is healthy	$18.1$ , DC $37.6$	Table 1
Query bank is degenerate	$1.9$ , DC $162.2$	Table 1

Appendix

Table 8: Each quantitative claim in the abstract and introduction, its value, and the table or figure that establishes it. Every value appears exactly once as a macro in the source, so prose and tables cannot diverge.

Block	Papers	Encoder	Used for
A (trained pipeline)	$57{,}903$	MiniLM-L6, $384$ -d	Tables 1 , 3 (cols. 2–3), 5 , 23 , 19 , 25 , 28 , 29 , 30
B (diagnostic harness)	$58{,}145$	MPNet-base, $768$ -d	Tables 2 , 3 (cols. 1, 4), 4 , 27 , 31
C (repair $+$ target audit)	$57{,}903$	MiniLM-L6, $384$ -d	Tables 6 , 20 , 7 , 15 , 22

Appendix

Table 9: Which run, corpus subset and sentence encoder produced each block. We keep these separate rather than pooling them, because the blocks measure different objects and the distinction carries the central results of Secs. 7 and 9 .

Symbol	Code name	Meaning
$r_{\mathrm{node}}$ ( $/128$ )	node_rk	effective rank of node latents
$r_{\mathrm{pool}}$ ( $/128$ )	pool_rk	effective rank of pooled patch vectors
$r_{\mathrm{cand}}$ ( $/128$ )	cand_erank	effective rank of the candidate bank
$r_{\mathrm{query}}$ ( $/128$ )	query_eff_rank	effective rank of the model’s predictions
$\mathrm{DC}_{\mathrm{ratio}}$	dc_ratio	$\\|\mu\\|/\mathbb{E}\\|x-\mu\\|$
$\mathrm{DC}_{\mathrm{energy}}$	dc_energy	$\\|\mu\\|^{2}/(\\|\mu\\|^{2}+\mathbb{E}\\|x-\mu\\|^{2})$

Appendix

Table 10: All effective ranks use Eq. 5 on mean-centred matrices. The distinction between

r_{\mathrm{cand}}

and

r_{\mathrm{query}}

is central: the former is what standard practice measures, the latter is where the failure lives.

Node type	count	Edge type	count	census-determined?
paper	57,903	(paper, has_claim, claim)	251,938	yes ( $=\|$ claim $\|$ )
claim	251,938	(paper, has_method, method)	57,903	yes ( $=\|$ paper $\|$ )
method	57,903	(paper, has_result, result)	57,903	yes ( $=\|$ paper $\|$ )
result	57,903	(method, produces, result)	57,903	yes ( $=\|$ paper $\|$ )
evidence	373,438	(result, grounds, claim)	251,938	yes ( $=\|$ claim $\|$ )
implication	251,938	(claim, supported_by, evidence)	251,922	no (data-derived)

Appendix

Table 11: Heterogeneous graph statistics (

57{,}903

maskable papers). The right-hand column is the reducibility audit of Sec. 9 : a relation whose cardinality equals an endpoint type’s node count is a deterministic function of the census and carries zero information.

Dissociation across 20 cells	probe $0.755$ – $0.960$ , $r_{\mathrm{pool}}$ $1.8$ – $13.9$	Table 29
Anchor dim. $2\!\to\!128$ , $r_{\mathrm{tgt}}$ flat	$1.6$ – $2.0$	Table 30
Repair reaches near-ceiling	$14.377$ bits ( $100.0\%$ )	Table 6
Loss is the lever (one variable)	$14.359\!\to\!0.307$ ( $14.05$ )	Table 6
Target-frame effect, matched budget	$+0.071$ bits	Table 20
Schedule effect, matched steps	$+1.337$ bits	Table 20
Bits and A1 show no reliable relation	$\rho=-0.24$ , $n=10$	Table 6

Appendix

Table 12: Per-aspect membership and untrained in-degree. The singleton rows are what make the pooling control decisive.

Block	Papers	Encoder	Used for
A (trained pipeline)	$57{,}903$	MiniLM-L6, $384$ -d	Tables 1 , 5 , 23 , 19 , 25 , 28 , 29 , 30 ; cols. 2–3 of Table 3
B (diagnostic harness)	$58{,}145$	MPNet-base, $768$ -d	Tables 2 , 4 , 27 , 31 ; cols. 1 and 4 of Table 3
C (repair $+$ target audit)	$57{,}903$	MiniLM-L6, $384$ -d	Tables 6 , 20 , 7 , 15 , 22 , 21

Appendix

Table 13: Which run, corpus subset and sentence encoder produced each block. We keep these separate rather than pooling them, because the blocks measure different objects and the distinction carries the central results of Secs. 7 and 9 .

Symbol	Code name	Meaning
$r_{\mathrm{node}}$ ( $/128$ )	node_rk	effective rank of node latents
$r_{\mathrm{pool}}$ ( $/128$ )	pool_rk	effective rank of pooled patch vectors
$r_{\mathrm{cand}}$ ( $/128$ )	cand_erank	effective rank of the candidate bank
$r_{\mathrm{query}}$ ( $/128$ )	query_eff_rank	effective rank of the model’s predictions
$\mathrm{DC}_{\mathrm{ratio}}$	dc_ratio	$\\|\mu\\|/\mathbb{E}\\|x-\mu\\|$
$\mathrm{DC}_{\mathrm{energy}}$	dc_energy	$\\|\mu\\|^{2}/(\\|\mu\\|^{2}+\mathbb{E}\\|x-\mu\\|^{2})$

Appendix

Table 14: Notation. All effective ranks use Eq. 5 on mean-centred matrices. The distinction between

r_{\mathrm{cand}}

and

r_{\mathrm{query}}

is central: the former is what standard practice measures, the latter is where the failure lives.

Node type	count	Edge type	count	census-determined?
paper	57,903	(paper, has_claim, claim)	251,938	yes ( $=\|$ claim $\|$ )
claim	251,938	(paper, has_method, method)	57,903	yes ( $=\|$ paper $\|$ )
method	57,903	(paper, has_result, result)	57,903	yes ( $=\|$ paper $\|$ )
result	57,903	(method, produces, result)	57,903	yes ( $=\|$ paper $\|$ )
evidence	373,438	(result, grounds, claim)	251,938	yes ( $=\|$ claim $\|$ )
implication	251,938	(claim, supported_by, evidence)	251,922	no (data-derived)

Appendix

Table 15: Heterogeneous graph statistics (

57{,}903

Aspect	members/patch (mean)	max	patches with $\geq 2$	mean in-degree	role
claim	$4.35$	$10$	$99.5\%$	$4.48$	maskable, multi-member
method	$1.00$	$1$	$0.0\%$	$2.00$	maskable, singleton
result	$1.00$	$1$	$0.0\%$	$6.35$	maskable, singleton

Appendix

Table 16: Per-aspect membership and untrained in-degree. The singleton rows are what make the pooling control decisive.

Estimator	Assertion	Expected
AUC	perfect / inverted / constant / random	$1.0$ / $0.0$ / $0.500$ / ${\approx}0.5$
AUC ties †	constant scorer, ties averaged	exactly $0.500$
balanced accuracy	always-majority on $95{:}5$	$0.500$
$\mathcal{B}^{\max}$ †	matches the training code’s printed ceilings	$10.108$ , $14.379$
retrieval	perfect / random bits	$\mathcal{B}^{\max}$ / ${\approx}0$
linear probe	separable / random labels	$>0.90$ / ${\approx}0.5$

Appendix

Table 17: The

20

assertions that must pass before any measurement is taken. The harness refuses to proceed on failure. Items marked

\dagger

were added after an earlier version of the pipeline produced a confidently wrong number that the assertion would have caught; we list them in that spirit rather than as boilerplate.

Name	Meaning	Value
collapse_bits	below this many bits, a system is “collapsed”	$1.0$
probe_encodes_bits	above this, the context linearly encodes identity	$4.0$
control_capable_bits	above this, a control demonstrates capability	$1.0$
paired_alpha	level for paired bootstrap intervals	$0.05$
degenerate_bits_eps	below this gap, a control is degenerate with the model	$10^{-3}$
sim_match_bits	tolerance for “simulation reproduces observation”	$1.0$

Appendix

Table 18: Decision thresholds, fixed and written to disk before any result was computed and not revised afterwards. Each converts a measurement into a verdict; publishing them is what makes the verdicts auditable. The last four are the target-side gates introduced for Sec. 9 .

Frame	MRR	note
raw	$1.7\times 10^{-4}$	—
centre	$2.1\times 10^{-4}$	shared mean removed
all-but-the-top- $1$	$2.6\times 10^{-4}$	best
all-but-the-top- $2$	$2.1\times 10^{-4}$	—
ZCA (fitted on candidates)	$2.2\times 10^{-4}$	full whitening
candidate bank	DC ratio $37.6$	$r_{\mathrm{cand}}=18.1$

Appendix

Table 19: Post-hoc retrieval-frame sweep (three seeds, block A). All frames are fitted on the candidate bank and applied to both sides. The best frame buys

0.4\text{ bits}

against a

14.4\text{ bits}

deficit, as Prop. 3 predicts. This is a different experiment from the trained target-frame effect of App. A14 and the two must not be conflated: this sweep re-metrises an already-collapsed query bank, whereas that one changes what the objective asks for.

Frame	shared	$\mathcal{B}$	% of $\mathcal{B}^{\max}$	A1 probe
raw	$0$ (baseline)	$14.289$	$99.4\%$	$0.964$
center	$0$	$14.359$	$99.9\%$	$0.862$
raw	$1$	$11.808$	$82.1\%$	$0.985$
center	$1$	$14.379$	$100.0\%$	$0.963$
target-frame main effect		$+0.071$	$79\%$ of the $0.090$ available
shared-basis main effect		$-2.481$

Appendix

Table 20: The

2\times 2

over target frame and shared basis at matched budget (

3

k steps, one seed, block C, ceiling

14.379

; center removes the per-aspect target mean, shared ties the input projection across node types). Read the factorial, not the best cell. The baseline cell already reaches

99.4\%

of ceiling, so only

0.090

bits are available to any factor here; this is the reducibility of Prop. 5 showing up as a compressed dynamic range.

Property	Value	Note
pre-registered void line	$0.70$	App. A10
observed range across cells	$0.711-0.985$	Table 6

Appendix

Table 21: The A1 probe: construction, references and floors. The void line

0.70

is pre-registered (App. A10 ): a reading below it is treated as carrying no signal, and no cell in Table 6 falls below it.

Quantity	Value	Note
claim nodes	$251{,}938$
supported_by edges	$251{,}922$	${\approx}1$ per claim
challenged_by edges	$121{,}516$	$48.2\%$ of claims
evidence nodes	$373{,}438$	$=251{,}922+121{,}516$ exactly
implies edges	$251{,}938$	$1$ per claim (census-determined)
papers with $\geq 1$ challenge	$35{,}650$	$61.6\%$ of corpus

Appendix

Table 22: Census of the data-derived relations (block C). Every evidence node is a leaf—

373{,}438=251{,}922+121{,}516

exactly—because the source fields are plain strings, so each claim yields one supporting and at most one contradicting evidence node. This is the structural fact behind the

+0.1465

-AUC edge-type leak.

Aspect	$m$	$r_{\mathrm{node}}$	$r_{\mathrm{pool}}$	$r_{\mathrm{query}}$	MRR	MRR (centred)	margin $z$
claim	$4.35$	$5.40$	$4.91$	$1.87$	$1.5\times 10^{-4}$	$2.2\times 10^{-4}$	$-0.006$
method	$1.00$	$47.42$	$47.31$	$1.97$	$2.1\times 10^{-4}$	$1.5\times 10^{-4}$	$-0.001$
result	$1.00$	$2.04$	$2.05$	$1.62$	$2.0\times 10^{-4}$	$2.0\times 10^{-4}$	$+0.033$
Chance					$1.99\times 10^{-4}$	$1.99\times 10^{-4}$	$0$

Appendix

Table 23: Per-aspect breakdown (block A, five seeds). method and result are exact singletons, so mean pooling is the identity map:

r_{\mathrm{node}}\approx r_{\mathrm{pool}}

to two decimals.

z

is the gold item’s standardised similarity margin;

z\approx 0

means the gold candidate is indistinguishable from the pool mean.

Figure 4: The control that exonerates pooling and localises the failure to the query side. (a) For method ,

m{=}1.00

, so mean pooling is provably the identity map: node rank

47.4

carries through to pooled rank

47.3

. The model’s predictions for those patches nevertheless occupy effective rank

1.97

(black diamonds), at or below the

3

-category bound of Prop. 1 . (b) A

23\times

range in pooled rank produces no movement in bits recovered; dotted line is chance, dashed line the

14.379

-bit ceiling. Five seeds.

Quantity	Value ( $/128$ )
Node-latent effective rank $r_{\mathrm{node}}$	$18.3\pm 0.9$	healthy
Pooled-latent effective rank $r_{\mathrm{pool}}$	$1.97\pm 0.04$	low, but see Table 23

Appendix

Table 24: Isolated pre-target rank measurement. A mean-pool JEPA with a Euclidean-cosine target, no hyperbolic map and no anchors. Mean

\pm

std over five seeds. The per-aspect breakdown above shows this aggregate conceals a

23\times

spread—which is why the aggregate, alone, misled us.

Features	Aspect	ctx MRR	no-summary	bm25	bm25-no-ov
whitened	claim	$\mathbf{0.973}$	$0.953$	$0.979$	$0.732$
	method	$\mathbf{0.908}$	$0.834$	$0.974$	$0.802$
	result	$\mathbf{0.938}$	$0.921$	$0.989$	$0.854$
raw	claim	$0.943$	$0.916$	—	—
	method	$0.878$	$0.796$	—	—
	result	$0.925$	$0.916$	—	—

Appendix

Table 25: Oracle retrieval and leakage controls , per aspect, block A. bm25-no-ov is a lower bound on the non-lexical signal; bm25 uses no learned representation. no-summary removes the paper title/abstract node from the context. A cheat query equal to the target’s own embedding validates the metric.

Figure 5: Four training-free or positive-control retrievers, all near ceiling; the baseline trained model at chance (dotted). bm25 is the strongest evidence because it uses no learned representation whatsoever, and bm25-no-ov still recovers

+14.250

bits—

99.1\%

of the

14.385

-bit ceiling—after deleting every shared

5

-gram.

	frozen text features	hetero node features
reference predictor	$+14.220$ bits ( p1 )	not yet reported
full pipeline	not yet reported	$0.00$ bits (Table 1 )

Appendix

Table 26: Factorial design. The two diagonal cells are the systems reported in the main text; the off-diagonal cells isolate each factor and are not yet reported. We list them explicitly rather than omitting them, so that the limits of the attribution are visible: on the evidence in this paper, p1 bounds the harness and does not separate feature space from trainer. Both missing cells are

3

k-step runs on the existing cache.

$\rho$	$0.0040$	$0.5$	$0.9$	$0.99$	$\mathbf{0.9961}$	$0.999$	$0.9999$
MRR, $\rho$ only	$1.000$	$1.000$	$0.8928$	$0.1068$	$0.0204$	$0.0043$	$0.0014$
$\mathcal{B}$ , $\rho$ only	$+12.845$	$+12.845$	$+12.200$	$+3.833$	$+1.911$	$+0.843$	$+0.266$
MRR, matched $\hat{\kappa},\hat{\varepsilon}$	$1.000$	—	—	—	$0.0171$	—	—
$\mathcal{B}$ , matched	$+12.845$	—	—	—	$+1.80$	—	—
bits lost, matched	$0.000$	—	—	—	$11.045$	—	—
Leave-one-out at the measured operating point

Appendix

Table 27: Phase analysis, full sweep (bank size

20{,}000

, so

\mathcal{B}^{\max}=12.845

by Eq. 4 ). Row 1 varies

\rho

with

\kappa=\varepsilon=0

; row 2 uses the measured

\hat{\kappa}=0.942,\hat{\varepsilon}=1.515

. The leave-one-out block shows that neither nuisance parameter is load-bearing.

Figure 6: The objective optimises the wrong variance component. Left: the frozen inputs,

86.05\%

of variance between papers. Right: the trained latents,

0.39\%

. Whitening the inputs raises the trained instance share to

9.86\%

and bits recovered still do not move—the invariance Prop. 1 predicts; changing the objective does (Table 6 ).

Pooling	Loss	$r_{\mathrm{pool}}$	probe	MRR	$\mathcal{B}$
mean	$1-\cos$ (baseline)	$1.97$	$0.871$	$1.9\times 10^{-4}$	$0.00$
mean	infonce	$1.96$	$0.867$	$2.1\times 10^{-4}$	$0.00$
sum	$1-\cos$	$1.25$	$0.963$	$1.8\times 10^{-4}$	$0.00$
deepsets	$1-\cos$	$1.70$	$0.604$	$1.9\times 10^{-4}$	$0.00$
deepsets	infonce	$1.81$	$0.742$	$1.7\times 10^{-4}$	$0.00$
attn	infonce	$2.03$	$0.964$	$\mathbf{2.3\times 10^{-4}}$	$0.00$

Appendix

Table 28: Seven interventions, one flat line (block A, five seeds, Protocol R).

r_{\mathrm{pool}}

spans

1.25

–

2.03

and the probe spans

0.604

–

0.964

; bits recovered stay at

0

in every cell, against a ceiling of

14.379

. What this grid does not vary is the target frame and the learning-rate schedule, both of which App. A14 shows to matter; we therefore present it as a negative result about the aggregator and the in-batch-negative form of the loss, not about the pipeline.

Figure 7: Seven interventions, one flat line. Raw and centred cosine, mean

\pm

std over five seeds. Dotted line is chance (

\mathcal{B}=0

), dashed line the

14.379

-bit ceiling.

Objective	Target construction	Probe acc	$r_{\mathrm{pool}}/128$	Rank verdict
hyperbolic, 2-D	mean (own only)	$0.840$	$1.9$	collapsed
	rel-hetero $+$ white	$0.822$	$3.1$	collapsed
	rel-hetero (raw)	$0.940$	$1.8$	collapsed
hyperbolic $+$ vic	mean (own only)	$0.859$	$2.0$	collapsed
	rel-hetero $+$ white	$0.812$	$\mathbf{13.9}$	rank-recovered
	rel-hetero (raw)	$0.954$	$2.2$	collapsed

Appendix

Table 29: Twenty objective

\times

target-construction cells; the dissociation holds across all of them. Probe accuracy spans

0.755

–

0.960

and pooled effective rank

1.8

–

13.9

—a

7.7\times

range including two cells that clear the conventional

r_{\mathrm{pool}}>10

threshold—with no accompanying change in retrieval. Mean over five seeds.

Objective	Target	Probe acc	$r_{\mathrm{pool}}/128$	$r_{\mathrm{tgt}}$
hyperbolic, 2-D	mean (2-D)	$0.840$	$1.9$	$2.0$ (ceiling)
	$k{=}4$ ( $\to\mathbb{R}^{8}$ )	$0.869$	$1.9$	$1.6$
	$k{=}16$ ( $\to\mathbb{R}^{32}$ )	$0.869$	$1.9$	$1.9$
	$k{=}64$ ( $\to\mathbb{R}^{128}$ )	$0.867$	$1.9$	$1.9$
hyperbolic $+$ vic	mean (2-D)	$0.859$	$2.0$	$2.0$ (ceiling)
	$k{=}4$	$0.859$	$2.0$	$1.7$

Appendix

Table 30: Raising the target’s algebraic rank ceiling from

2

2k

does not raise the measured target rank. Each patch angle is replaced by

k

angles against QR-orthonormalised anchors. The ceiling rises to

128

k{=}64

and the measured target effective rank

r_{\mathrm{tgt}}

does not move.

Probe	Detail	Value
aspect classification, function words only	fixed stopword vocabulary	$0.937$
aspect classification, full TF-IDF	$50$ k features	$0.986$
chance	three aspects	$0.333$
templating index	$(\text{acc}-\text{chance})/(1-\text{chance})$	$0.906$
top $4$ -gram coverage, claim	“this suggests that the”	$31.8\%$
top $4$ -gram coverage, method	“the study employed a”	$39.0\%$

Appendix

Table 31: Templating audit on the

58{,}145

set; text probes on an

8{,}000

-document subsample, variance probes on all records. The function-word classifier uses a fixed stopword vocabulary and cannot access content. The final block is the placeholder-text audit of Sec. 9 , on the block-C relations.

Dataset	ours	reported	verdict
MUTAG	$0.686$	$0.874$	not reproduced
PROTEINS	$0.739$	$0.750$	within tolerance

Appendix

Table 32: Our reference re-implementation on the original method’s graph-classification benchmarks. We do not clear MUTAG and therefore do not claim implementation faithfulness.

	claim	method	result	paper
whitened	$320.8$	$309.2$	$313.1$	$313.1$

Appendix

Table 33: Effective rank of the frozen input features (of

384

) per node type after PCA/ZCA whitening, applied per node type before the encoder. The per-node-type raw input ranks were not written to the released logs for this run and we therefore do not tabulate them; raw input anisotropy is instead characterised by the depth-

0

DC ratios of App. A29 (

3.1

–

3.6

), and the aggregate raw-to-whitened change is the

310.9

-of-

384

figure quoted for rung 4 of Table 1 . The whitened values below are the ones the ladder’s rank-restoration claim rests on.

Figure 8: Trained patches form one point-mass per aspect. First two principal components. Trained representations (left, centre) occupy two or three tight clusters corresponding to aspect type; the frozen features (right) form a diffuse cloud in which individual papers are separable. The representation encodes which aspect , not which paper —the visual form of Table 3 and of the $\operatorname{erank}\leq|\mathcal{A}|$ bound of Prop. 1 .

Figure 9: Depth is not the binding factor. (a) Bits recovered across six encoder configurations from a depth-

0

linear map to a depth-

3

GNN; all at chance, the

14.379

-bit ceiling dashed. (b) Message passing compresses node rank monotonically (

54.9\to 18.0

) and the pooled rank is flat regardless.

Figure 10: Nothing is learned away. (a) Training loss converges within

{\sim}15

epochs, on a twin axis because the cosine and InfoNCE losses differ by an order of magnitude—and note that this is exactly the comparison Sec. 9 warns against making on magnitude alone, since a regression loss near zero is collapse while an InfoNCE loss near zero is discrimination. (b) Pooled effective rank is already at its final value at epoch

0

and stays flat: the baseline pathology is present at initialisation rather than induced by optimisation, which is why Sec. 7.1 measures the variance allocation per checkpoint rather than asserting a re-allocation. Contrast the repaired configuration, whose bits rise from

-0.07

at step

1

+14.24

by step

500

(App. A14 ); the difference between the two trajectories is the objective , as the single-variable regression control establishes.

Figure 11: Bits recovered against pool difficulty for the three ceilings and the lexical no-overlap control (Table 2 ). All four series are ceilings or controls— bm25 , bm25-no-ov , oracle and the p1 positive control—and all four track the

\frac{1}{N}\log_{2}N!

envelope with a near-constant deficit across a

5{,}800\times

change in pool size. The baseline Graph-JEPA is the single point at

0

bits at the right-hand end. This figure replaces every ratio-to-chance statement in earlier drafts.

Figure 12: Shared-mean dominance by pipeline stage. DC ratio

\|\mu\|/\mathbb{E}\|x-\mu\|

on a log axis. The frozen features sit near

10^{-2}

; the trained query bank sits at

162.2

—a factor of

{\sim}10^{4}

Figure 13: A constant query cannot be rescued by any frame applied after training. Five retrieval frames fitted on the candidate bank; the best buys

0.4\text{ bits}

against a

14.4\text{ bits}

deficit to the ceiling (dashed). This is a quantitative confirmation of Prop. 3 rather than a failed ablation: the way out is to change what the objective asks for, and the single-variable demonstration of that is the loss ablation of Sec. 9 (

14.359\!\to\!0.307

bits), not any transformation of the retrieval space.

Stage	Wall-clock	Peak GPU	GPU-hours
Preprocess: graph build $+$ sentence embed (one-time)	${\sim}7$ min †	—	—
Preprocess: RWSE cache (one-time)	${\sim}20$ s	—	—
Instrument self-test ( $20$ assertions)	$18$ s	$1.2$ GB	$<0.01$
Untrained rank / DC vs. degree	$1$ s	$4.0$ GB	$<0.01$
Protocol R, raw $+$ whitened (10 runs)	$6$ m 45 s	$18.8$ GB	$0.11$
Per-aspect breakdown (5 seeds)	$3$ m 16 s	$18.1$ GB	$0.05$

Appendix

Table 34: Measured compute budget. The Protocol R diagnosis costs

0.77

GPU-hours; the additional diagnostic harness of Secs. 6 – 7 costs

4.7

minutes on the same node. The low cost is deliberate: every check recommended in Sec. 10 is cheap enough to run before committing to a training budget—and the two checks that changed this paper’s conclusions, the reducibility audit and the duplicate census, are the two cheapest lines in the table.

Representation Collapse

Representation collapse is the loss of diversity in learned features, causing distinct inputs to map to similar or low-dimensional embeddings.

See all 47 papers

Aug 5, 2026

NodeJEPA: Structure-Conditioned Latent Prediction for Node-Level Graph Self-Supervised Learning

Self-supervised learning on graphs is largely shaped by contrastive methods that depend on carefully designed augmentations, and by generative methods that reconstruct node attributes in the input space. Both paradigms can entangle representations with low-level input statistics rather than with relational structure. Joint-embedding predictive architectures (JEPA) instead learn by predicting latent targets rather than reconstructing inputs. Recent work has explored this idea for graph-level representation learning, but how to design JEPA-style objectives for node-level tasks, and which structural signals the predictor should condition on, remains less clear. We present NodeJEPA, a joint-embedding predictive architecture for node-level graph self-supervised learning. NodeJEPA masks structure-aware k-hop ego-subgraphs and trains a context encoder to predict the latent representations of the masked nodes. These targets come from an EMA-updated target encoder with stop-gradient. A structure-conditioned predictor integrates spectral and centrality descriptors through cross-attention. Variance, covariance, and Laplacian spectral regularizers help stabilize the embedding geometry, and an optional curriculum gradually increases masking difficulty during training. Because prediction occurs in latent space, NodeJEPA does not rely on input reconstruction or hand-crafted graph augmentations. We evaluate NodeJEPA on standard node classification benchmarks under linear probing and fine-tuning protocols, and conduct ablations on masking, prediction, and regularization design choices. Our study offers a practical recipe for node-level JEPA-style latent prediction on graphs, and clarifies when structural conditioning helps representation learning. Code, configurations, and evaluation scripts are publicly available at https://github.com/OliverZ-dot/Node-Jepa.

Tinghe Zhang, Jian Xu, Jiaheng Chen +3

1Northeastern University

Graph Representation Learning Graph SSL Joint-Embedding Predictive Architecture Predictive Representation Learning

Ask PDF

Aug 18, 2026

ClosureBench: A Constructive Benchmark for Compositional Graph Reasoning

Large language models fail on multi-step compositional reasoning, but measuring that failure is hard, because new models are trained on the benchmarks used to evaluate them. A fixed test set becomes a memorisation check soon after release. Constructive benchmarks avoid this by generating instances on demand. We introduce ClosureBench, a constructive benchmark for graph-relational logical reasoning. Each task is built from explicit primitives (reachability, degree, set operations, connectivity, aggregation), and its reference answer is computed by executing code that implements that logic exactly. Ground truth is therefore verified, and the supply of fresh instances is unlimited. The benchmark spans 26 task categories at three compositional levels, with three independent difficulty axes: graph size, edge density, and query depth. We evaluate models from 1.5B open weights to frontier systems (o3, GPT-4.1, Gemini 2.5, Claude Sonnet 4). Accuracy falls as graph size and query depth increase, and the two axes interact. The difficulty does not lie in the surface form, since it persists when the graph is given as a JSON edge list or an adjacency matrix rather than prose, nor in the reasoning rule, which models state correctly. It lies in carrying that rule out over the graph across many steps. A 4B model fine-tuned to emit verified programs instead of answers stays nearly flat across compositional levels, while every frontier model degrades. o3 falls from 96% on atomic queries to 82% on the most compositional; the 4B model holds at 93% at a fraction of the token cost. The program offloads multi-step execution to a runtime, and the model's remaining errors are almost entirely misread edges. Constructive generation also supports a direct memorisation check, comparing accuracy on seen and fresh instances.

Stefano Goria

AIM Research Lab

LLM Reasoning with Graphs Language Model Reasoning Evaluation Graph Reasoning LLM Reasoning

Ask PDF

Jul 20, 2026

PEARL: Auditable Repair for Scientific Reasoning Graph Extraction

Scientific Reasoning Graph Extraction (SRGE) aims to recover explicit links among observations, evidence, intermediate claims, and paper-level conclusions. LLMs can produce graph-like scientific explanations, but their outputs often mix malformed syntax, drifting edge labels, incorrectly oriented roots, and weak source anchors. We propose PEARL (Peircean Extraction via Abstraction and Repair Layer), a training-free framework that turns noisy LLM graph responses into auditable reasoning graphs and repairs them toward strict semantic validity. PEARL first materializes explicit graph content under a closed Peircean schema, then uses matched evidence-grounded judge feedback to repair rejected edge types, local inference steps, and terminal roots while preserving an audit trail. On five 70-paper model archives from ARCHE, a benchmark for latent reasoning-chain extraction, PEARL raises strict gate passes from 0/350 for the LLM baseline to 300/350, with average REA improving from 0.339 to 0.906. The graphs provide a reliability layer for research-agent and AI scientist workflows that need inspectable reasoning traces rather than unconstrained graph regeneration. Code and audit artifacts are available at https://github.com/BohanSu/auditable-repair-reasoning-graphs/tree/300-350_workshop .

Bohan Su, Pengze Li, Yuchen Lu +1

School of Computer Science, Wuhan University · Artificial Intelligence Innovation and Incubation, Fudan University · Department of Computer Science, Faculty of Science, University of Bath +1

LLM Reasoning with Graphs LLM Auditing LLM Reliability LLM Information Extraction

Ask PDF

Abstract

Representation Collapse

Explore similar work

NodeJEPA: Structure-Conditioned Latent Prediction for Node-Level Graph Self-Supervised Learning

ClosureBench: A Constructive Benchmark for Compositional Graph Reasoning

PEARL: Auditable Repair for Scientific Reasoning Graph Extraction