A Bounded Evaluation Protocol for Representational Systematicity
Organizations: School of Future Technology, South China University of Technology, China
Abstract
Multiple studies have shown that neural networks can be behaviorally systematic, producing correct outputs for novel combinations of familiar parts. Whether they are also representationally systematic, that is, whether they represent these parts and their roles and use them to produce the outputs, remains hard to test, because each kind of evidence can mislead on its own: correct outputs can come from shortcuts, a decodable signal need not be used, an intervention can act through unintended pathways, and a failed search does not show absence. We propose a bounded evaluation protocol for representational systematicity (Bounded-RS): for an explanation that attributes a systematic behavior to a declared representation, it tests on a matched behavioral contrast whether that representation is recoverable and whether it is causally used, checks the validity of this evidence, and assigns the explanation one of four bounded states, Unsupported, Present-only, Causal-use, or Undetermined, through control-calibrated scores, gates, and stop rules. On a semantic-parsing testbed of 96 matched sets that we release, the tested role-filler Binding explanation of fine-tuned T5 models is Causal-use and the distributed Geometry explanation is Unsupported within its declared search space, verdicts that hold across calibration-compatible parameter configurations, training seeds, and T5-base. On 41 cases with known causal status, full adjudication licenses a false Causal-use in 1 of 34 noncausal cases, against 13 when recoverability is taken as use, without missing any causal case. Bounded-RS thus turns the question of representational systematicity, debated since Fodor and Pylyshyn, into comparable and falsifiable verdicts on declared explanations, each reported with the evidence pattern that licenses it and, for a negative, the search space that bounds it.
Figures & tables
| Decision | Minimum evidence pattern | Licensed claim |
|---|---|---|
| Unsupported | Weak, unstable, or unlinked presence evidence. | The protocol does not support the relevant representation or its causal use. |
| Present-only | Positive recoverability evidence, with insufficient intervention, necessity, or repair evidence. | The representation is recoverable, but causal use is not supported. |
| Causal-use | Recoverability plus controlled intervention, necessity, or repair evidence, with realism and linkage gates passing. | The representation is supported as present and causally used under the tested operations. |
| Undetermined | Incomplete evidence, state-score margin below threshold, or failed or conflicting gates. | No stronger state decision is licensed. |
| Evidence role | Blocks | What it rules out |
|---|---|---|
| Behavioral anchor | Exact match and clause-level F1 on matched recombinations. | Representation-level claims when the target behavior is absent or unstable. |
| Recoverability | Family-specific presence readout with surface, filler, and predicate controls. | Claiming causal use for a signal that is not stably recoverable. |
| Intervention tests | Interchange, necessity, and repair interventions during generation. | Treating a merely decodable or correlational signal as used by the model. |
| Validity checks | Realism, item-linkage, shuffled, wrong-role, non-role, and generic-gain controls. | Accepting off-manifold interventions, aggregate-only effects, or generic decoder improvements. |
| Matched item | Controlled relation | Protocol role |
|---|---|---|
| Matched easy source | Same gold meaning as hard target; active voice without embedding. | Provides source signal for repair and matched behavior controls. |
| Hard target | Same gold meaning; the set’s construction (passive voice, a relative clause, or neither). | Target for repair and failure analysis. |
| Agent replacement | Only the agent filler changes. | Tests role-specific sensitivity to who acts. |
| Theme replacement | Only the theme filler changes. | Tests role-specific sensitivity to who is acted on. |
| Role swap | Agent and theme exchange roles. | Tests role identity rather than name memorization. |
| Filler replacement | Both role fillers change. | Tests recombination under the same role structure. |
| Panel | Outcome | Diagnostic role | |
|---|---|---|---|
| Binding calibration | 3 | all intended states | Controls separate the three evidential regimes. |
| Binding held-out controls | 6 | acc. 0.833; macro-F1 0.822 | The frozen rule holds outside the calibration panel; one Present-only control is escalated. |
| Binding target audit | 5 | 1 Causal-use , 1 Present-only , 1 Unsupported , 2 Undetermined | The normally fine-tuned target is Causal-use ; other regimes are separated or withheld. |
| Geometry held-out controls | 9 | acc. 0.778; macro-F1 0.796 | Present-only is the least stable Geometry state. |
| Geometry target audit | 7 | 2 Causal-use , 1 Present-only , 3 Unsupported , 1 Undetermined | Operation-level states before closure; no family-level claim. |
| Geometry closure | 36 objects in 3 classes | 0 objects pass closure | Licenses a family-final Unsupported under the frozen basis. |
| Closure component | Candidates | Passed | Failure mode |
|---|---|---|---|
| Observability seal | 1,536 trace rows | sealed | A1 selected; predicate-step coverage 1.0; argument steps not treated as validated object evidence. |
| Interaction objects | 12 objects; 4 survivors | 0 | Class not accepted at discovery, so no object can pass the causal-use (necessity) gate. |
| Cross-step bridges | 12 objects; 8 survivors | 0 | Class accepted; all 12 objects fail the causal-use (necessity) gate: gaps below threshold, null bands violated, realism below threshold. |
| Query–source relations | 12 objects; 4 survivors | 0 | Class not accepted at discovery, so no object can pass the causal-use (necessity) gate. |
| Direct Present-only route | 8 object–regime rows | 0 | No global Present-only route. |
| Slice-local Present-only route | tested slices | 0 | No local slice passes. |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Block | Core question | Evidence produced | Controls or rejection checks |
|---|---|---|---|
| Behavioral anchor | Does the model exhibit the target systematic behavior on the matched contrast? | Exact match and clause-level F1 between generated and gold logical forms; pass, fail, and control strata. | Used only as an anchor; cannot by itself assign a state or produce Causal-use . |
| Presence readout | Is the family-specific target signal recoverable from hidden states? | Readout accuracy, controlled readout gaps, and stability across matched items. | Surface position, filler identity, predicate identity, and other simple correlates. |
| Interchange intervention | Does transferring the signal across matched items change generation in the predicted direction? | Intervention effect and advantage over the strongest matched control. | Wrong-role, non-role, and shuffled controls. |
| Necessity intervention | Does removing or dampening the signal selectively degrade the target output? | Selective degradation of target parses relative to controls. | Degradation must be selective rather than a global disruption of generation. |
| Repair intervention | Does inserting a matched source signal into a hard target repair its output? | Repair advantage on hard targets over controlled repair baselines. | Path-only, wrong-role, non-role, shuffled, and generic-decoder-gain controls. |
| Item linkage | Do improvements occur on the items that the explanation predicts? | Item-linked effect pass rate and aggregate-versus-item diagnostics. | Rejects aggregate-only improvements not linked to intended items. |
| Evidence group | Feature | Scoring configuration | Calibration source |
|---|---|---|---|
| Behavior | Clause-F1 anchor | Direction ; boundary with floor ; scale ; weight prior. | Behavioral pass/fail controls |
| Presence | Semantic-role readout accuracy | Direction ; boundary ; scale ; weight . | State-validating controls |
| Presence | Surface-control gap | Direction ; boundary ; scale ; weight . | Surface, filler, and predicate controls |
| Use | Interchange or repair advantage over strongest control | Direction ; boundary ; scale ; weight . | Causal-use vs. Present-only controls |
| Use | Permutation or shuffled-control significance | Direction ; boundary ; scale ; weight . | Shuffled and wrong-role controls |
| Gate | Criterion | Required for | Calibration source |
|---|---|---|---|
| Realism gate | Natural-state realism pass rate must exceed . | Intervention states | Realism calibration controls |
| Linkage gate | Item-linked effect pass rate must exceed . | Intervention states | Item-linkage controls |
| Coverage gate | Required evidence coverage must exceed . | All state decisions | Held-out control panels |
| Decision gate | Top-state score margin must exceed . | States stronger than Undetermined | Held-out control panels |
| Stage | Evidence | Result |
|---|---|---|
| Behavior | Exact match 6/7 ; clause-F1 0.947. | Behavior score 0.977. |
| Presence | Role readout 0.938; surface-control gap 0.071. | Both presence features pass. |
| Controlled use | Interchange 0.512 vs. strongest control 0.181; necessity 0.276 vs. global control 0.058; repair 0.464 vs. strongest control 0.161. | Interchange and repair advantages 0.331 and 0.303. |
| Validity and decision | Realism 0.714, linkage 0.571, coverage 0.857. | All gates pass; final state Causal-use . |
| Layer | Shared component | What a new explanatory family must provide |
|---|---|---|
| Data | Matched sets for the behavioral contrast. | Usually unchanged when the family explains the same contrast; redesigned when the contrast changes. |
| Evidence slots | Behavioral anchor, presence readout, interchange, necessity, repair, item linkage, and gates. | Target signal, detector, interchange, necessity, and repair operations, and controls. |
| Adjudication | State semantics, scoring grammar, gates, stop rules, and closure driver. | Control-fitted feature boundaries and, for a family-level negative, a closure basis. |
| Artifact | Data generator, protocol runners, reports, and documentation. | Documentation of operations, expected runtime, and failure modes. |
| Object class | Matched detector | What the detector tests |
|---|---|---|
| Interaction objects | Predicate-conditioned relation detector | A layer-local relation subspace read conditionally on the predicate. |
| Cross-step bridges | Cross-step bridge detector (cross-step alignment) | Whether relation information forms a traceable route across decoder steps; a max-statistic controls multiplicity over the frozen candidate search. |
| Query–source relations | Argument-locus detector | Whether a decoding query bears a structured relation to the relevant source argument. |
| Analysis | Design | Conclusion |
|---|---|---|
| Parameter sensitivity | 729-configuration grid; 487 calibration-compatible | Both verdicts hold in all 487 configurations; the nearest closure flip lies outside the compatible region. |
| Training seeds | Five T5-small seeds | Binding Causal-use and Geometry Unsupported in 5/5. |
| Matched-set bootstrap | 1,000 cluster replicates over 96 sets | No Geometry object passes closure; no Causal-use reversal. |
| Truth controls | Engineered triplets, three seeds | The cross-step Geometry branch and Binding recover all three evidential states. |
| Component rules | 41 predeclared cases with known causal status | Full adjudication licenses the fewest false Causal-use states and keeps all causal cases. |
| T5-base transfer | Three seeds; frozen and recalibrated rules | Both verdicts replicate in 6/6 seed-by-rule runs. |
| Panel | Outcome | Note | |
|---|---|---|---|
| Initial calibration | 3 | all intended states | Separates Unsupported , Present-only , and Causal-use controls. |
| Held-out controls | 6 | acc. 0.833; macro-F1 0.822 | Two controls per state. |
| Behavior-feature ablation | 6 | no change in acc. or macro-F1 | Behavior feature share is 0.041. |
| Initial target audit | 3 | all Causal-use | Mean top-state score and decision margin are both 1.0. |
| Heterogeneous target audit | 5 | 1 Causal-use , 1 Present-only , 1 Unsupported , 2 Undetermined | Normally trained, weak, decoder-scaled, instruction-prompted scaled, and surface-shifted targets. |
| Geometry panel | Causal-use | Present-only | Unsupported | Undetermined | |
|---|---|---|---|---|---|
| Pre-closure target audit | 7 | 2 | 1 | 3 | 1 |
| Variant A | 6 | 3 | 1 | 2 | 0 |
| Promoted Variant A | 6 | 0 | 1 | 5 | 0 |
| Variant B | 6 | 0 | 0 | 6 | 0 |
| Variant C | 6 | 0 | 0 | 6 | 0 |
| Variant D | 6 | 6 | 0 | 0 | 0 |
| Detector class | Canonical examples | Coverage | Validation outcome |
|---|---|---|---|
| Argument locus | 336 | 1.0 | not validated |
| Predicate-conditioned relation | 336 | 1.0 | not validated |
| Cross-step bridge | 336 | 1.0 | not validated |
| License rule | False Causal-use (noncausal cases) | Causal cases retained | Abstentions |
|---|---|---|---|
| Recoverable causal | 13/34 | 7/7 | – |
| Ungated intervention | 7/34 | 7/7 | – |
| Full Bounded-RS | 1/34 | 7/7 | 6/41 |
| Paper term | Artifact label |
|---|---|
| Unsupported , Present-only , Causal-use , Undetermined | H0 , H1 , H2 , underdetermined |
| Presence readout, interchange, necessity, repair, item linkage | Modules M1 , M2 , M3 , M4 , M5 |
| Binding and Geometry families | G1_binding and G2_distributed_geometry |
| Matched set | counterpart_family |
| Matched easy source and hard target | rescue_donor and rescue_target |
| Repair | rescue |
Explainability Evaluation
Explainability evaluation tests whether AI explanations are faithful, stable, understandable, useful, and reliable across models and settings.
See all 348 papers