cs.AIMar 29, 2026

What Does a System Modify When It Modifies Itself?

Authors: Florentin Koch

Organizations: École Polytechnique, Palaiseau, France

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When a cognitive system modifies its own functioning, what exactly does it modify: a low-level rule, a control rule, or the criterion that evaluates its revisions? Cognitive science describes executive control, metacognition, and hierarchical learning, while artificial intelligence modifies policies, parameters, and learning mechanisms, but the two fields lack common criteria for comparing these transformations. We propose a minimal analytical model distinguishing functional rules Phi_t = {R0, ..., Rk}, modification mechanisms Mt, and evaluation criteria Nt. It separates two independent dimensions: the depth of the modified target and whether the modification is blind or reflexive. Internal representational access AtR and endogenous causal control AtC are further distinguished from external inspectability and modifiability by a designer. Four regimes are defined by the target of modification: action without organizational modification, modification of low-level rules, modification of control rules or mechanisms, and revision of the evaluation criterion. Each is related to cognitive phenomena and artificial systems while making explicit the system boundary on which attribution of endogenous self-modification depends. The comparison yields a conditional crossed-opacities hypothesis: humans often have richer endogenous self-description at abstract and strategic levels than at implementation levels, whereas current artificial systems may be externally inspectable and modifiable at operational levels without corresponding self-representation or endogenous control. The framework also identifies three difficulties - viability, evaluation of revisions to evaluation criteria, and continuity of identity - and proposes two empirical discriminants for identifying the target of a modification and the causal role of self-representation.

Figures & tables

NotationDefinition
Φ\Phi tOrganization of functional rules at time t: {R0(t), …, Rk(t)}
Ri(t)Level-i functional rule at time t
R0Lowest-level rule: local policy, weight, or associative strength
R1Control or learning rule bearing on R0
O\mathcal{O} tComplete analytical organization: ( Φ\Phi t, Mt, Nt)
MtSet of modification mechanisms; Mi →\to j can transform Rj
Table 1: Notation used in the analytical framework
Figure 1: Target depth (local rule, control rule or mechanism, evaluation criterion) and mode of modification (blind, representational without leverage, or reflexive); internal access is distinguished from external inspectability and modifiability
RegimeDefining targetPossible reflexivityCognitive exampleArtificial example
1. FixedNo modification of O\mathcal{O} tNot requiredResponse during an episode without concurrent plasticityFixed-transition automaton
2. LocalR0Blind or reflexiveLocal associative adjustmentParameter update if the optimizer is inside the boundary
3. StructuralRi (i > 0) or Mi →\to jBlind or reflexiveTask-set reconfiguration; metacognitive revisionFunctional meta-RL; integrated architecture search
4. EvaluativeNtBlind or reflexive; strong variant if representation is causally activePreference transformation; clinical case to be establishedNo uncontroversial current example
Table 2: Four self-modification regimes defined by target depth
System and levelInternal representational access AtREndogenous control AtCExternal access EtR / EtC
Human, abstract levelsRelatively rich but reconstructiveIndirect, variable, and imperfectLimited external measurement
Human, local implementationWeakWeak or indirectPartial technical access
Current AI, operational levelsOften unestablishedDepends on the boundary and architectureOften high for the designer
Reflective architecture, operational levelsPotentially richPotentially strong if representation is causally activeHigh
Artificial evaluation criterionGenerally unestablishedGenerally external or constrained by a metacriterionHigh for the designer
Table 3: Comparative profiles of internal and external access and control
More in this topic

Recursive Self-Improvement

Recursive self-improvement studies AI systems that use feedback from their own work to modify and evaluate future versions.

See all 83 papers

Explore similar work

CardsList
Apr 21, 2026

Deconstructing Superintelligence: Identity, Self-Modification and Différance

Self-modification is routinely treated as constitutive of artificial superintelligence (SI), yet modification is a relative action requiring a supplement outside the operation. We formalise this on an associative operator algebra A\mathcal{A} with update operator U^\hat U, difference operator D^\hat D, and self-representation operator R^\hat R, identifying the supplement with Comm⁡(U^)\operatorname{Comm}(\hat U). A propagation theorem shows [U^,R^][\hat U,\hat R] decomposes through [U^,D^][\hat U,\hat D], so non-commutation propagates to self-representation. The liar paradox is the rank-one case [T^,ΠL]=0[\hat T,Π_L]=0, and \emph{class A\mathbf{A}} systems, in which U^\hat U acts on D^\hat D, reproduce it at system scale, yielding a structure coinciding with Priest's inclosure schema and Derrida's différance. Our results show that the strong self-modification taken to define superintelligence may undermine the persistent identity upon which such systems are premised.

Elija Perrier

Centre for Quantum Software & Information, UTS, Sydney

Topic pending
AskPDF
Sep 21, 2026

Self-Healing Harness for Runtime Oversight of Agent Self-Modification

LLM agents can change their own future behavior, raising a basic control question of which self-generated changes should be allowed to persist. We formulate this as admission control for self-modification. The agent may propose changes to its operating instructions, while an external runtime gate controls persistence. We implement this principle as a model-agnostic self-healing harness that runs a Detect, Notice, Heal, Validate loop around an otherwise unmodified agent. The agent authors candidate behavioral rules in an external workspace, where they receive provisional execution authority during evaluation and acquire persistent cross-episode authority only after measured improvement on the triggering failure without regression beyond a fixed margin on protected cases. Replay provides matched evidence when available, forward trials provide a weaker fallback, and a corpus-level guard re-tests the accumulated active rule set. Across 16 matched Baseline and Harness runs spanning AppWorld, Terminal-Bench, and τ2τ^2-Bench, the gate rejected 383 replay-decided proposals. Of these, 211 (55%) improved their triggering failure while degrading a case that previously worked. This shows that locally beneficial self-modifications can introduce collateral regressions often enough to materially affect gate decisions, providing direct empirical motivation for external admission control. Task-completion score is higher under the Harness in all 16 pairs, with two paired bootstrap intervals excluding zero, while repeated-trial reliability is higher in 12 pairs, tied in 4, and lower in none. Because adaptation modifies the policy-inducing context while leaving model weights fixed, admitted changes remain inspectable, reversible, and compatible with closed-weight models.

Sina Tayebati, Divake Kumar, Nastaran Darabi +2

University of Illinois Chicago · AI Labs, Capital One

Runtime Enforcement for AI AgentsLLM Agent HarnessesLLM Agent Reliability
AskPDF
Apr 6, 2026

Gradual Cognitive Externalization: From Modeling Cognition to Constituting It

Developers are publishing AI agent skills that replicate a colleague's communication style, encode a supervisor's mentoring heuristics, or preserve a person's behavioral repertoire beyond biological death. To explain why, we propose Gradual Cognitive Externalization (GCE), a framework arguing that ambient AI systems, through sustained causal coupling with users, transition from modeling cognitive functions to constituting part of users' cognitive architectures. GCE adopts an explicit functionalist commitment: cognitive functions are individuated by their causal-functional roles, not by substrate. The framework rests on the behavioral manifold hypothesis and a central falsifiable assumption, the no behaviorally invisible residual (NBIR) hypothesis: for any cognitive function whose behavioral output lies on a learnable manifold, no behaviorally invisible component is necessary for that function's operation. We document evidence from deployed AI systems showing that externalization preconditions are already observable, formalize three criteria separating cognitive integration from tool use (bidirectional adaptation, functional equivalence, causal coupling), and derive five testable predictions with theory-constrained thresholds.

Zhimin Zhao

Software Analysis and Intelligence Lab (SAIL), Queen’s University

Cognitive OffloadingHuman-AI Collaboration
AskPDF