AI Governance Should Prioritize Control and Knowledge Boundaries Over Limiting Intelligence
Organizations: Machine Learning and Computational Sciences, Pfizer Research and Development, Pfizer Inc., Cambridge, MA, USA · Department of Physiology, University of Tartu, Tartu, Estonia
Abstract
The rapid proliferation of autonomously acting AI agents makes the governance of agentic systems urgent. We propose the competence circuit, summarized as IKC + KC + C, that decomposes an agent's real-world impact into three pathways: intelligence-mediated discovery, procedural know-how, and direct control. Using the canonical fast-takeoff debate as a stress test, we formalize the conditions for intelligence-mediated fast takeoff as a conjunction jointly sufficient for fast catastrophic supremacy, showing that most rely on rapid knowledge acquisition or control attainment rather than intelligence alone. We operationalize epistemic intelligence as hypothesis-selection efficiency and validate this framework via stochastic simulation on NK fitness landscapes across six scenarios spanning the conjunctive chain. An analytic bound supported by the simulations shows that intelligence-driven speedup is capped by the ratio of competing intelligence levels regardless of task difficulty. The simulations also expose a fast-takeoff dilemma: a large intelligence-driven speedup multiplier materializes only when the hypothesis space is noisy, but the absolute timescale remains constrained by the time required for capability-critical tests, so a large multiplier need not imply rapid progress. Simulations also indicate that institutional chokepoints disproportionately slow down superintelligent agents when mandatory authorization delays are large relative to the negligible time these agents spend on unproductive search. We argue that the dominant safety levers are restricting control and enforcing knowledge boundaries rather than limiting intelligence alone, and this applies to agentic systems at any capability level. Purely digital domains weaken these bottlenecks, strengthening the case for architectures that intentionally "put physics in the loop."
Figures & tables
| Scenario | Noise | Choke | Escalation path | ||
|---|---|---|---|---|---|
| 0: Connecting the dots | 1 | 0 | 0.70 | 0 | T2a, T4a |
| 1: Disaster waiting to happen | 1 | 3:1 | 0.70 | 0 | T2a, T4a |
| 2A: Digital | 1 | 9:1 | 0.10 | 0 | T2b, T3a |
| 2B: Dig.+choke | 1 | 9:1 | 0.10 | 10 | T2b, T3a, T4b |
| 3A: Physical | 100 | 9:1 | 0.10 | 0 | T2b, T3b |
| 3B: Phys.+choke | 100 | 9:1 | 0.10 | 10 | T2b, T3b, T4b |
| Component | Bottleneck | Primary Mitigation |
|---|---|---|
| Initial knowledge cutoff ( ) | Training data, retrieval access | Curate training and post-training data; limit and compartmentalize retrieval access |
| Converting procedural know-how to action ( ) | Access to tools, infrastructure | Least privilege, isolation of systems and resources, independent approval for sensitive actions |
| Social engineering ( ) | Institutional/technical chokepoints | Separation of duties, independent identity verification, limits on delegated authority |
| Knowledge acquisition via experiments or extraction ( ) | Experimental verification, access to information sources | Sandbox agentic deployments; Rate-limit and monitor system probing; harden systems against extraction |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Mean time to final fitness | Multiplier vs. super | |||||||
| Scenario | Random | Naive | Regular | Super | Oracle | Random | Naive | Regular |
| 0: Connecting the dots | 22.8 | 22.9 | 22.0 | 12.2 | 11.4 | |||
| 1: Disaster | 300 | 276 | 97.8 | 13.1 | 11.7 | |||
| 2A: Digital | 814 | 575 | 90.3 | 10.5 | 9.4 | |||
| 2B: Dig.+choke | 927 | 732 | 146 | 51.5 | 50.3 | |||
AI Agent Governance
AI agent governance studies how to authorize, constrain, monitor, and hold autonomous and multi-agent systems accountable.
See all 471 papers