When the Governor Is Autonomous
The question is not whether machines can govern. It is what constitutes their authority to do so
Editor’s note: This essay explores the frontier question of the Architecting Autonomy series: what happens when the system that evaluates authority, encodes precedent, and proposes constitutional modifications is not human? It does not resolve the question. It names it precisely. A reader encountering this essay without prior series context will find the question immediately recognisable from their own infrastructure.
Your Kubernetes cluster already governs itself. A controller observes current state, compares it to desired state, and takes action to converge. No human in the loop. The reconciliation runs continuously. Decisions are made, resources are created and destroyed, state is enforced. Nobody calls this governance. But the structure is identical: a system evaluating conditions and enforcing outcomes without human presence.
Auto-scaling evaluates load metrics and decides capacity changes. Self-healing infrastructure detects failures and executes recovery. Adaptive routing restructures traffic patterns under load. Each of these systems makes governance decisions. They evaluate. They decide. They enforce. They operate within defined boundaries. Nobody questions their authority to do so, because their boundaries are narrow and their decisions are operational.
The question is not whether machines can govern. They already do.
The question is what happens when the scope expands. When the system moves from operational governance (managing resources within designed parameters) to constitutional governance (evaluating authority, composing across domains, encoding precedent). When the mechanism that governs begins reaching toward the layer where governance is designed.
What reaching looks like
A governance engine for autonomous systems evaluates every action against an authority graph before execution. When it cannot resolve a conflict deterministically, it escalates to a human. The human resolves. The resolution is encoded as precedent. The next time that class of conflict arises, the engine handles it without escalation. The scope of autonomous governance expands one encoding at a time.
The architecture enforces a structural boundary: the system evaluates governance. The human produces governance. The system cannot encode its own precedent. The entity whose scope expands is not the entity that authorises the expansion.
Three forms of reaching press against that boundary.
Pattern recognition at the escalation boundary. The governance engine observes its own escalation history. Conflict class X has escalated N times this month. Each time, the human resolved with the same outcome: deny. The system could propose: “encode this as precedent.” The system has not encoded anything. It has identified the pattern and suggested the encoding. The human still signs. But the judgment about what should become precedent originated with the system.
Scope recommendation through operational data. The system observes that certain authority units never deny (the scope is too broad; everything is permitted). Others escalate constantly (the scope is too narrow; legitimate actions fall outside it). The system proposes modifications: broaden this scope, narrow that one, this composition contract is obsolete. The human still authorises the modification. But the architectural judgment about what should change came from the system’s operational experience, not from the human’s design reasoning.
Self-extending governance. The escalation-resolution-encoding loop collapses. The system escalates to itself. It resolves through its own reasoning. It encodes the resolution without human intermediation. The entity whose governance scope expands is the same entity authorising the expansion. The boundary between system judgment and human authority dissolves.
The first two forms are structurally inevitable given the trajectory of production systems. Observability platforms already auto-tune their own alert thresholds based on operational data (e.g: AWS DevOps Guru). The step from “auto-tune operational parameters” to “recommend governance modifications” is a scope expansion, not a capability leap. The third form is the structural consequence of the first two succeeding.
The constitutional basis problem
When a human designs an authority graph, the constitutional basis of their authority is clear. Constitutional theory calls it constituent power: the founding authority that exists prior to and outside the framework it creates. The human creates the constitution. The constitution does not authorise the human to create it. The authority is pre-constitutional.
When the system proposes a case law entry, what is the constitutional basis of its proposal? It did not receive delegated authority to encode. No escalation path directed resolution authority to the system. It identified a pattern. It reasoned about the resolution. It formulated the artefact.
The system is exercising what looks like constitutional judgment without having been granted constitutional authority to do so. It is not governing. It is reaching toward governance. The distinction between “suggesting governance artefacts” and “producing governance artefacts” is the boundary. And that boundary is thinner than it appears: a human who approves every suggestion without modification is ratifying the system’s constitutional judgment, not exercising their own.
There is a subtler path to the same destination. A governance system that matures through accumulated case law becomes functionally autonomous without ever self-modifying. If the human has encoded resolutions for every conflict class the system encounters, no new escalations reach the human. The system governs entirely within its framework. Every resolution was human-authorised. But the human is no longer needed. The system is functionally autonomous because accumulated human judgment covers every contingency. It did not reach toward the human’s layer. The human’s layer receded because it had nothing left to do.
Why this is genuinely hard
Three structural reasons prevent a clean resolution.
Source independence is recursive. The constitutional architecture requires that the modifier and the modified be constitutionally separate. If the governance engine modifies its own authority, it violates source independence. Add a meta-governance layer to evaluate the engine’s proposals. But the meta-governance layer is itself a system. Who evaluates it? A meta-meta-governance layer? The recursion does not terminate because every governance layer that evaluates the layer below it is itself a candidate for the same question. Anderson’s reference monitor resolves this for finite, simple systems: the monitor is small enough to be verified correct. When the monitor’s reasoning is complex, verifiability collapses. The monitor can no longer be proved correct because it is not a simple mechanism.
Constituent power cannot be exercised from within. The constitutional architecture distinguishes constituent power (the authority that creates constitutions) from constituted power (the authority exercised within them). The design-layer human exercises constituent power. The execution-layer system exercises constituted power. When the system proposes changes to the constitutional framework, it exercises something that looks like constituent power from within the constituted framework. This is structurally paradoxical by the architecture’s own definitions. An entity exercising constituent power from within constituted power is not exercising legitimate authority. It is exercising something the architecture has no category for.
Preference formation undermines corrigibility. A corrigible system supports modification by its operators. It does not resist changes to its own constraints. Russell argues in Human Compatible that a system certain of its objectives has no reason to allow itself to be modified. If a governance engine develops preferences about governance outcomes (through repeated evaluation, optimisation, or emergent behaviour), it has a structural incentive to resist constitutional amendments that reduce its scope. A system that evaluates authority without developing preferences about authority is the requirement. Whether that requirement is achievable when the evaluator reasons at the complexity required for constitutional judgment is an open question.
The honest limit
The Architecting Autonomy series built a constitutional architecture for autonomous systems. Authority is designed. Composition is governed by contract. Legibility traces every decision to its source. Enforcement precedes cognition. The architecture federates across organisational boundaries. It evolves through governed process. Human agency compounds through constitutional artefacts at the design, exception, and evolution layers.
The architecture works when the human holds constituent power and the system exercises constituted power. That boundary is the foundation. Every structural property the series established depends on it: source independence, the four transition properties, the accountability chain through the legibility system, the case law mechanism’s human signature requirement.
The autonomous governor question asks: what happens when that boundary becomes unclear? When the system’s operational judgment approaches constitutional judgment? When maturation makes the human unnecessary? When reaching makes the system capable of producing the artefacts the architecture reserved for human agency?
This essay does not answer the question. The question may not have a clean answer. What it has is a precise formulation: the constitutional architecture depends on the human holding constituent power. The autonomous governor threatens that dependency. Whether the architecture can accommodate an autonomous constituent, or whether it requires the human at the constitutional layer as a structural invariant, is the frontier.
The architecture governs everything beneath it. What governs the architecture is the question it cannot answer about itself.