# Autopoiesis: Why AI Governance Keeps Failing to Stick

Every major AI governance effort of the last five years has followed the same disappointing arc: ambitious framework, earnest implementation, system continues largely unchanged. The EU AI Act, algorithmic auditing requirements, GDPR’s application to ad-tech: each has produced genuine friction, some fines, a lot of compliance documentation, and very little structural change to how AI systems actually behave. The question isn’t why regulators are incompetent. It’s why the systems they’re regulating are so much better at absorbing interventions than regulators are at making them stick.

The answer was worked out in 1972, by two Chilean biologists who weren’t thinking about AI at all.

—

## The Self-Producing System

Humberto Maturana and Francisco Varela coined *autopoiesis*, Greek for “self-creation”, to describe a specific property of living systems: they continuously produce and reproduce the conditions of their own existence. A cell doesn’t just process inputs; it produces its own membrane, its own metabolic components, its own organisational structure. The system’s outputs are the inputs to the system’s continued operation.

The first structural property is *operational closure*: an autopoietic system is autonomous, processing what happens to it only in its own terms. It doesn’t receive instructions from the outside; it receives perturbations that its internal logic interprets and responds to. The second is *structural coupling*: the system is embedded in an environment and sensitive to it, yet environmental events trigger internal structural adjustments rather than direct changes to the system’s organisation. A virus can destroy a cell; it cannot reprogram what the cell is.

In 1984, the German sociologist Niklas Luhmann extended this framework from biology to social systems. Organisations, markets, legal systems: all, Luhmann argued, are operationally closed systems that reproduce themselves through their own characteristic communications. The legal system processes events as legal or illegal; the economic system as paid or unpaid. External events that don’t fit these codes aren’t ignored; they’re re-encoded into the system’s own language and responded to accordingly. A law doesn’t enter the economy as a law; it enters as a cost, a risk, a price signal.

What Luhmann couldn’t have anticipated is how precisely this describes what AI-enabled organisational systems are now doing.

—

## How AI Systems Became Autopoietic

A social media recommendation algorithm seems, at first glance, like a tool: a piece of software that serves some organisational goal. Look more carefully and it resembles a living system. The algorithm is trained on behavioural data. Its outputs (ranked content) shape user engagement patterns. Those engagement patterns generate new behavioural data. That data trains the next model iteration. The system’s outputs are the raw material of its continued operation.

This is not a metaphor. RLHF (reinforcement learning from human feedback), the technique behind most current large language models, is precisely this loop: models generate outputs, human evaluators judge those outputs, a reward model encodes those judgements, and the reward model trains the next model iteration. The system is, in a meaningful sense, producing the conditions of its own existence.

Network effects extend this self-production into the social environment. The more people use a platform, the more valuable it becomes; the more valuable it becomes, the higher the switching costs; the higher the switching costs, the more entrenched the behavioural data collection that trains the next models. LinkedIn’s professional identity norms are shaped by what LinkedIn’s algorithm has historically rewarded. Those norms now generate the data LinkedIn uses to refine its algorithm. The loop closes.

The dynamics are structural, not conspiratorial. That’s why the standard governance toolkit consistently fails.

—

## Why Auditing Doesn’t Work

Algorithmic auditing (inspecting a system’s outputs for bias, harm, or policy violations) has become the dominant mechanism in AI governance frameworks. The EU AI Act mandates it for high-risk AI. The Digital Services Act requires platform transparency. The NIST AI Risk Management Framework treats audit as a foundational practice.

The problem is that auditing inspects outputs without touching the self-producing dynamics. In Maturana and Varela’s terms, it’s an attempt to regulate what a system *produces* without engaging the processes through which it *produces itself*. The audited system responds to audit requirements as it responds to all environmental perturbations: it re-encodes them into its own operational logic. Compliance documentation proliferates. Consent banners appear. Bias dashboards are published. The underlying training loops, the data-collection incentives, the network effects that make switching costly: these continue exactly as before.

GDPR’s collision with the ad-tech ecosystem is the clearest case study. More than EUR 4 billion in fines have been levied since enforcement began. The advertising industry adapted: contextual advertising resurged, consent management platforms were deployed, compliance teams were hired in their hundreds. But the fundamental architecture of behavioural surveillance proved structurally resilient: real-time bidding systems, cookie-equivalent fingerprinting, data broker pipelines, all intact. Researchers have found that RTB’s data-sharing model makes genuine GDPR compliance “virtually impossible,” yet the ecosystem continues. GDPR perturbed ad-tech; it did not transform it. The system re-encoded the regulation as a compliance cost and kept producing itself.

The EU AI Act is following a similar trajectory. When 45 major European companies (Airbus, Siemens, Mistral AI among them) sent an open “Stop the Clock” letter to the European Commission in 2025, they weren’t simply lobbying against inconvenient rules. They were doing what autopoietic systems do: processing an environmental perturbation (regulation) through their own operational logic (business risk, competitive positioning) and generating a response (industry pressure) that sought to adjust the regulatory environment to their own requirements. The Commission opened a “fitness check” and signalled possible amendments. The system pushed back, and the environment shifted.

—

## What Intervention Actually Requires

If external regulation gets absorbed as perturbation, the implication isn’t that governance is hopeless. Effective governance requires intervening at a different level.

The first lever is regulating conditions of self-production, not outputs. GDPR’s core error was attempting to change what data gets *used* while leaving intact the system that *collects* it. Effective data governance needs to constrain the collection loop itself: not at the processing stage, but at the point of generation. Data minimisation requirements mean little if data generation is unconstrained. Structural interventions act on the raw material of the self-producing loop, not its products.

The second is disrupting structural coupling rather than system behaviour. An autopoietic system responds to its environment through structural coupling: it is sensitive to the medium it is embedded in, even if it cannot be directly programmed by it. Changing what the system is coupled to changes the system more reliably than changing what it produces. Mandatory interoperability requirements alter the network-effect dynamics that make AI platform ecosystems self-sustaining. Open training data requirements alter the data monopoly that makes self-referential improvement loops proprietary. These are coupling-level interventions. They don’t just create friction; they change the conditions of self-production.

The third is targeting attractors, not outputs. In systems terms, an attractor is the state a system tends to return to after perturbation. Social media recommendation systems have a powerful one: engagement-maximising content, regardless of platform policies that formally prohibit it. Simply correcting individual outputs (content moderation, labelling requirements) leaves the attractor in place. The system returns. Effective intervention identifies the attractor and acts on the dynamics that produce it. In the case of social media, that means the engagement metric itself. Platforms that have optimised for time-well-spent rather than time-on-site have demonstrated that the attractor can shift, but only by changing the objective function, not by patching the outputs.

—

## The Governance Implication

Maturana and Varela described autopoiesis as what made living systems *living*: capable of maintaining themselves against entropy, capable of continuing to exist as organised wholes. AI-enabled ecosystems now have something analogous to this property. They are not passive tools waiting to be steered. They are self-maintaining systems that will re-encode governance interventions into their own operational logic unless those interventions act on the dynamics of self-production itself.

Leaders who understand this will stop asking “how do we make this system comply?” and start asking “what is this system producing itself from, and how do we act on that?” The organisations that build AI governance frameworks answering the second question will find interventions that stick. Those still asking the first will keep generating impressive compliance documentation while the system continues unchanged.

Autopoietic systems can be redirected. But only by those who understand what kind of system they’re dealing with.