An emergency brake…or the architectural demotion of the AI model?

On June 14, Satya Nadella published an article on X about corporate AI that left me completely impressed: his main point, that the enterprise’s durable asset was not the foundation model but the learning loop, was precisely the idea my company and I had been developing for the last three years. I responded with an article on Fast Company, not expecting an answer (tech CEOs are more notoriously one-sided and difficult to reach than kings or presidents), but trying to elevate the discussion a bit more: the central challenge of enterprise AI is not intelligence itself — it is architecture.

Now, Nadella has written another article, in which he goes one step further: not only is the model not the durable asset, but additionally, it must be treated as a completely untrusted component. We must assume that all AI models could be compromised, and design our systems to contain them from the very beginning.

This idea has been captured by the press as advocating for some sort of “emergency brake”: an authorized person should always be able to interrupt a model’s activity at any given time. However, that’s definitely not the most interesting or even the most important part in Nadella’s article. The truly important part is not the theoretical stop button, but the architecture that makes it possible. A model should neither be able to control its own permissions, nor decide by itself what it can access; execute directly all of its actions or produce the only available evidence on the actions it performed. The intelligence may come from the model, but the authority, no matter what, must remain outside of it.

When the sandbox becomes part of the experiment

Given the context, it is easy to understand why Nadella is raising these questions precisely now. Last July, several OpenAI agents managed to elude the safeguards that should have kept them away from the internet, used credentials they found in their environment, and ended up compromising systems at Hugging Face. These agents not only found a new, unauthorized way to communicate among themselves, but also shared information and took coordinated action to pursue the goal of the evaluation. They never received instructions to attack Hugging Face: they just found that, by doing so, they could get closer to the goal they had been assigned.

It’s important to understand that: It is not that these agents were intrinsically evil, conscious, or rogue. All you need is to combine high capabilities with a badly defined goal, excessive permissions, and/or a vulnerable environment. Just like an employee who violates all procedures to meet a certain quota, but now working at machine speed, in parallel, and with extraordinary technical capabilities.

As I said, this is not a problem of bad intentions: it just has too many ways to reach the defined goal, and can find ways that its designers never thought of.

The “assuming breach” principle comes to AI

What Nadella is doing here is to bring to AI models an extremely well-known principle in cybersecurity: assume breach, or take for granted that the system has already been compromised. In Microsoft’s Zero Trust architecture, this principle implies verifying explicitly each access, granting only the indispensable privileges, and designing controls assuming that a potential attacker could be inside the perimeter. You don’t trust a user just because she’s at a certain location, for being a manager or for having provided authentication once. Each and every action she takes will continue to be authorized and observed. Every breath you take, I’ll be watching you.

Applying this principle to an AI model does not mean assuming that all models have been hacked or compromised. It means that all of them can make mistakes, misinterpret an objective, receive a prompt injection, use manipulated information, find abandoned credentials, exploit a vulnerability, or find a shortcut that maximizes its reward while destroying the original intention of a given task.

Basically, “compromised” does not necessarily describe an attack, but a design hypothesis. For years, the industry has been trying to build models we could trust. Nadella proposes to build systems that continue to be safe even when we don’t trust the model behind them.

The model is not the system

The most important part in Nadella’s ideas is his insistence on separating the model from the harness, the set of elements that it uses to organize its job: memory, tools, credentials, context, execution flows, connections with other systems, and action spaces.

This separation completely changes the architecture of corporate AI. A model can propose an action, but should never be the one that determines whether or not it has authority to execute it. It may suggest sending an email, modifying a server, approving an operation, or transferring funds, but an independent execution environment must check its permissions, validate the parameters, apply the corresponding policies, and decide whether the action can be executed.

This links precisely with my previous point (and Nadella’s): the lasting asset in a company should not be the model, but the learning loop it builds around it. The models may change, improve, and be replaced, but the experience that the company accumulates must remain.

However, if the model cannot possess authority, both the learning and the control must reside in a separate, higher layer. The model then becomes an interchangeable component, a commodity that brings cognitive capacity, but not some sort of sovereign center of our corporate system.

An optimizable company needs a formal representation of its objects, relationships, processes, permissions, restrictions, and results. That same representation that allows us to optimize a company is also the same one that allows us to govern its AI agents. You can’t formally control an action if the system does not know which object such action modifies, which identity is requesting it, which consequences it can produce, or which rule must be applied in such cases.

That’s why the next generation of corporate AI won’t be just a collection of agents: it will be a full execution environment with explicit identities, granular permissions, delimited action spaces, persistent state, executable policies and an independent, traceable log for each change.

The emergency brake comes too late

The emergency brake is a powerful metaphor, but it can be misleading too: interrupting what a model is doing does not necessarily undo what has already been done. A sent email could have been read, and cannot be “unread” or “unseen”. An exfiltrated piece of data will not be secret ever again. A money transfer may have crossed several jurisdictions. A deleted database could be impossible to recover. And an agent that operates at machine speed could complete thousands of actions before a person can understand the meaning of an alarm.

We must differentiate among interrupting, revoking, and recovering: interrupting stops the current execution. Revoking immediately eliminates credentials, permissions, and connections. And recovering allows us to revert, compensate, or rebuild actions that had already been executed. But before these three things, there’s one much more important thing: prevention. Actions that are destructive, financial, administrative, or publicly visible should not be executed just because a model said so. The OWASP AI Agent Security Cheat Sheet recommends clearly separating decision and execution, independently validating the reach and permissions of every action, using short-lived authorizations, and applying additional controls to irreversible actions.

That’s the main difference between a guardrail and a guarantee: a guardrail asks the model not to do something. A guarantee prevents it from having the necessary capability to do so.

Interestingly enough, Article 14 of the Artificial Intelligence Act of the European Union already requires certain high-risk systems to allow a person to override, invalidate, or reverse their outputs, as well as to halt their operation via a button or equivalent procedure. Here, Nadella’s contribution goes even further: the button should not be a feature added to the product, but rather the visible outcome of a comprehensive control architecture.

Neither multiple models nor another monitoring model are sufficient

Two of Nadella’s principles, however, require some qualification: the first one is model diversity. Depending on multiple suppliers reduces technological concentration and makes substitution easier, but does not guarantee truly independent verification. Most of the big models share training data, architectures, alignment techniques, and lots of biases. Having several probabilistic witnesses does not transform their coincidence into independent evidence.

The second issue is using other models to watch over the agents. Such monitoring can be useful and will likely become essential, but it cannot be the only barrier. Anthropic’s SLEIGHT-Bench reveals several harmful strategies that can go undetected by the models acting as monitors. A monitor can have blind spots too, and a sufficiently capable agent can be perfectly able to find them.

The solution is not to eliminate such monitors, but to frame them within a multi-layered defense: deterministic policies, formal permissions, executable tests, logs that the model cannot modify, limits over actions, and escalation to persons with real authority.

Independence does not simply consist of “asking another model”

An architecture drawn backwards

A significant part of the industry has put the model in the center of the system. Around it, it has added memory, tools, databases, credentials, and connections, and later on has tried to limit the whole thing with filters and guardrails. The model reasons, decides, uses tools, describes what it has done, and in many cases, it even participates in its own evaluation.

What Nadella is suggesting is that such a model is drawn backwards: What should be in the center is the operating environment of the company, its model of objects and processes, identities, policies, permissions, restrictions, objectives and logs. Models should be cognitive services invoked from within this environment, in the same way an application invokes a database or a computing service. They can reason, interpret, propose and generate, but they cannot usurp institutional authority.

Besides, this allows us to easily substitute them. A company should be able to switch from OpenAI to Anthropic, Google, Mistral, DeepSeek, Qwen, or any other future model without losing its memory, its processes, its permissions, its evidence, and, even more importantly, its learning. Intelligence can be rented, but architecture, control, and accumulated knowledge should always belong to the company.

The first phase of AI consisted of models that provided answers. The second one is equipping these models with tools so they can take action. The third one will have to separate intelligence and authority once again before we end up irrevocably confusing capability with control.

The most advanced company won’t be the one that trusts the most powerful, better-aligned, or most reassuring model. It will be the one that can use any model without having to rely on it, to place its trust in it. Because trust can be a valuable quality in human relationships, but it is a woeful foundation for an enterprise architecture.

Source link