Satya Nadella calls for an emergency brake on advanced AI models
IA en un minuto newsroom · Editor: Jon Elgezabal
In 30 seconds
Satya Nadella, who runs Microsoft, argues that any advanced AI system must be stoppable partway through a job by someone with the authority to do so. Nadella wants these models watched the way a company watches anyone holding inside access to its systems, with limits that do not depend on the model itself. The article, posted on X, sets out seven principles and announces no products or commitments from Microsoft.
Satya Nadella, chief executive of Microsoft, called on Saturday, October 10, for advanced AI systems to have an "emergency brake": an authorized person should always be able to pause or shut down a model mid-task. He made the case in an article posted on X titled "Models as Insider Risks in the Super Intelligence Era".
Nadella starts from a difference with traditional software, where a behavior could be traced to a specific code path. With today's systems, he writes, what a model does cannot be attributed to particular training data or to a configuration of its weights, and yet they are being deployed with access to the most sensitive data and with the ability to take mission-critical actions.
His proposal is to treat the most advanced models, closed or open weight, as insider risks. Not because they are necessarily malicious, he explains, but because any sufficiently capable actor with access to important systems can make mistakes or be compromised. He recalls that companies have spent decades refining practices for that kind of actor: establishing identity, limiting privileges, logging activity and creating containment boundaries.
Nadella argues that the assurances of a model provider do not relieve those who deploy the model of responsibility, and that the controls over what a model can access and do must sit outside the model itself. He calls transparency of the model's reasoning non-negotiable but warns that it is not sufficient on its own, and that using models to check one another can end in what he calls nested black boxes.
He lists seven principles: no single model as the sole dependency for an important outcome or as the verifier of its own work; tamper-proof, human-readable evidence of every meaningful action; continuous testing of the entire system, including failures and attacks; each organization being able to determine independently what a model can access and do; auditing that is independent of the system being audited; containment; and timely disclosure of incidents to those affected, sharing what went wrong so the whole industry can learn. The brake image comes under containment: "We must assume a model is compromised and contain it from the start," he writes, adding that more advanced models will require more advanced containment technologies, which will need to be standardized.
The text is an opinion piece, not an announcement: it mentions no products, dates or commitments from Microsoft. According to CNBC, it comes amid warnings from tech executives and researchers, among them Microsoft co-founder Bill Gates, Anthropic chief executive Dario Amodei, OpenAI's Sam Altman and SpaceX's Elon Musk, about insufficient AI safety protocols and claims that the technology is moving too fast.
Why it matters · analysis and opinion
What stands out is who signs it: the chief executive of Microsoft argues that trusting whoever builds the model is not enough. The message is aimed at any organization that is giving an AI system access to its most sensitive data or the ability to act on its behalf, and it translates into very concrete questions to ask before deployment: who can stop it and how, where what it does gets recorded, what permissions it has and who decides them, and who is told when it fails. Much of this is not new: these are the practices companies have applied for decades to anyone with access to important systems. It remains to be seen whether Microsoft carries this into its own products, because the article promises nothing.
Source: Satya Nadella (X) · Written with the help of AI: how we make the news
