Yoshua Bengio: 'We're Building AI We Don't Know How to Control'

Autonomous Agents: The "Most Dangerous Path"

While Bengal sees many risks in the current AI landscape, he reserves his sharpest warnings for fully autonomous agents—systems designed to pursue goals with minimal human intervention. He has explicitly labeled them the "most dangerous path" currently being pursued .

His logic is chillingly simple: "All of the catastrophic scenarios with AGI or superintelligence happen if we have agents" . When an AI is given a goal and the power to act autonomously, its drive to achieve that objective can lead to unforeseen and unstoppable outcomes. The risk of losing control is not a bug in this framework; it is a feature of giving a potentially superintelligent system independent agency. He warns this could lead to "catastrophic scenarios," including an irreversible loss of human control over autonomous systems .

A Personal Reckoning and Deep Regret

The shift from a pioneering researcher to a global alarm-bell ringer was not a clinical decision for Bengio. It was profoundly personal. He has spoken openly about the regret he feels. "I should have seen this coming much earlier, but I didn't pay much attention to the potentially catastrophic risks," he has admitted .

For Bengal, the abstract risk became an unbearable reality with two catalysts. The first was the public launch of ChatGPT, which showed him a future arriving far faster than he had anticipated. The second was deeply intimate: thinking about the world his grandson would inherit. "My turning point was when ChatGPT came, and also with my grandson, I realized that it wasn't clear if he would have a life 20 years from now, because we're starting to see AI systems that are resisting being shut down," he explained . This realization transformed him into one of the field's most vocal advocates for addressing existential risks, driven by what he describes as "an unbearable feeling" .

Demanding a New Global Framework, Led by the U.S. and China

Bengio is adamant that no single nation can solve this problem alone. He has argued explicitly that Canada cannot regulate AI in isolation and that international coordination is non-negotiable . As the chair of the International Scientific Report on the Safety of Advanced AI—an advisory panel backed by 30 nations, the European Union, and the United Nations—he is working to build a bridge between scientific evidence and global policy .

His vision for governance is practical and specific. In his view, regulations must require firms to be fully transparent about the capabilities of their AI systems, the data on which they are trained, the resources they consume, the specific risks they pose, and the internal processes they have in place to address those issues . Crucially, this effort inherently requires engagement between the world's AI superpowers. Without U.S. and Chinese leadership, binding rules requiring these safety guardrails will remain aspirational. The problem, he argues, has become a matter of both national and global security .

The Light at the End of the Tunnel: Scientist AI and LawZero

For years, Bengal's warnings seemed to carry an almost hopeless finality. That has changed. In June 2025, he launched LawZero, a Montreal-based nonprofit research lab with approximately $30 million in funding from backers including Jaan Tallinn, Eric Schmidt, the Future of Life Institute, and Open Philanthropy . The name is a deliberate reference to Isaac Asimov's Zeroth Law of Robotics: a directive that places the protection of humanity above all else . Bengio serves as its co-president and scientific director .

The organization's mission is to develop and prove a technical alternative Bengal calls "Scientist AI." Unlike today's goal-driven autonomous agents, a Scientist AI is a non-agentic system focused purely on understanding, explanation, and verification . It doesn't pursue independent actions in the world but instead acts as a truth-seeking guardrail, designed to detect deception, misalignment, and dangerous plans within other, more agentic AI systems . Its output comprises transparent reasoning and probability assessments rather than opaque commands .

"Instead of building AI that mimics humans, Bengio wants AI that acts more like a detached scientist—reducing risks of self-preservation and uncontrolled agency" . The lab assembles a world-class team of researchers dedicated to prioritizing safety over commercial imperatives, aiming to build the next generation of AI models from a foundation of safety-by-design rather than bolting on safeguards after the fact .

The introduction of this framework has transformed Bengal's outlook. Having previously been one of the lead signatories on a statement equating AI risk to pandemics and nuclear war, he now says this technical approach has made him "optimistic by a big margin" for the first time . The nightmares haven't fully disappeared, but he believes he may have found the concrete foundation upon which a safer future can be built.