King Charles III convened senior AI and UK government figures at Dumfries House in Ayrshire on September 17 to discuss how AI can benefit society without slipping beyond human control. The private meeting brought together representatives from Nvidia, Google DeepMind, OpenAI and Anthropic, alongside UK AI minister Kanishka Narayan. Reporting ahead of and around the event described a discussion of shared principles, not a binding regulatory settlement.
1
9
12
What King Charles asked of AI leaders
Charles framed AI as both promising and alarming. He urged participants to ensure the technology remains in the service of people, communities and the planet, while warning of the “existential dangers” of powerful systems falling into the wrong hands and being used catastrophically. His central message was that safeguards should be considered before it is too late.
9
11
13
That framing made the summit an intervention in a live argument about frontier AI: whether rapidly improving systems could outpace the institutions, safety practices and social norms meant to govern them. The meeting’s purpose was to consider how AI could advance human dignity and social benefit while limiting those risks.
1
9
Jensen Huang’s alternative: regulate the application, not AI as a category
Nvidia CEO Jensen Huang did not argue that AI safety is optional. His position is that companies should test systems rigorously and should not release products when they are not confident in their safety. He characterizes this as an engineering responsibility.
18
23
30
Where Huang differs is the target of regulation. He argues that AI is an underlying technology rather than a single consumer product such as social media. In his view, regulators should focus on the products and services built with AI, monitor deployments that go wrong, and add rules to existing statutes where gaps emerge. He pointed to sectors including healthcare, transportation and entertainment as fields that already have regulatory frameworks.
26
28
This is a product- and sector-specific approach:
- assess risk in the concrete use case, such as a healthcare or transport product;
- apply or extend the relevant existing rules when an AI deployment creates a problem; and
- require careful, contained testing before release.
23
28
It places less emphasis on a sweeping technology-wide AI law or a mandated pause in model development.
Why this contrasted with the summit’s precautionary tone
Charles’s warning was concerned with severe misuse and risks that may emerge before a harmful product is easily identifiable. That is a more anticipatory view of governance: safeguards and international cooperation should be in place early enough to prevent catastrophic outcomes.
1
11
Anthropic CEO Dario Amodei had separately called for a slower pace of frontier-model development so safety measures could keep up. Huang pushed back on the idea that new laws or special arrangements to coordinate a slowdown were needed, arguing instead for engineering, testing and existing legal tools.
8
30
The practical disagreement is therefore about where intervention belongs:
| Approach |
Primary focus |
Preferred timing |
| Huang’s approach |
AI-enabled products and specific deployments |
Test before release; act through sector rules when risks appear or gaps are clear |
| Precautionary approach voiced at the summit |
System-wide and potentially catastrophic risks from increasingly capable AI |
Establish stronger safeguards before risks or capabilities outpace controls |
Both perspectives support safety and human oversight. But they assign responsibility differently. Huang puts greater weight on developers’ release discipline, engineering standards and regulators of particular products. Charles’s remarks—and the broader push for shared principles—stress that some AI risks may require coordinated guardrails that operate before a specific sector deployment fails.
9
23
28
The takeaway
Dumfries House did not settle the AI-regulation debate. It clarified its fault line: whether AI can be governed mainly through existing product laws and rigorous testing, or whether frontier systems create cross-sector risks that demand new, preventive coordination. The summit’s common objective was straightforward—AI should serve humanity and remain under human control—but the path to that objective remains contested.
9
12
28