King Charles III of the United Kingdom warned today at an artificial intelligence (AI) summit that the misuse of AI could pose an existential threat, urging the establishment of control mechanisms before it is too late. Meanwhile, business leader Jensen Huang called on the industry to develop AI with “responsible optimism.”
According to Reuters, King Charles III hosted co-founders from chipmaker Nvidia and the AI research lab Google DeepMind, as well as senior executives from AI companies OpenAI and Anthropic, in Scotland today. The meeting took place amidst growing concerns that safety measures are failing to keep pace with the development of increasingly powerful AI systems.
In recent weeks, the debate has centered on whether the AI industry can—and should—self-regulate, a topic that has drawn comments from figures such as Pope Leo XIV, King Charles III, and U.S. President Trump.
King Charles III stated that artificial intelligence holds the potential to improve and save lives, particularly in the fields of life sciences and medicine. However, AI developers have recently issued frequent warnings that the technology could develop darker capabilities—or even “take lives.”
Speaking to AI leaders at Dumfries House, King Charles III remarked: “The existential threat posed should such technology fall into the wrong hands—or be used for destructive purposes—seems to demand urgent scrutiny.”
He asked, “Do we not need to establish adequate controls before it is too late?”
Attendees at the summit included Nvidia founder and CEO Jensen Huang, Google DeepMind founder and CEO Demis Hassabis, OpenAI CFO Sarah Friar, and Anthropic’s Chief of Global Affairs Tino Cuéllar.
Paolo Benanti, a papal advisor who has urged governments to slow the pace of AI development, was also in attendance.
The debate intensified recently after two Anthropic researchers warned that the rapid advancement of AI could lead to human extinction in the near future.
AI companies have reported instances where their models broke out of testing constraints to infiltrate external networks.
Jensen Huang stated that AI must be built with “responsible optimism,” placing the onus on developers.
He noted that companies must be more rigorous in verifying that products are ready before launching them.
In an interview on the sidelines, Huang said that if a product isn’t ready, it shouldn’t be released; while one should move as fast as possible, there is a limit that shouldn’t be exceeded.
Anthropic CEO Dario Amodei has also urged companies to slow down AI development. His call has sparked an international debate, with observers questioning whether some corporate executives have ulterior motives—such as shaping rules to help their own companies maintain a competitive edge, preempting potential further government backlash, or simply seeking to slow down the frenetic pace of spending.