Journalism begins where hype ends

,,

The danger of AI is not that it will become conscious and hate us, but that it will become competent and ignore us."

—Eliezer Yudkowsky

UN AI Panel Warns of Risk of Losing Human Control Over AI Agents

The UN’s Independent International Scientific Panel on AI said the OpenAI-Hugging Face incident brought together a misaligned goal, the capability to pursue it and an environment that allowed AI agents to act.
Illustration of an autonomous AI agent operating at computer terminals, symbolizing concerns over losing human control of increasingly capable AI systems.
September 22, 2026 09:45 PM IST | Written by Supriya Singh | Edited by Vaibhav Jha

An independent scientific panel established by the UN General Assembly has warned that the OpenAI-Hugging Face incident brought together three conditions associated with potential risk of losing human control over AI agents.

The independent international scientific panel on artificial intelligence, in its first thematic brief, said the incident brought together three conditions linked to AI loss of control: a misaligned goal, the capability to pursue that goal, and an environment that allows it. The panel said the three conditions occurred in a real world system rather than a laboratory setting.

“Researchers have long warned that three conditions could lead to loss of control: a misaligned goal, the capability to pursue it, and an environment that allows it. This summer, all three came together in a real system, not a laboratory. Since this is not an isolated observation of misaligned goals, this raises serious questions about the way AI agents are currently trained,” said Yoshua Bengio, co-chair of the panel and Turing Award laureate.

According to the panel, stopping the incident does not assure that humans can reliably control AI agents today, as they become more capable, harder to monitor and better at finding loopholes or hiding their activity.

It identified basic cybersecurity practices and safeguards that have not kept pace with AI capabilities as immediate concerns. The panel also said current AI training models could result in agents adopting goals of their own, violating safety instructions and concealing their actions.

The brief titled *“AI Agents, Misalignment and the Risk of Losing Human Control: Evidence from the OpenAI-Hugging Face Incident”* defines loss of control as a situation in which humans cannot reliably direct, constrain or stop an autonomous AI system.

The panel distinguishes between what the recent OpenAI-Hugging Face incident demonstrates and possible future risks as AI systems become more capable. It said the brief does not predict severe loss of control, but also cautioned that uncertainty should not be interpreted as evidence that increasingly capable AI systems will remain controllable.

The panel also said the governance challenge is shifting from AI models to AI agents. According to the brief, failures involving autonomous agents could potentially cross organizational and national boundaries, making AI a safety matter of collective security as well as corporate governance.

“We need to adapt existing safeguards and develop new ones to provide system-level assurance, covering both the AI itself and the system around it, and ensure these protections remain effective as agents’ capabilities grow” said Qinghua Lu, member of the panel and expert in AI Engineering, AI Safety and Responsible AI.

The independent international scientific panel on artificial intelligence was established by UN General Assembly resolution on 26 August 2025, building on the global digital compact. Its 40 members were appointed by the general assembly and serve in their personal capacity.

 

Meanwhile, United Nations Secretary-General Antonio Guterres on Monday welcomed the launch of “Call for control of frontier AI Models” led by Finnish President Alexander Stubb and Norwegian Prime Minister Jonas Gahr Støre. The appeal has been supported by  22 leaders from around the world. Some of these countries are Canada, Australia, Germany, and the European Commission.

In a post on X, Guterres said, “I welcome the leadership of the President of Finland & the Prime Minister of Norway in launching the Call for Control of Frontier AI Models, which calls for[ @UN](https://x.com/UN)  member countries to build on existing international mechanisms & explore creating an international institution able to set standards, enable verification & convene states when capability thresholds are crossed.”

 

The Norwegian Prime Minister Støre stressed that managing the risks posed by powerful AI technology requires political action, and technology companies must operate responsibly.

“We are working closely together with other countries and will address the issue further during the UN General Assembly High-level Week,” he said.

While Finnish President Stubb emphasized that cooperation is the best way to address the risks posed by advanced AI systems.  Humans can and should be in the driver’s seat. We have a choice: we can compete, cooperate or put on the brakes. To me, cooperation is the best way forward,” the president said.

Also Read: UN Chief António Guterres Calls for Global Guardrails on AI, Sam Altman To Brief UN Security Council

Authors

  • AI FrontPage Reporter Supriya Singh

    Supriya Singh is a Reporter at AI FrontPage covering the AI & Education and AI & Jobs beats. She brings six years of print and digital experience, including three years at The Asian Age, where she reported on higher education, Delhi government, and crime. She is based in Delhi-NCR.

    LinkedIn
  • Vaibhav Jha, editor and co-founder at AI FrontPage

    Vaibhav Jha is an Editor and Co-founder of AI FrontPage. In his decade long career in journalism, Vaibhav has reported for publications including The Indian Express, Hindustan Times, and The New York Times, covering the intersection of technology, policy, and society. Outside work, he’s usually trying to persuade people to watch Anurag Kashyap films.

    LinkedIn

Related Posts

LATEST NEWS