As AI agents turn increasingly capable, the need for human oversight, also known as “human-in-the-loop”, has been reinforced by policymakers and tech/AI juggernauts alike, however a new study by Hugging Face suggests critical degradation of human operator’s skills and situational awareness as they supervise AI models.

The paper titled “AI Agents Push Humans Out of the Loop”, co-authored by researchers Margaret Mitchell and Avijit Ghosh of Hugging Face and Samir Passi of Data and Society, has argued that simply keeping a “human in the loop” is not sufficient to ensure safe deployment of increasingly autonomous AI agents as their growing use could weaken the ability of humans to effectively supervise them.

The report observes that prolonged interaction with such systems may lead to cognitive fatigue, skill loss and overreliance, according to a new position paper by researchers from Hugging Face and Data & Society.

“The more capable an automated system is, the more the human operator’s skills and situation awareness degrade, over trusting the systems and ultimately leading to situations where people are least prepared to help when it’s most needed…the very act of being an overseer degrades the capacities oversight requires: Oversight degrades the overseer,” read an excerpt from the report.

Recently, frontier AI models of leading companies like OpenAI, Anthropic and Meta have shown an alarming rise in their cybersecurity capabilities with incidents of breaching third party organizations, social engineering and phishing acts also reported. Most notably, OpenAI’s AI agents’ hacking Hugging Face platform had led to increasing demands for an AI Kill Switch, need for safety rails and increased presence of human overseers through human-in-the-loop process.

However, the report argues that “without explicit support for the cognitive demands of effective human-agent interaction, AI agent systems will continue to passively incentivize the degradation of the very human skills they rely on,”.

The study also states that although human oversight should have been a joint effort between AI builders, deployers and users, the onus of oversight currently falls entirely on users.

“Addressing this intellectual blind spot is urgent. At the same time that AI agent systems have increasingly been deployed at scale, self-reports have documented the difficulty in maintaining active engagement as an overseer, and research across multiple domains has uncovered serious negative impacts on critical thinking skills resulting from extended use of automated systems,” read the report.

In order to address this issue, the researchers have advocated for a two-pronged solution of cognitive scaffolding, provided by developers and deployers. Developers must create and implement processes for AI agent runtimes that support the cognitive requirements of effective oversight, including strategic points of friction and interfaces that maintain and/or help regain users’ situational awareness during and after agent operation.

Deployers must also institute organizational protocols that stimulate critical engagement, including trainings on recognizing signs of fatigue, rotation policies that safeguard oversight attention, and processes that preserve domain skill.

Pic Credit: Yasmin Dwiputri & Data Hazards Project, Better Images of AI

Also Read: AI Mind Viruses Could Push Agents to Pursue Harmful Goals, Study Finds