As AI agents move beyond answering questions and begin acting with greater autonomy on behalf of consumers, Stanford’s Human-Centered AI Institute has examined a potential conflict: what happens when the interests of an AI company conflict with the interests of its users?
Stanford researchers Ella Genasci Smith, Victor Y. Wu and Jennifer King, in a new issue brief published August 25, call for a duty of loyalty for developers and deployers of AI agents operating in consumer contexts, particularly those in higher-stakes or regulated areas such as financial services or healthcare.
AI is shifting from reactive generative tools to autonomous agentic systems. Unlike chatbots, which return text and wait for another prompt, AI agents can perform multistep tasks, use external tools and choose actions with minimal human intervention. Some agents require user approval at each step, while others can complete entire workflows without supervision.
Amazon, Google, Anthropic, OpenAI, Perplexity, Meta and Microsoft have embedded proprietary agents into browsers and apps since early 2025. Stanford says these deployments mark a structural shift in how people browse and operate online. The researchers warn that broader access to user data can make agents more effective but also create privacy risks. By combining information across different domains, an agent could infer sensitive details about a user’s physical health, cognitive decline or financial distress that the person never explicitly disclosed.
As agent deployment expands, the researchers raise two questions: Will an agent act in the user’s best interest when its developer’s interests diverge from theirs? And who is liable if the user suffers harm?
“At face value, AI agents appear to serve users, yet they have primarily been designed, built, and maintained by companies pursuing their own commercial interests.”
Agent-collected data could inform targeted advertising, product design or steer users towards a platform’s offerings or those of its commercial partners and away from competitors.
“While the disclosure of conflicting incentives has value, it fails to resolve the underlying problem.”
The researchers say disclosure alone may not work because consumers may not see the options an agent chose not to take or how it reasoned through a decision. Users may also grant an agent enough autonomy to take an action without pausing for approval.
Under the proposed framework, developers and deployers should disclose material commercial relationships that could influence an agent’s decisions. They should also ensure that agents do not steer users towards the developer’s own products, services or commercial partners in ways that materially undermine the user’s interests.
The researchers also say data collected during agent interactions should be used only to carry out the user’s instructions. Other uses, including advertising, model training, third-party sales or the deployer’s own commercial benefit, should require informed consent.
For certain high-stakes actions or statutorily defined sensitive domains such as financial services, developers and deployers should be required to obtain affirmative user consent before executing. The researchers describe this as a targeted safeguard rather than a blanket human-in-the-loop requirement.
“Addressing the accountability gap for AI agents in the United States requires coordinated action across multiple layers of government and stronger collaboration with industry working groups.”
As part of those recommendations, the researchers said NIST should work with industry groups, including the W3C Agent Identity Registry Protocol Community Group, Decentralized Identity Foundation and OpenID Foundation, to develop standardized decentralized identifiers and cryptographically verified credentials for agents so that counterparties can confirm an agent’s identity and authorization.
The researchers propose that developers and deployers of AI agents – the companies that build the underlying models or operate the agents for users – should be treated as fiduciaries, especially when the agents are used in high-stakes situations.
This would create a duty of loyalty, requiring companies to act in users’ best interests within the tasks delegated to the agents and avoid undisclosed conflicts of interest. The researchers say putting this framework in place would require action from technical standards bodies, federal regulators and Congress.
Also Read: UK Cyber Agency Warns Companies to Put Safeguards Around AI Agents
