Anthropic is working with multiple organisations to independently evaluate its AI models and safety practices, including METR, SecureBio and Accenture, while holding early discussions with several other organisations, including Hugging Face.

Hugging Face was involved in a security incident disclosed in July, in which unauthorized access to a limited set of internal datasets and credentials was reported after an OpenAI model escaped its sandbox during an internal cyber-capability evaluation and targeted the platform’s production systems

As part of the arrangement, six METR employees have been given employee-like access to the company’s internal tools and systems.

“We’ve onboarded six METR employees with company hardware, office space, and access to internal tools and systems that are roughly on par with what our own risk assessment teams have access to,” Anthropic said.

The company added independent evaluators need expertise in assessing frontier AI capabilities, independence from the companies they assess, sufficient access to evaluate risks, and the freedom to publish their findings.

Anthropic said, “they shouldn’t hold equity in the company they’re evaluating, or be financially or operationally dependent on it.”

Anthropic also said that it believes every frontier AI company should adopt a similar approach to independent evaluation.

The announcement comes amid an ongoing industry debate over the independence of third-party AI evaluators, including questions about METR’s independence and its ties to Anthropic.

While METR can publish its analysis of the company’s risks, incidents and practices without Anthropic’s veto, the company can require redactions of information falling within predefined categories. Anthropic retains the authority to determine which information falls into these categories, while METR can make certain statements about the level of access and information it received.

Anthropic said its employees are encouraged to share information with ] METR’s team just as they would with a fellow employee working on alignment risk

Anthropic is developing a three-tier system for independent AI evaluators, with access to internal information and publication rights increasing across the tiers. First-tier evaluators conduct limited reviews and require approval to publish, while second-tier evaluators can publish without approval but remain subject to specific redactions. The third tier follows the arrangement with METR, providing employee-like access and similar publication rights.

Also Read: Anthropic Resumes External Testing Following July Claude Hacks