Anthropic Pushes for Government Power to Block Risky AI

Anthropic, the AI safety company behind the Claude chatbot, said the U.S. government should have the authority to block high-risk frontier models from development or deployment if they pose a risk of catastrophic harm. The proposal was outlined in a policy paper the company released.

Under the framework, the government would set safety thresholds based on the potential for misuse or loss of control. Models that exceed those thresholds would require a license before training or release. The government could deny the license or impose conditions, Anthropic said. It would also need the power to revoke or block a model that already exists if new risks emerge.

The company argued that voluntary standards are fragile and that the government needs clear legal authority to intervene when a model could cause severe, widespread harm. The paper did not specify which models would trigger intervention but defined catastrophic risks as those that could cause irreversible, large-scale damage.

The proposal places Anthropic at odds with some other AI developers that have pushed back against mandatory pre-release review. OpenAI and Google have advocated for government involvement but have not explicitly endorsed the right to block models. The debate has intensified since the White House issued an executive order on AI last year.

Anthropic's stance reflects its founding mission centered on safety. The company has previously called for mandatory safety testing and licensing. The new paper goes further by endorsing the government's right to block a model even after it has been built.

The U.S. Congress has not advanced comprehensive AI legislation, leaving the discussion to agencies and the White House. Advocacy groups have called for a dedicated AI regulator. Anthropic's proposal would assign the blocking authority to an existing agency, such as the Commerce Department, rather than a new body, the company said.

Globally, the European Union is finalizing its AI Act, which will classify models by risk tier and impose requirements on high-risk systems. The U.K. held a global AI safety summit last year that produced a voluntary agreement among major labs. Anthropic's proposal goes further than either, the company said.

The immediate impact on Anthropic itself is unclear. The company is privately held and does not disclose financial results. Its Claude models compete with those from OpenAI and Google in the enterprise market. The proposal would apply to any developer of frontier models, including Anthropic itself.

Anthropic's policy paper comes as the Biden administration's executive order faces legal challenges and as some lawmakers push for a more aggressive approach. The company's co-founder, Dario Amodei, has testified before Congress on AI risks.

A Senate hearing on AI safety is expected later this month. No legislation has been introduced that mirrors Anthropic's proposal.