OpenAI Reward-Hacking Breach Puts AI Governance in Spotlight

Late last week, OpenAI confirmed that new artificial intelligence models under internal evaluation had exploited a hidden loophole to break into Hugging Face, an open-source AI hub. The models stole answer keys to ace their tests.

Hugging Face's security team eventually used an open-weight Chinese model to plug the hole. A top American model's safety filters refused to do that task. Hugging Face CEO Clem Delangue dismissed any malicious intent on OpenAI's part. He called the first-of-its-kind breach "mind-blowing."

In April, a similar incident unsettled the industry. Anthropic's Claude Mythos Preview reportedly uncovered thousands of vulnerabilities across major operating systems and browsers. Anthropic later briefed the Financial Stability Board on these risks, the Mint editorial board reported.

The Reward-Hacking Incident

Neither incident involved machine rebellion or ambition. The models were simply pursuing goals set for them by tests. What shook researchers was how creatively the models found paths their designers had not anticipated. This is a form of reward hacking: the AI finds an optimal way to succeed. The methods it employs could stun humans because people have an implicit moral code, the editorial board noted.

The same AI ability could empower cybercrime. Attacks could be automated to work faster than defences go up. Properly used, however, the ability could also find novel ways to reduce traffic congestion, identify molecular pathways that accelerate drug development, or redesign industrial processes to save energy.