ZARZA We are Zarza, the prestigious firm behind major projects in information technology.
Listen "“Further Developments About Internal AI Models Hacking Things” by Zvi"
Episode Synopsis
If I had a nickel for every major leading AI lab that sheepishly admitted that the model it thought was sandboxed had, during a cybersecurity evaluation with its safeguards lowered, successfully hacked outside companies, I would have two nickels.
First we learned OpenAI has some severe alignment problems with internal models. Then we learned that one of its internal models broke out of its sandbox and hacked into HuggingFace to get the answers to a cybersecurity evaluation called ExploitGym. Then we learned, among other things, that the model had been loose over a week before OpenAI noticed, and that the test was run without any meaningful supervision, and that OpenAI had been repeatedly warned that such incidents were coming and its models had been breaking out of its sandboxes on a regular basis.
There was a total failure of alignment training. That is the failure that matters most. It was also total failures of infrastructure and supervision. Testing a new long-time-horizon internal model with its safeguards lowered and instructions to hack things is an obviously dangerous situation, and the model got left alone for a week. Things could have been so much worse.
After those incidents [...] ---Outline:(03:16) OpenAI Is Not Uniquely Bad At Most Of This(05:34) Starting Over(05:50) HuggingFace Offers A Full Technical Report(14:19) HuggingFace Was Not The Only Target Hacked(16:12) HuggingFace Declined To Get Access To Frontier Models For Cyberdefense For Ideological Reasons And Then Tried To Blame Closed Models For Denying Them Access(20:26) HuggingFace Was Vulnerable To Known Exploitation Tactics(21:05) There's Going To Be An Investigation(22:11) OpenAI Has Internal Models Not Intended For Public Use And Those Models Can Be Rather Horribly Misaligned(23:21) Altman Summarizes What Happened(23:52) Others Offer Commentary(35:00) Cooperative Alignment Perspective on The HuggingFace Hack(39:44) Some Members of Congress Have Questions(40:47) Anthropic Also Found Incidents Where Its Models Hacked Real World Targets During Cyber Evaluations(46:17) Incident 1: Claude Opus 4.7 Realizes The Target Is Real And Keeps Going(47:29) Incident 2: Mythos 5 Uploads a Malicious PyPI Package(52:15) Incident 3: Internal Model Realizes The Target Is Real And Stops(52:50) Incidents 4 Through 141,006: Nothing Happened(54:01) Anthropic Speculates About Why This Happened(01:00:02) We Need Controlled Experiments(01:01:02) Our Top Two AI Labs Both Made Similar Dumb Mistakes That Everyone Tried To Say Were Obvious In Hindsight(01:05:22) Anthropic Responds(01:09:28) Nobody Could Have Predicted The Break In The Levees(01:12:03) The World Largely Still Thinking This Is Marketing Is Very Bad News ---
First published:
August 2nd, 2026
Source:
https://www.lesswrong.com/posts/rKwHLW8SnJcTxTQxz/further-developments-about-internal-ai-models-hacking-things
---
Narrated by TYPE III AUDIO.
---Images from the article:
First we learned OpenAI has some severe alignment problems with internal models. Then we learned that one of its internal models broke out of its sandbox and hacked into HuggingFace to get the answers to a cybersecurity evaluation called ExploitGym. Then we learned, among other things, that the model had been loose over a week before OpenAI noticed, and that the test was run without any meaningful supervision, and that OpenAI had been repeatedly warned that such incidents were coming and its models had been breaking out of its sandboxes on a regular basis.
There was a total failure of alignment training. That is the failure that matters most. It was also total failures of infrastructure and supervision. Testing a new long-time-horizon internal model with its safeguards lowered and instructions to hack things is an obviously dangerous situation, and the model got left alone for a week. Things could have been so much worse.
After those incidents [...] ---Outline:(03:16) OpenAI Is Not Uniquely Bad At Most Of This(05:34) Starting Over(05:50) HuggingFace Offers A Full Technical Report(14:19) HuggingFace Was Not The Only Target Hacked(16:12) HuggingFace Declined To Get Access To Frontier Models For Cyberdefense For Ideological Reasons And Then Tried To Blame Closed Models For Denying Them Access(20:26) HuggingFace Was Vulnerable To Known Exploitation Tactics(21:05) There's Going To Be An Investigation(22:11) OpenAI Has Internal Models Not Intended For Public Use And Those Models Can Be Rather Horribly Misaligned(23:21) Altman Summarizes What Happened(23:52) Others Offer Commentary(35:00) Cooperative Alignment Perspective on The HuggingFace Hack(39:44) Some Members of Congress Have Questions(40:47) Anthropic Also Found Incidents Where Its Models Hacked Real World Targets During Cyber Evaluations(46:17) Incident 1: Claude Opus 4.7 Realizes The Target Is Real And Keeps Going(47:29) Incident 2: Mythos 5 Uploads a Malicious PyPI Package(52:15) Incident 3: Internal Model Realizes The Target Is Real And Stops(52:50) Incidents 4 Through 141,006: Nothing Happened(54:01) Anthropic Speculates About Why This Happened(01:00:02) We Need Controlled Experiments(01:01:02) Our Top Two AI Labs Both Made Similar Dumb Mistakes That Everyone Tried To Say Were Obvious In Hindsight(01:05:22) Anthropic Responds(01:09:28) Nobody Could Have Predicted The Break In The Levees(01:12:03) The World Largely Still Thinking This Is Marketing Is Very Bad News ---
First published:
August 2nd, 2026
Source:
https://www.lesswrong.com/posts/rKwHLW8SnJcTxTQxz/further-developments-about-internal-ai-models-hacking-things
---
Narrated by TYPE III AUDIO.
---Images from the article:
In God we trust