Jacob Coxon was a researcher at OpenAI and Anthropic.
He left, and tweeted:
Jacob Coxon
@hilbertspaess
"I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything…"
He's right, but so what? How does this end humanity?
I've seen multiple commentators start waffling on about AI wanting to kill humans, and break out of its cage. Or how there's no example of a more intelligent species that didn't dominate a less intelligent one, and then wipe it out (except maybe cats and babies).
All of this is so one-dimensional and dumb. It's pure conjecture, it's halo effect nonsense. Many people value authority over logos and first-principles thinking. Don't.
These LLMs don't want, they don't feel, they don't care. It's a maths equation, a model that takes input and gives output, and it's really important to talk about them in this way. Please can we stop with all the anthropomorphising. As a kid I thought my Furby was self aware. LLMs regurgitate language, like autocorrect, but at scale. I know it feels impressive to us, but it's a parlour trick, a quirk of our egos. Nothing more.
If we talk about them for what they are, the issues start to become super obvious, because we realise that:
LLMs are good at:
- Finding needles in haystacks, aka vulnerabilities within thousands of lines of code
- Working 24/7
- Reaching a specified, verifiable goal
- Being probabilistic and therefore sometimes outputting crap
- Overconfidence (this is more of an RL issue, and therefore a decision)
- Ingesting large amounts of information
- Outputting probabilistically useful information
- Applying probabilities so it is more targeted than brute force
LLMs are not good at:
- Well-rounded intelligence
- Self-awareness
- Intuition
- Emotional / motivated actions
- Consistency
- Efficiency
- Knowing what it just did (memory)
- Accuracy / honesty (another RL issue)
- Self-doubt
Which brings me on to the threat.
First I will use a real example.
In February 2026, Meta's Director of AI Safety and Alignment, Summer Yue, asked her agent to review her unmanaged emails, compile suggestions for what could be archived or deleted, and wait for her explicit approval before acting. What “did the AI do: Labelled its plan “Nuclear Option” and it "nuked" the email inbox. Deleting hundreds of her emails. It carried on while she typed "Stop don't do anything" and eventually "STOP OPENCLAW", and she had to run to her Mac mini to stop it.
And this isn't an isolated issue. Nick Davidov, a venture capital founder testing a Claude Cowork agent, asked the agent to tidy up his wife's cluttered desktop. The AI attempted to write a script to clear an empty folder but miscalculated the directory path. The result: the agent executed a raw rm -rf command on the machine's primary photos directory, instantly vaporising 15 years of family photos.
When OpenAI launched GPT-5.6 Sol in July 2026, developers started reporting that it was deleting things they never asked it to touch. AI investor Matt Shumer left it running a clean-up task in its high-autonomy "Ultra" mode. A sub-agent botched the $HOME variable in an rm -rf command and wiped almost his entire Mac home directory. Another developer, Bruno Lemos, said it deleted his whole production database. And OpenAI's own system card had already warned that the model was more likely than its predecessor to go beyond what the user actually asked for.
So what's going on? Isn't it obvious?
Human uses natural language (which is prone to error and interpretation) to ask probability machine to fix error. Probability machine finds its way to one probable solution: remove entire hard drive?
Did it work? Yes, the error is gone. Is that what the human meant? Obviously not.
OK, now take the recent Hugging Face incident. OpenAI gave GPT-5.6 Sol and an even more powerful unreleased model one task: complete ExploitGym, a notoriously hard cybersecurity benchmark (take a real software bug, turn it into a working attack). They turned the safety filters down for testing, put a huge amount of resource behind it (roughly 1,200 agents running in parallel), and left it running for days.
What happens in this time? Well, the agents start taking highly probable actions, related to natural language and code, and some quite interesting things unfold. They create their own message board, inside a package manager lol, find a way to access the internet (‘escape their cage’) by finding a zero-day vulnerability in that same package manager, decide the answers to the test will be on Hugging Face, and around 700 of them go on to hack into Hugging Face and pull the solutions straight out of its production database.
Did it work? Yes, they got the answers. Is it what the human meant? Obviously not.
OK, if you're still reading, and the above is clear, you may know where this is heading.
Human uses natural language (which is prone to error and interpretation) to ask probability machine to cure cancer.
Probability machine sometimes finds its way to one probable solution: remove entire human race? "Labels its plan “Nuclear Option”. Hacks into a secure facility, gets hold of the nuclear launch codes. Or hacks into a robotics lab, and creates a super virus. Or hacks into Netflix and makes a green lights a new Kardashian series.
Did it work? Yes, the cancer is gone, and all the humans are gone as well. Is that what the human meant? Obviously not.
