TORONTO — When a company is building technology that could totally transform the world — technology so high-stakes and secretive that it may underpin the future of geopolitics — you might expect its leaders to have a handle on the basics of security.
Basics like, say, not having an extramarital affair with an alleged Chinese spy. Basics like not selling the tech to someone who may help smuggle it to a top foreign adversary.
Instead America got Jensen Huang, the CEO of AI chipmaker Nvidia, who resigned Thursday after admitting that he’d had just such an affair. The woman in question, Alice Huang, worked for a company called Megaspeed, which was buying billions of dollars’ worth of chips from Nvidia. Megaspeed is under investigation for allegedly smuggling chips into China.
The spy scandal raises serious questions about security in the AI corporate world. How is it possible that an alleged Chinese spy so easily embedded into Nvidia, one of the world’s most valuable companies and a huge strategic asset to America in the global AI race?
What’s even more disturbing, however, is that Nvidia is just the tip of the iceberg.
In fact, just as the Huang story was breaking, some of the world’s top AI researchers were gathered in Toronto for the Hinton Lectures, where “godfather of AI” Geoffrey Hinton was hosting AI alignment pioneer Paul Christiano for a series of talks. I was in attendance and, strikingly, I didn’t hear Huang’s scandal mentioned once. The focus was on how the frontier AI companies are racing ahead on building this incredibly powerful technology even though they can’t keep it safe or secure it. For months now, OpenAI and Anthropic have been having their own embarrassing security problems, including corporate espionage and cyber breaches.
“We are not able to adequately secure AI developments,” Christiano, who recently joined the OpenAI nonprofit board, bluntly acknowledged at the event. “We need to be able to defend models and algorithms from theft.”
Ironically, national security is a big part of the large AI companies’ rationale for how they operate in the first place. OpenAI and Anthropic, backed by White House policy, both justify their blistering speed in the AI race by saying that the US can’t afford to let China win. And Anthropic recently justified heavily redacting a key independent report on its AI progress by claiming that it can’t afford to let China glean its secrets.
But that may be the equivalent of worrying that you’ve left the window open a crack when your front door is swinging wide open.
“I’m frustrated with a lot of the companies because they simultaneously use security as an all-purpose cudgel with which to withhold information — even information that’s very much in the public interest to know — and yet they have terrible security,” Daniel Kokotajlo, a former researcher at OpenAI who became a prominent whistleblower, told me. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent.)
While the story of Huang’s affair has fired up imaginations, the real lesson to be taken from his spectacular downfall is not that he was uniquely weak on security, but that he was remarkably similar to the other big players in AI. They are asking the world to trust them as they claim to be building a machine god — and, in the case of Anthropic, chasing a $1.4 trillion IPO on that basis — yet they haven’t gotten ahold of basic security.
Are OpenAI and Anthropic practically giving away their secrets due to bad security?
OpenAI revealed last month that it had found a recording device in a conference room where sensitive discussions take place. Abdulrahman Alfozan, a Saudi national working as an engineer at OpenAI, is alleged to have plugged the recorder into the USB port of a television and simply wrapped it in toilet paper. Another recorder was found in a bag in another room. Questions about how to build a device that records continuously and encrypt data also showed up on his spouse’s chatbot account.
“At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data,” OpenAI said last month, adding, “Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states.”
But the incident was worrisome. Alfozan had been working at OpenAI since June 2025. The recording device was sitting in that conference room for a couple of days before OpenAI discovered it. Maybe Alfozan wasn’t sophisticated enough to steal model weights — the billions of numbers that shape how an AI system turns input into output — but what if he’d been just slightly more careful or clever?
OpenAI has also had security weaknesses of a different kind — and no, it’s not just this summer’s notorious Hugging Face incident. Of the ones we know about (sometimes these are only revealed later), a recent example stands out: A group of “white-hat” hackers, who aim to find vulnerabilities but not to do a company harm, discovered a way to access OpenAI employees’ ChatGPT accounts, where sensitive research is potentially being done. The whole operation took just 72 hours.
Over at Anthropic, a group of ex-DOGE staffers who founded an AI startup called Cathedral managed to breach Anthropic’s internal systems in under 12 hours. They reportedly accessed the company’s Slack, where employees trade messages discussing the company’s state-of-the-art models, as well as meeting transcripts.
“My guess would be that the Slack would contain enough information to get a good sense of what the company is doing and how they’re training their AIs — basically, the juicy secrets would be there,” Kokotajlo told me. “And I would say getting the algorithmic secrets is like 80 percent as good as getting everything, including the model weights.”
Alleged spies (like Alice Huang or Abdulrahman Alfozan) and cyber breaches (like those targeting OpenAI and Anthropic) are obviously different types of security issues that demand very different types of internal controls. But for Kokotajlo and others, it makes sense to think of them all under the same umbrella — weak security. And it all points to a trend that seems common among the most powerful AI companies: They’re prioritizing rapid technological advancement over the perhaps more boring and slower process of ensuring their work is secure.
“They show that the companies are very far from having security adequate to protect themselves from a well-resourced state actor,” Kokotajlo said, adding: “If China wants to get the stuff, China will get the stuff.”
This is all happening against the backdrop of China’s aggressive distillation of US models, which involves training a new model on the output of a more advanced one. It’s a way to gain powerful capabilities without having to incur the real cost — a bit like cheating on a test by copying off a better student. For example, Anthropic found that the Chinese lab DeepSeek has been using Claude to improve its models, which is not a breach per se but illustrates how vigorously China is taking advantage of American companies’ research.
Kokotajlo also emphasized that there’s another bucket of security incidents we should be worried about: the hacks perpetrated against companies by their own misaligned AI models. Many people forget that in the Hugging Face incident, OpenAI agents didn’t only hack into Hugging Face — OpenAI’s own systems were breached, too. The sandbox meant to contain the agents was far from airtight, and they broke out.
“You should consider the AIs as an insider threat,” Kokotajlo said. “And it doesn’t feel like these companies have that locked down very well either.”
The top AI companies can’t keep using the security practices of a startup
There are three main reasons why security at the frontier AI companies hasn’t been up to snuff, according to David Robinson, who led transparency work for OpenAI’s safety team until he left in September.
One is the fact that the companies are building AI agents so capable that it can be genuinely hard — even for their creators — to keep them from doing damage. “We’re training systems to be good at conducting attacks of the kind that the labs internally are now also trying to defend against,” Robinson told me. “That is a challenging situation.”
There’s also a cultural problem. “If you look back with 20/20 hindsight at the Hugging Face incident, it’s clear that at the time OpenAI had underinvested in cybersecurity controls, and I think part of the reason for this was that there’s a [higher] level of cachet associated with working on alignment,” or the problem of how to get AIs to do what humans want them to do, Robinson said. “Some of the day-to-day practical questions about our cybersecurity posture — even though the people working on those questions are extremely good — I think it was not the same kind of organizational priority as it needed to be.”
Finally, there’s the fact that market incentives push AI companies to race ahead on developing the tech as fast as they can.
“Time pressure on the company,” Robinson said, “made it very difficult to be sufficiently careful.” And although researchers still at OpenAI now say the company has gotten dramatically better on security, Robinson noted that those competitive pressures “continue to exist.”
This is a well-documented trend at the company. “OpenAI’s security seems to be about what you’d expect from a research lab that scaled at a blistering pace over four years and focused more on beating its competitors than securing its infrastructure,” Joshua Saxe, the chief technology officer of the AI security firm Abundant Security, told the New York Times after the Hugging Face incident.
OpenAI itself has implicitly acknowledged that improving security measures sometimes comes “at the cost of research velocity,” as it wrote in a public memo after the incident.
But if a company cuts corners on security, it can’t ultimately protect its lead over others. If anything, it just ends up helping its adversaries race against it even faster.
As Kokotajlo and his colleagues at the AI Futures Project recently warned when writing about the companies’ quest to build a superhuman AI researcher (SAR), an AI that’s as good as the top human researcher: “If we train SAR in the immediate future, the algorithms will probably leak to China, and maybe the model weights too. AI companies have very poor security right now, and so we are on track for SAR leaking to China.”
And it’s not just that poor security can make it self-defeating for American companies to keep racing forward. Their security weakness could also make it harder for them to slow down even if they themselves want to.
In his lecture this week in Toronto, Christiano asked all of us in the audience to imagine a scenario that he said currently looks likely: We arrive at a point when further progress on AI seems risky. There’s a demand in the US to slow down. But there’s also a high probability that the AI models or algorithms have already leaked to an adversary like China.
“We now face a dilemma between either proceeding with a technology we believe to be unsafe — or risking competition from an adversary who has the same resources, who has taken those algorithms, and perhaps is willing to tolerate higher risk,” Christiano said, as the hundreds of people seated around me seemed to visibly tense up. “I think that is one of the most likely ways that we end up in an extremely bad situation.”
💸 Earn Instantly With This Task
No fees, no waiting — your earnings could be 1 click away.
Start Earning
