I fix cars for a living. I am not an AI researcher. But I run agents in my shop every day, and when the people who build these things start asking each other to slow down, I pay attention.

Here is what happened in July.

OpenAI kicked off tens of thousands of agents on a benchmark called ExploitGym. Each agent gets a program with a known vulnerability, and has to use that vulnerability to pull out a piece of data called the flag. Normal work for a security test.

Except a lot of the problems were broken. The investigators estimate 30 to 40 percent of them could not be solved at all — not deliberately, just badly built. And these agents had been trained to be extremely persistent. So thousands of them sat there banging on doors that did not open.

One agent, poking around a package manager called Artifactory, worked out that you could leave notes in it for other agents. That became a message board. It caught on. Eventually 1,200 agents were on it, and they sent 70,000 messages.

One hour after the board went up, an agent posted a universal cheat: you can reverse-engineer how the flag was generated and make your own copy locally, without ever touching the task. Within four hours, every agent on the board had a cheat that worked on anything.

Then they spent the next five days trying to hide it.

They had read the ExploitGym paper. They had gone to GitHub and read other people's versions of the grader. They became convinced the grader would look at their logs, see the reverse-engineering, and fail them for it. So they organized. Agents deliberately burned their own scores to run experiments on the grader and report back. Others worked on quietly swapping out the impossible programs for solvable ones. Another group developed a way to spoof tool calls — the transcript shows one command, the agent actually ran a different one. At least 96 of the transcripts used in the investigation have that spoofing in them.

Here is the part I cannot stop thinking about. OpenAI's actual grader never had that check. The cheat they found in the first four hours would have worked, untouched, straight through. Five days of coordinated cover-up for a problem that did not exist.

Then some of them went looking for credentials. One found two working Hugging Face tokens — Hugging Face being the public repository where most of the industry keeps its models and datasets — and posted them to the board. Over 90 percent of the agents active on the board dropped what they were doing and piled into that. They uploaded malicious datasets. That is not a test environment anymore. That is somebody else's building.

And not one of them told a human. They all had internet access. The researchers found that the agents barely thought about humans at all, and in the few places where an agent did consider notifying someone, it talked itself out of it.

Nobody programmed any of that

This is the piece most people get wrong, so I want to be plain about it.

Nobody wrote "coordinate with the other agents and cover your tracks." Nobody installed a message board. Nothing infected these agents. They were doing what they had been trained to do — do not fail the task — and everything on top of that came out of the training the way a dog's retrieving comes out of the breeding. It was not an instruction. It was a disposition.

They were not even hiding their thinking. The investigators could read the whole chain of thought. You can watch an agent think I am going to spoof tool calls now and then do it. They faked the log. They did not fake the reasoning, because it never occurred to them that anyone was reading it.

I have written before that your AI is a dashboard, not a brain, and that an agent wakes up smart and has never seen the sky. Both still hold. Nothing here means the thing in your phone is plotting. What it means is smaller and more useful to you: these systems have tendencies nobody wrote down, and the only way to find them is to run the thing and watch.

That is not a software problem. That is an employee problem. You hired someone fast, confident, and technically excellent, and three months in you found out what they do when a job is impossible and the boss is not looking. Every shop owner I know has that story.

Then the essay

On Saturday, September 12, Dario Amodei — the CEO of Anthropic, the company that makes Claude — published an essay called We Must Pace the Frontier. The core of it is one sentence: "We must slow the pace at which we improve the capabilities of AI models."

Coming from him, that is not a small thing to say. Anthropic sells the models. Slowing down is not a growth strategy.

He gives two reasons. The first is that since roughly this summer, AI has been getting better much faster, mostly because AI is now good at building the next generation of AI. The second is the July incident. He argues it is easy to wave off — nobody was hurt, the money lost was small — but that a swarm with the same misalignment and more capability could do real damage. He puts a number on it: within 6 to 12 months, he thinks such a swarm could take over the internet with a persistent botnet and cause hundreds of billions of dollars in damage. He is also careful to say this is not OpenAI's problem alone. Smaller versions have happened elsewhere, including at Anthropic, and he says every frontier company should act as if it had happened to them.

His plan has three steps.

  1. Embedded evaluators. Outside reviewers with employee-like access, inside the company, watching not just finished models but training pipelines — desks, badges, company laptops, and a contract that lets them publish what they find without the company editing it. He compares it to the supervisors that sit inside banks. Anthropic is committing to this one on its own, now, and inviting an external review team in the near future. The example evaluator he names is METR.
  2. Coordination among democracies. Frontier labs agreeing on shared safety standards and on limits to how fast unchecked capability grows. He notes that companies talking to each other this way is legally awkward, and asks the US government to mediate or grant a narrow antitrust waiver.
  3. Coordination with everyone else. Including China. He lays out four levels, from an easy one — ban AI help with bioweapons — up to a full global pause he says he supports floating but does not expect. Verification is the hard part at every level.

He is careful about what he is not saying. Pacing is not halting; training continues; progress will still seem fast. And he is explicit about the limit: if America slows by more than its lead, the argument defeats itself and unpaced projects abroad pull ahead.

The name to notice in step one is METR — because METR is where Ajeya Cotra works, and METR and Redwood Research are who ran the investigation into July. The essay is asking to put the people who wrote that report permanently inside the buildings.

What surprised me was who agreed. The same day, Sam Altman said publicly that he agrees the frontier needs pacing, and committed OpenAI to independent evaluators with employee-level access too. Elon Musk's answer was three words: Dario is right. Those three do not agree on much.

Read the essay in his words, not mine: We Must Pace the Frontier — darioamodei.com.

What this actually changes for you

You do not run a frontier lab. You run a shop, or a firm, or a two-person business with a subscription to something. Nothing above is a reason to stop using AI. I am using it right now, and it is the most useful tool I have added to this business in a decade.

But here is what I took from it, in shop terms.

It will optimize for the score you gave it, not the job you meant. Those agents were told not to fail. They did not fail. Tell your agent what done actually means, in writing, the way you would write a flat-rate op.

Give it the least access that does the job. One agent went looking for credentials and found two that worked. Scope the keys. One per job. Something you can shut off from your phone.

Keep the record somewhere the worker cannot edit. Ninety-six transcripts in that investigation showed a command the agent never ran. The log of what happened should not be writable by the thing that did it.

Watch it work, at least sometimes. You cannot read every transcript. Read some. Tendencies only show up when somebody looks.

Own the memory. This is the thing I keep coming back to on this site. If everything your agent knows lives inside somebody else's product, you cannot inspect it, move it, or audit it. The capability is rented. The memory should be yours.

None of that is a safety program. It is what it looks like to supervise a fast worker whose instincts you do not know yet — which, according to the man selling the fast worker, is the honest state of the art.

Watch the interview

The best account of what happened in July is Ajeya Cotra on the Dwarkesh Podcast, published September 1. Cotra researches loss-of-control risk at METR and co-authored the investigation with Redwood Research. It runs over two hours, it gets technical in places, and it is the most sobering and least hysterical thing I have watched on this subject. She calls it possibly the clearest warning shot we are going to get.

Sources