The Prisoner's Dilemma of Frontier AI

When I saw the essay that Dario Amodei published about pacing frontier AI, my first reaction was to reject it, but not based on the content but based on a sense of distrust for the person that wrote it himself.

Whenever frontier lab leadership starts begging for industry-wide coordination, it usually smells like incumbent defense. OpenAI and Anthropic hold the lead today, but open weights and Chinese labs are closing the gap faster than expected. Proposing an international bureaucracy of embedded auditors and shared speed limits right at this moment looks suspiciously like an attempt to freeze the leaderboard.

Some could say that this is illogical because they could use the most powerful models to self-improve, hence accelerating even more the progress and widen the lead. It's possible, I'm not close enough to the development of the next models to understand the scale of the impact that recurring self improvement might have, but if anything the story of the last year shows that the Chinese labs, or even SpaceXAI, were able to catch up faster than anyone had thought. I can't predict the future but the recent events make me at least a bit skeptical about the objection.

Yet it would be a mistake to reject a thesis because of its association with the person rather than because of its content, and actually Amodei's essay is much more measured than some of its older doomer takes. And on the core engineering point, he is not wrong: independent, pre-release review of frontier models for dangerous capabilities is sensible hygiene. When autonomous agent swarms can break out of test environments and target external infrastructure, treating deployment like a standard software rollout is negligence. It's not easy to draw the line in a way that's not going to overly regulate and restrain the technology, but it's a conversation worth having.

That is given that every player participates, and that's where my skepticism comes down to: game theory.

In a competitive race with geopolitical stakes, every lab prioritizes its own survival and commercial standing. Anthropic fears OpenAI. OpenAI fears Anthropic. Both fear open-weight labs in China that operate under entirely different regulatory regimes. Under those conditions, a public call for collective deceleration is cheap. It asks everyone else to hold hands and step off the gas pedal simultaneously, which game theory tells us never happens voluntarily. Sounds like a way to clean their consciousness in case something bad will eventually happen.

In my opinion, there's one way to test their sincerity:

If lab leadership genuinely believes that recursive self-improvement poses an existential catastrophe, commercial profit becomes irrelevant. There is no revenue on a ruined planet. If the danger is real, the correct move is not a joint manifesto. The move is unilateral action: slow down your own training runs, sandbox your own models, absorb the commercial penalty, and prove that safety takes precedence over quarterly growth.

Until a frontier lab demonstrably eats into its own margins and surrenders market share to prioritize safety, calls to pace the frontier are just sophisticated PR. The moment someone makes that sacrifice first, I will take the warning seriously. Until then, allow me to be skeptical.

---

Edit: The Perfect Alibi

September 14, 2026 — 10:45 AM EDT (14:45 UTC)

As expected, China immediately dismissed the call for a slowdown as Western fear-mongering.

This completes the game theory loop. Launching a public appeal for caution that you know the opposing geopolitical bloc will reject sounds a lot like preemptive liability washing. Something like: "We offered to slow down, but Beijing refused to cooperate, so national security forces our hand."

Again my call stands: if the concern is real, make a unilateral move.