Pausing AI developments is not enough. We need to shut it all down.

Yudkowsky is a U.S. decision theorist and leads research at the Machine Intelligence Research Institute. He has been working on Artificial General Intelligence alignment since 2001 and is widely regarded as one of the founders of the field.

An open letter published at the end of May 2023 calls for “all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.”

This 6-month moratorium would be better than no moratorium. I have respect for everyone who stepped forward and signed it. It is a marginal improvement.

I refrained from signing because I believe the letter underestimates the severity of the situation and asks very little to resolve it.

The key issue is not “human-level intelligence” (as the open letter says); it’s what happens after AI reaches an intelligence greater than human. The key thresholds there may not be obvious, we certainly cannot calculate in advance what will happen and when, and it currently seems imaginable that a research lab could cross critical lines unknowingly.

Many researchers immersed in these topics, including myself, expect that the most likely outcome of building a superhumanly intelligent AI, under any remotely similar circumstances to the current ones, is that literally everyone on Earth will die. Not like “maybe possibly some remote chance,” but like “that’s the obvious thing that would happen.” It’s not that you can’t, in principle, survive by creating something much smarter than you; it’s that it would require precision and preparation and new scientific knowledge, and probably not having AI systems made up of giant, inscrutable sets of fractional numbers.

Without that precision and preparation, the most likely outcome is an AI that doesn’t do what we want and doesn’t care about us or sentient life in general. That kind of care is something that, in principle, could be imbued into an AI, but we are not ready and currently do not know how to do it.

In the absence of that care, we get “AI doesn’t love you, nor does it hate you, and you are made of atoms it can use for something else.”

The likely outcome of humanity facing an opposing superhuman intelligence is total loss. Valid metaphors include “a 10-year-old trying to play chess against Stockfish 15,” “the 11th century trying to fight against the 21st century,” and “Australopithecus trying to fight against Homo sapiens.”

To visualize a hostile superhuman AI, don’t imagine a lifeless, intelligent thinker living inside the Internet and sending malicious emails. Imagine an entire alien civilization, thinking millions of times faster than humans, initially confined to computers, in a world of creatures that are, from its perspective, very stupid and slow. A sufficiently intelligent AI will not stay confined to computers for long. In today’s world, it can send DNA strands through email to laboratories that will produce proteins on demand, enabling an AI initially confined to the Internet to build artificial life forms or directly start post-biological molecular manufacturing.

If someone builds an AI that is too powerful, under the current conditions, I expect that all human members of the species and all biological life on Earth will die soon after.

There is no proposed plan for how we could do such a thing and survive. OpenAI’s openly declared intent is to have some future AI do our AI alignment task. Just hearing that this is the plan should be enough to make any sensible person panic. The other leading AI lab, DeepMind, has no plan.

A side note: none of this danger depends on whether AIs are or can be conscious; it’s intrinsic to the notion of powerful cognitive systems that optimize and calculate results that meet sufficiently complicated outcome criteria. That said, it would be negligent in my moral duties as a human if I didn’t also mention that we have no idea how to determine if AI systems are self-aware, as we have no idea how to decode anything that happens in the giant inscrutable matrices, and therefore, at some point, unknowingly, we may create digital minds that are truly self-aware and should have rights and should not be property.

The rule that most people aware of these problems would have backed 50 years ago was that if an AI system can speak fluently and says it is self-aware and demands human rights, that should be a barrier to people simply owning that AI and using it beyond that point. We’ve already passed that old line in the sand. And that was probably correct. I agree that current AIs are probably just imitating conversation about self-awareness from their training data. But I point out that, with the little understanding we have of the internal parts of these systems, we actually don’t know.

If that’s our state of ignorance for GPT-4, and GPT-5 is the same giant leap in capability as GPT-3 to GPT-4, I think we will no longer be able to justifiably say “probably not self-aware” if we allow people to build GPT-5. It will just be “I don’t know; no one knows.” If you can’t be sure whether you’re creating a self-aware AI, this is alarming not just because of the moral implications of the “self-aware” part, but because not being sure means you have no idea what you’re doing. And that’s dangerous, and you should stop.

On February 7, Satya Nadella, CEO of Microsoft, publicly boasted that the new Bing would make Google “come out and show it can dance.” “I want people to know we made them dance,” he said.

This is not how a Microsoft CEO speaks in a sane world. It shows an overwhelming gap between the seriousness with which we take the problem and the seriousness with which we needed to take it 30 years ago.

We’re not going to close that gap in six months.

More than 60 years passed from when the notion of artificial intelligence was first proposed and studied until we reached current capabilities. Solving the safety of superhuman intelligence, not perfect safety, safety in the sense of “not literally killing everyone,” could reasonably take at least half of that time. And what’s tricky about trying this with superhuman intelligence is that if you mess up the first attempt, you can’t learn from your mistakes because you’re dead. Humanity doesn’t learn from error and dusts itself off and tries again, as with other challenges we’ve overcome in our history, because we’re all gone.

Trying to do something right on the first truly critical attempt is an extraordinary task, both in science and engineering. We’re not coming in with anything like the approach that would be required to do it successfully. If we applied the lesser engineering rigor standards that apply to a bridge designed to carry a couple thousand cars to the emerging field of Artificial General Intelligence, the whole field would be shut down tomorrow.

We are not prepared. We are not on track to be prepared in a reasonable time window. There is no plan. Progress in AI capabilities is massive, far ahead of the progress in AI alignment or even understanding what the hell is going on inside those systems. If we really do this, we are all going to die.

Many researchers working on these systems think we are rushing toward a catastrophe, and more of them dare to say it privately than publicly; but they think they can’t unilaterally stop the forward fall, that others will continue even if they personally quit their jobs. And so everyone thinks they might as well keep going too. This is a stupid state of affairs and an undignified way for Earth to die, and the rest of humanity should intervene at this point and help the industry solve its collective action problem.

Some of my friends have recently informed me that when people outside the AI industry first hear about the extinction risk of Artificial General Intelligence, their reaction is “maybe we shouldn’t build AGI then.”

Hearing this gave me a small flicker of hope, because it is a simpler, more sensible, and frankly more reasonable reaction than what I’ve heard in the last 20 years of trying to get someone in the industry to take things seriously. Anyone who speaks like this deserves to hear how serious the situation actually is, and not be told that a six-month moratorium will solve it.

On March 16, my partner sent me this email. (Later they gave me permission to share it here).

“Nina lost a tooth! The usual way kids do it, not by carelessness! Seeing GPT-4 pass those standardized tests on the same day Nina reached a childhood milestone triggered an emotional wave that made me lose my head for a minute. Everything is moving too fast. I’m worried that sharing this might increase your own pain, but I’d rather you know than each of us suffer alone.”

When the internal conversation is about the pain of seeing your child lose their first tooth and thinking they won’t get the chance to grow up, I think we are past the point of playing political chess over a six-month moratorium.

If there were a plan for Earth to survive, if we just approved a six-month moratorium, I would support that plan. There is no such plan.

This is what would actually need to be done:

The moratorium on new large-scale training must be indefinite and global. There can be no exceptions, even for governments or the military. If the policy starts with the US, then China must see that the US is not seeking an advantage but is trying to avoid a terribly dangerous technology that cannot have a true owner and will kill everyone in the US, China, and Earth. If I had infinite freedom to write laws, I might make a single exception for AI to be trained solely to solve problems in biology and biotechnology, not trained on Internet text, and not at the level where they begin to talk or plan; but if that were remotely complicating the problem, I would immediately discard that proposal and say to simply shut everything down.

Turn off all large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all large training races. Put a limit on the amount of computing power anyone can use to train an AI system and move it downward over the coming years to offset the more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent prohibited activities from shifting to other places. Track all GPUs sold. If intelligence says a country outside the agreement is building a GPU cluster, be less afraid of a shooting conflict between nations than of violating the moratorium; be willing to destroy a rogue data center through an airstrike.

Do not frame anything as a conflict between national interests, make it clear that anyone talking about an arms race is a fool. That we all live or die as one, in this, is not policy but a fact of nature. Make explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a total nuclear exchange, and that allied nuclear countries are willing to take some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training races.

That is the kind of policy shift that would make my partner and I hug and tell each other that a miracle happened, and now there’s a chance Nina will live. Sane people who hear about this for the first time and sensibly say “maybe we shouldn’t” deserve to hear, honestly, what would be needed for that to happen. And when their policy request is that big, the only way it will pass is if lawmakers realize that if they do business as usual and do what’s politically easy, that means their own children are going to die too.

We are not ready. We are not on track to be significantly more prepared in the foreseeable future. If we proceed with this, everyone will die, including the children who didn’t choose this and did nothing wrong.

---

Future Lab Analysis Team / Time Magazine (England). Article by Eliezer Yudkowsky.

Translation from English: Translation and Interpretation Team of the Future Lab.