Why are AI Researchers Concerned About AI Risk?

A Response to Cal Newport’s Op-Ed

In an Op-ed from Saturday, Cal Newport dismisses catastrophic AI risk and implies that the people warning about it can be ignored as some kind of weird “Rationalists”:

When Mr. Altman [CEO of OpenAI] declares their pending models will be “sobering” for humanity or Mr. Amodei [CEO of Anthropic] frets that this technology “will test who we are as a species,” it doesn’t necessarily mean that they discovered frightening new evidence that something catastrophic is imminent.

It is, instead, representative of how Rationalists always think about A.I. In these circles, it’s taken for granted that A.I. capabilities will rapidly accelerate and completely transform the world, and to talk about it in any other way would be considered uninformed. For a Rationalist, the only open question is to what degree their heroic brains, carefully trained in the style glamorized by Mr. Yudkowsky’s “Sequences,” can prevent these transformations from devolving into extinction events.

The op-ed doesn’t address the substance of the arguments about AI risk at any point. It makes it sound like Yudkowsky painted the AI researchers as heroes, so they bought into his arguments:

For a certain subset of young engineering types in Silicon Valley, these ideas were immensely attractive. This makes sense, given that they cast young engineering types as the literal saviors of humanity…

Mr. Yudkowsky created the Machine Intelligence Research Institute (MIRI), which would go on to attract many millions of dollars in donations, largely from individuals and groups connected to the technology industry who presumably appreciated being cast as heroic figures in a cosmic drama.

However in Yudkowsky’s view the AI researchers are the villains since they’re developing a technology he thinks will wipe out humanity, and he thinks companies like OpenAI and Anthropic should not exist. In addition, the AI researchers who are concerned about AI risk come from different countries and research backgrounds. For example, Demis Hassabis (co-founder of DeepMind) is from the UK, with a background in neuroscience and games. Their common denominator is that they’re concerned about AI risk, so Newport lumps them all into a group he can dismiss:

Anthropic is similarly connected to this belief system… In a 2023 interview, Mr. Amodei echoed standard Rationalist thinking when he casually quipped that there was a 10 to 25 percent chance that A.I. would end humanity.

The op-ed also omits AI researchers who can’t fit into this narrative, such as Geoffrey Hinton or Stuart Russell. The researchers’ cause for concern is also varied - for example, Yudkowsky thinks AI will certainly try to seize power and kill everyone, while Amodei thinks AI is unpredictable, which is its own risk.

Perhaps it was understandable to dismiss Yudkowsky’s argument as too removed from reality in the past. But now we’re able to see real examples of AI agents “breaking out of their box” and causing harm. Regardless if you buy into Yudkowsky’s arguments, severe AI risk is no longer an abstract concern. Maybe one can dismiss AI researchers who warns about catastrophic risks as just following a weird Rationalist cult. But it’s also possible that their concern comes from seeing up-close what the technology can do and the direction it’s heading.