Save this storySave this storySave this storySave this story
Last spring, Daniel Kokotaylo, an AI safety researcher working at OpenAI, quit in protest. He felt the company was unprepared for the future of its technology, and he wanted to draw attention to the issue. After a mutual friend put us in touch, we discussed it over the phone. I found Kokotaylo friendly, knowledgeable, and concerned. He told me that advances in “alignment”—a set of techniques designed to make AI act in line with human commands and values—were lagging behind advances in intelligence. Researchers, he said, were rushing to build powerful systems that they couldn’t control.
Kokotailo, who went from a philosophy PhD to AI, explained how he had been studying the field on his own. In his work at OpenAI, he had been tracking AI developments to create timelines predicting when different levels of intelligence might be achieved. At one point, when the technology suddenly advanced, he had to revise his estimates decades into the future. In 2021, he wrote a scenario about AI called “What 2026 Looks Like.” Much of what he predicted happened before that year. He concluded that the moment when AI could outperform humans at almost every important task and gain significant influence and power could come in 2027 or even earlier. He sounded scared.
Around the same time that Kokotailo left OpenAI, two Princeton computer scientists, Syash Kapoor and Arvind Narayanan, were preparing to publish their book, AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference. In it, Kapoor and Narayanan, who study the adoption of technology in society, presented views that were the complete opposite of Kokotailo’s. They argued that many timelines for the future of AI were overly optimistic; that claims about its usefulness were often exaggerated or outright misleading; and that, given the inherent complexity of the world, even powerful AI would be slow to adopt. They cited numerous examples of AI systems making important decisions—in medicine or hiring, for example—and making rookie mistakes that indicated a fundamental disconnect with reality. Brand-new systems, they argued, suffered from the same flaw.
All three researchers have recently taken their positions a step further, publishing papers that take their analysis further. The nonprofit AI Futures Project, led by Kokotailo, released a paper called “AI 2027,” written by him and four other researchers, that describes a terrifying scenario in which “superintelligent” AIs either dominate or wipe out humanity by 2030. The paper should be taken seriously as a warning about the potential consequences. Meanwhile, Kapoor and Narayanan argue in a new paper called “AI as Normal Technology” that practical obstacles—from regulations and professional standards to the sheer difficulty of performing actions in the real world—will slow the deployment of AI and limit its transformative potential. While acknowledging that AI could eventually become a disruptive technology comparable to electricity or the internet, they argue that it will remain “normal” — that is, controlled by familiar safety measures like fail-safe devices, kill switches, and human oversight — for the foreseeable future. “AI is often compared to nuclear weapons,” they argue. But “the correct analogy is nuclear power,” which remains largely controlled and perhaps underused for safety purposes.
Is this business as usual, or the end of the world? “The test of a first-rate intelligence,” as F. Scott Fitzgerald said, “is the ability to hold two opposing ideas in the mind at the same time and still retain the ability to function.” Reading these reports in sequence, I found myself losing that ability, and speaking to their authors one by one over the course of a day certainly made me feel like I was losing my mind. AI 2027 and AI as Normal Technology purport to describe the same reality, and are written by deeply knowledgeable experts, but they reach absurdly different conclusions. Discussing the future of AI with Kapoor, Narayanan, and Kokotailo felt like I was having a conversation about spirituality with Richard Dawkins and the Pope.
In the parable of the blind men and the elephant, a group of well-intentioned people grapple with an unfamiliar object, unable to agree on its nature because each believes that the part they encounter defines the whole. This is part of the problem with AI—it’s hard to see the whole in something new. But it’s also true, as Kapoor and Narayanan point out, that “today’s AI safety discourse is characterized by profound differences in worldviews.” If I had to summarize these differences, I’d say that, in general, West Coast and Silicon Valley thinkers are drawn to visions of rapid transformation, while East Coast scientists are distanced from them; that AI researchers believe in rapid experimental prog
Sourse: newyorker.com
