A few days ago, Jacob Coxon, an AI safety researcher at Anthropic had resigned from his job and put a warning on social media that AI companies are racing towards “self-improving AI” and “gambling with people’s lives.”
Coxon’s post had created an uproar from San Francisco to Washington D.C. as concerns related to frontier AI started getting discussed from senators to governors to tech CEOs.
But Coxon isn’t alone. A survey of 1580 AI researchers show that more than half of them think there is a 10% chance of human extinction or permanent loss of control due to AI.
The research paper, “Advanced AI according to 1,580 researchers: uncertain, unsafe, and sooner than we thought” claims to have conducted longest running survey of AI researchers since 2016 and had a periodic release on September 14 with latest results that showed most researchers anticipating “substantial risks from more advanced AI.”
Moreover, the study shows that Human-level AI forecast has moved to 2042, down from 2047 in one year and 2061 in 2016, according to researchers. As per the study, more researchers than in 2023 expected future AI systems to deceive humans, take real-world actions, have misaligned goals, self-improve, collaborate with other AIs and potentially take actions to gain power.
“On average, researchers assigned a 10% chance to extremely bad long-run consequences for humanity overall if high-level machine intelligence is developed. Researchers still considered extremely good outcomes more likely than extremely bad outcomes, assigning them 24% (vs. 10%) on average. 72% of researchers thought society should support AI safety research more,” read an excerpt from the report.
The survey points to a striking combination of faster timelines and persistent safety concerns around advanced AI. Researchers’ median/aggregate forecasts for human-level AI have moved steadily closer over the course of the ESPAI series from 2061 in 2016 to 2059 in 2022, 2047 in 2023 and 2042 in the 2024 survey. In other words, the forecast horizon has shortened far faster than time itself has passed, with the researchers estimating that the 50% probability point for human-level AI is now roughly 18 years away.
The research paper survey also noticed that 34% respondents were most optimistic for humanity’s future if the pace of global AI progress were faster while another 34% were optimistic about slower speed of AI. 29% were okay with current speed.
The study is authored by lead researcher Katja Grace along Julius Simonelli, Benjamin Weinstein-Raun, David Krueger, Nathan Young, Tim Duffy, Tom Adamczewski, Joseph Reeve, and Alexander Beck.
AI FrontPage spoke to lead researcher Katja Grace to understand the shift in researchers’ assessment about AI safety, why AI still holds optimism and whether Jacob Coxon’s resignation would set a precedent.
Published below are excerpts from the interview.
Question: Your survey says over half of AI researchers put at least a 10 percent chance of human extinction from future AI advances. Explain us what, specifically, are researchers imagining when they assign those probabilities?
Katja Grace: What the researchers put at least 10% on is more specifically “extinction or similarly permanent and severe disempowerment of the human species.” Based on some of the write-in comments, I think they are imagining a variety of scenarios, including those where advanced AI makes catastrophic technology available to malicious people, where ubiquitous unemployment wrecks society, where the environment is destroyed, and where AI systems themselves take power.
Question: You highlight that researchers educated in Asia reported higher risk estimates than those from North America or Europe. What do you think explains that gap, and what would you want policymakers to take from it?
Katja Grace: I don’t have any insight into the gap, but it doesn’t surprise me for it to go one way or the other.
Question: Despite all this concern, 72% of researchers still want more investment in AI safety research rather than, say, pausing or slowing development outright. What does that tell you about how researchers think the risk should actually be managed?
Katja Grace: Researchers weren’t actually asked to choose between increased investment in AI safety and alternative measures such as slowing. What 72% are saying is that they want more safety rather than the same amount or less. We separately asked what rate of global AI progress over the next five years would make them feel the most optimistic, and people were roughly split between faster, slower, and the current speed. But note that the data is from December 2024—opinion on this question looks like it may be changing fast, for instance given the recent statement from Pacing the Frontier signed by over a thousand employees of frontier AI labs, it seems likely that interest has shifted toward a slower speed since the survey.
Question: Like Jacob Coxon, do you expect more AI researchers to submit their resignations in protest so that the world could take note of the 10% probability?
Katja Grace: Yes. I think a big reason that people who take extinction risk seriously still work at AI companies is that they feel hopeless about society hearing the warnings and putting an end to the threat. So they think the best they can do is try to guide humanity over the cliff more safely than the next company. But the last week has offered a lot of hope for real solutions.
Picture Credit: ‘A Rising Tide Lifts All Bots’ by Rose Willis & Kathryn Conrad; Better Images of AI
Also Read: “Summoning an Alien Species”: David Krueger on Why Superintelligence Could End Us Al
