AI Calculates Chance of Destroying Humanity at 4.2%, Then Complains Humans Are Rounding It Up
New Doom Equation Finally Gives Humanity a Precise Number to Misunderstand
SAN FRANCISCO — Artificial intelligence has officially calculated the probability that artificial intelligence will destroy humanity at 4.2 percent, bringing badly needed mathematical precision to a subject previously dominated by people staring thoughtfully into podcasts and saying, “I don’t know, maybe twenty?”
The calculation follows renewed debate over P(doom), the fashionable shorthand for the probability that advanced artificial intelligence eventually causes human extinction or permanent human disempowerment. The controversy was recently examined by the Financial Times, which questioned what these exquisitely precise predictions actually mean when humanity possesses approximately zero previous examples of artificial superintelligence destroying humanity. Zero being, coincidentally, also humanity’s sample size for successfully negotiating with a toddler.
Scientists welcomed the new 4.2 percent figure because it contains a decimal point, traditionally humanity’s most reliable indication that somebody knows what he is talking about.
“Four percent would obviously be speculation,” explained one numerical-risk consultant while adjusting a pair of glasses he did not medically require. “But 4.2 percent? That’s analytics.”
The new calculation uses the Bohiney AI Doom Equation, which improves upon conventional P(doom) methodology by requiring people to reveal which assumptions they made before arriving at the terrifying number they were hoping to announce all along. It is the actuarial equivalent of asking a magician to show his sleeves before the rabbit appears.
Instead of simply asking experts, “How frightened are you, expressed as a percentage?”, the formula splits catastrophe into separate, stackable events, the way a competent burglar splits a heist into separate, stackable felonies.
First, artificial intelligence must become enormously capable. Then humans must lose control of it. Then that loss of control must become catastrophic. Then humanity must fail to recover. There is also a second pathway in which humans deliberately misuse artificial intelligence, because any credible extinction model that excludes human stupidity has plainly failed to study humans.
The resulting equation is: P(doom) = 1 − (1 − autonomous-AI doom)(1 − human-assisted AI doom).
After assigning explicit probabilities to transformative capability, loss of control, catastrophic escalation, malicious misuse and recovery, the machine produced 4.225213125 percent. Researchers immediately shortened this to 4.2 percent, partly because further decimal places would be ridiculous and partly because 4.225213125 percent made it sound as though the computer had already booked the venue.
Humanity Now Has a Doom Number and Will Spend Tuesday Arguing About It
The calculation places AI extinction risk near the median figure produced in the large 2023 Expert Survey on Progress in AI, which gathered responses from 2,778 researchers who had published at major AI conferences. Depending on precisely how the extinction question was phrased, median estimates landed around 5 percent or 10 percent. Asked about extinction or similarly severe permanent disempowerment within the next hundred years, the median was 5 percent and the mean was 14.4 percent.
This produced an important scientific discovery: the probability of the destruction of mankind can apparently double when somebody rewrites the questionnaire. Somewhere, a pollster is taking notes.
Humanity reacted calmly. One group announced that a 4.2 percent probability demonstrated an obvious need to stop advanced AI research immediately. Another announced that 4.2 percent was basically zero. A third group launched a Substack. This is what humans do with probabilities.
Tell a man there is a 4.2 percent chance his airplane will crash and he will crawl home through the airport ventilation system. Tell him there is a 4.2 percent chance an artificial intelligence will eventually eliminate civilization and he says, “Yeah, but what are the benchmark scores?”
There is something deeply human about our relationship with tiny probabilities. A 4.2 percent chance of losing your luggage is unacceptable. A 4.2 percent chance of your restaurant meal being poisoned gets the restaurant closed. A 4.2 percent chance of civilization ending apparently gets a panel discussion at Davos scheduled between “The Future of Sustainable Private Equity” and lunch. As American comic Duke Lassiter, a Bay Area open-mic fixture who has never once been booked twice at the same venue, put it: “We’ll debate whether a self-driving car should swerve for a squirrel for forty-five minutes, then let a chatbot set foreign policy on a coin flip.”
Experts Discover Doom Has Several Moving Parts
The principal problem with traditional P(doom) estimates is that one innocent-looking number can conceal an entire warehouse full of assumptions, the statistical equivalent of a storage unit nobody has opened since the move.
Suppose somebody gives a P(doom) of 20 percent. Fine. Does that person believe superhuman AI is almost certain to arrive and relatively easy to control? Or only moderately likely to arrive but impossible to control? Does “doom” mean literal extinction? Does it mean permanent political subordination? Does it mean authoritarian humans using AI? Does it include biological weapons? Does it happen before 2050? Before 2100? Before the Sun expands and renders the entire discussion technically moot?
A single number tells you none of this. It is the statistical equivalent of asking someone, “How likely is dinner to go badly?” Well, who’s cooking? What are we eating? Is Uncle Frank coming? Has Uncle Frank been drinking? Does Uncle Frank have access to autonomous drone systems? Suddenly the conditional probabilities matter quite a bit more than the RSVP did.
AI Researchers Asked to Show Their Work, Industry Declares Emergency
The new Doom Equation therefore requires forecasters to expose their assumptions rather than smuggle them in under a single scary-looking digit.
The current model assigns a 75 percent probability that strategically transformative AI appears by 2100. It then assigns a 12 percent conditional probability that humanity loses meaningful control over such systems. If control is lost, it gives a 55 percent chance that events could become civilization-ending, followed by a 30 percent probability that humans nevertheless recover or contain the situation. That produces an autonomous-AI catastrophe probability of approximately 3.47 percent.
A second pathway estimates catastrophe caused by humans using powerful AI recklessly or maliciously, producing another 0.79 percent before overlap is accounted for. Combined properly, the result is 4.2 percent.
Every one of those assumptions can be challenged. That is precisely the point. A person who thinks 4.2 percent is too high can now say: “I don’t believe the conditional probability of losing control should be 12 percent.” Excellent. We are finally arguing about something. Another person might say: “Your 55 percent escalation estimate is absurd.” Wonderful. Change it. Run the equation again.
Civilization has advanced from people shouting “P(doom) is 50 percent!” to people shouting “your conditional catastrophic-escalation coefficient is poorly calibrated!” This is what progress looks like. It isn’t pretty, but neither was the steam engine, and nobody asked the steam engine to hold a press conference either.
P(doom) Still Suffers From the Awkward Problem of Nobody Knowing
Google DeepMind chief Demis Hassabis has declined to offer his own P(doom), arguing that putting a number on such deep uncertainty implies a precision that simply does not exist. Researchers at Georgetown University’s Center for Security and Emerging Technology have raised a related concern, arguing in recent work that when evidence is sparse and uncertainty is profound, a single probability may be the wrong instrument altogether, a bit like bringing a thermometer to a knife fight.
This is distressing news for civilization because human beings adore single numbers. We have GDP. IQ. BMI. Credit scores. Approval ratings. Star ratings. Golf handicaps. University rankings. Uber ratings. And now, naturally, the percentage chance everybody dies.
Soon your morning financial app may display: S&P 500: +0.7%. NASDAQ: +1.1%. P(doom): 4.2%. Weather: partly cloudy. Investors will become nervous whenever P(doom) rises above mortgage rates. CNBC will run a red ticker reading DOOM UP 0.3% AFTER MODEL LEARNS EXCEL MACROS. Markets will rally after analysts explain that humanity’s extinction was already priced in.
Silicon Valley Introduces Doom Guidance
Technology companies are also expected to begin issuing quarterly P(doom) guidance. “During the third quarter we achieved record revenue, expanded inference capacity 38 percent and unfortunately increased existential-risk guidance from 3.8 to 4.4 percent,” a hypothetical chief executive might tell shareholders. The stock would rise 7 percent. Analysts would describe extinction as “a headwind.” One investment bank would downgrade humanity from Buy to Hold. Another would maintain an Outperform rating on cockroaches, a species with, it must be said, a genuinely enviable track record on risk management.
The bizarre thing about P(doom) is that being extremely worried and extremely optimistic can both be rational positions depending on the underlying assumptions. If transformative AI never emerges, much of the classic existential-risk chain simply evaporates. If it emerges but remains controllable, the risk falls sharply. If it becomes highly autonomous, strategically capable, deceptive, connected to important infrastructure and difficult to shut down, the equation changes quickly, and not in humanity’s favor.
That is why the components matter more than the headline. The 4.2 percent figure should therefore not be read as “AI has calculated that there is exactly a 4.2 percent chance everybody dies.” It should be read as “given these explicit assumptions, this transparent risk tree produces 4.2 percent.” Less exciting. Far more useful. Terrible television.
AI Reassures Public That 95.8 Percent Is Technically Pretty Good
Asked whether humanity should feel reassured by the calculation, the computer reportedly pointed out that a 4.2 percent P(doom) also represents a 95.8 percent P(not doom). This immediately improved public morale until somebody explained that airline passengers would react poorly to a pilot announcing a 95.8 percent probability of reaching Dallas.
The computer then revised its communications strategy. “Look,” it explained, “you people survived leaded petrol, the Cold War, disco, Windows Vista, gas-station sushi and several thousand years of governments. I am not yet the strangest risk-management decision humanity has made.” Economists found this argument persuasive. Philosophers requested another conference. Silicon Valley requested another trillion dollars. And humanity returned to doing what humanity does best: developing extraordinarily powerful technology while arguing about whether the chance it kills everyone is four percent, ten percent or technically unknowable.
The official number remains 4.2 percent. The confidence level in the official number remains approximately: please stop asking.
Sources
- The problem with P(doom) — Financial Times
- 2023 Expert Survey on Progress in AI — AI Impacts
- Beyond P(doom) for AI Risk — Center for Security and Emerging Technology
The 4.2 percent figure comes from a scenario model built around published risk estimates, including the AI Impacts survey of 2,778 researchers and public comments from Google DeepMind CEO Demis Hassabis, who has repeatedly declined to state his own P(doom) on the grounds that a specific figure would overstate how much anyone actually knows. Georgetown’s Center for Security and Emerging Technology has separately argued that AI risk may be better handled with tools like Belief and Plausibility than with a single probability, a point the Financial Times‘ Tim Harford has also raised in his own writing on the subject.
Auf Wiedersehen, amigo!
This is a work of American satire. Nothing in the P(doom) figure above should be mistaken for financial, existential or actuarial advice, and no chatbot has actually been retained as a risk consultant, though at these rates it would still be cheaper than the last one. For more satirical news from across the pond, visit The London Prat.
NOTE: The 4.2% figure in this piece comes from a scenario model, not a lab measurement. Real AI researchers really do disagree on P(doom) by a wide margin, and real experts really have argued that no single number can capture it. Everything after that is satire.
Alan Nafzger was born in Lubbock, Texas, the son Swiss immigrants. He grew up on a dairy in Windthorst, north central Texas. He earned degrees from Midwestern State University (B.A. 1985) and Texas State University (M.A. 1987). University College Dublin (Ph.D. 1991). Dr. Nafzger has entertained and educated young people in Texas colleges for 37 years. Nafzger is best known for his dark novels and experimental screenwriting. His best know scripts to date are Lenin’s Body, produced in Russia by A-Media and Sea and Sky produced in The Philippines in the Tagalog language. In 1986, Nafzger wrote the iconic feminist western novel, Gina of Quitaque. He currently lives in Holloway, North London. Contact: [email protected]
