Existential risk from artificial general intelligence

The existential risk from artificial general intelligence is the idea that advancements in artificial intelligence (AI) technology may one day create an uncontrollable superintelligence that would result in catastrophic consequences for human civilization. These hypothetical consequences range from crippling disruptions in computer networks to the breakdown of society and, in the worst-case scenario, the extinction of humanity. While the idea of an existential risk from AI has raised concerns among scholars and scientists, experts are divided as to how much of a risk, if any, a future superintelligence might pose.

Most experts suggest there are still considerable obstacles to the creation of a superintelligent AI. Some believe it may never be possible. Others suggest that even if science succeeds in creating artificial general intelligence (AGI), the popular concept of a malevolent “machine apocalypse” portrayed in many works of science fiction is still unlikely. Rather, the dangers posed by superintelligent AI would likely come from it developing goals that do not align with the goals of its programmers. The AI could then veer away from its intended function in an unpredictable direction.

Background

The genesis of AI is typically traced back to the 1940s, when scientists invented the first electronic computers capable of storing information. However, people had been interested in the concept of intelligent machines for centuries. Due to technological limitations throughout most of history, such musings were long mostly in the realm of philosophy and fiction. Even so, many thinkers suggested that intelligent machines could pose a major threat to humanity. A famous example is Czech author Karel Čapek's 1920 play R.U.R., which originated the word "robot" and portrays artificial humanoids rising up against humans and essentially replacing them.

In the early 1950s, British mathematician Alan Turing, a pioneer in computing technology, further explored the idea that a machine could be built that could “think” on its own. To Turing, a “thinking” computer would be able to take its programmed information, learn from its use, and employ “reason” to correct itself and make decisions. He conceived of what is now known as the Turing test as a way to judge whether a machine had achieved intelligence. The term artificial intelligence was coined later in the decade to describe a computing machine that could mimic human intelligence. At the time, this type of technology was still far from reality, but the idea began to grow in popularity among computer scientists.

Interest in and research on AI boomed in the early 1970s as computer technology advanced considerably. However, within a few years funding declined sharply as early optimistic predictions of AI breakthroughs fell short. This start-and-stop progress would continue over the next few decades, as new technological developments often spurred fresh excitement about AI only for experts to further narrow the definition of what truly qualified as "intelligence." The concept of artificial general intelligence (AGI) came to represent a hypothetical AI that could not only mimic certain aspects of intelligence, but equal or even surpass human intelligence with some level of autonomy. Meanwhile, philosophers continued to debate the merits and risks of AI, while the concept remained a staple of science fiction.

During the 1980s, scientists developed the concept of deep learning, a type of machine learning in which computers acquire information by following a hierarchy of data. The computer may come to recognize one data point and, through experience, move from that point to recognize other points that become increasingly more complex. AI technology achieved a breakthrough in 1997 when an IBM computer named Deep Blue defeated chess grandmaster Garry Kasparov. Deep Blue was able to analyze Kasparov’s moves, predict their outcome, and construct a strong response. In the first two decades of the twenty-first century, AI continued to improve rapidly thanks to major improvements in computer processing power, data storage, machine learning (including neural networks), and other areas. By the 2020s AI technology was incorporated into many software tools, from word processor spell-checkers to smartphone facial recognition. Indeed, such systems were so commonplace that some observers no longer considered them as AI. On the other hand, the concept of AGI continued to be hotly debated.

Overview

Many experts classify two types of artificial intelligence. The first is often known as weak or narrow AI. This type analyzes data, recognizes patterns, and makes decisions based on what it learns; however, narrow AI can only accomplish the task that it is programmed to perform. For example, self-driving cars can monitor speed and traffic flow, navigate a route, and arrive at a destination but cannot decide on their own to stop and pick up dinner. Most scientists agree that all functioning examples AI into the 2020s were examples of narrow AI. This included virtual assistants like Amazon’s Alexa, face- and voice-recognition technology, and the algorithms used to power search engines.

In contrast, strong AI is artificial intelligence that mimics human thought entirely. It is often considered synonymous to AGI, though definitions vary between sources. According to some observers, AGI is an AI that is equal to the capability of the human mind, while artificial super intelligence (ASI) surpasses the human mind. Regardless of the exact term used, the core criteria is that the AI would be self-aware and possess the ability to learn, solve problems, and anticipate future events without input from humans.

As of the early twenty-first century, most experts continued to suggest humans were nowhere close to developing true AGI. However, significant breakthroughs such as the rise of powerful large language models (LLMs)—which drew much media attention following the debut of the AI chatbot ChatGPT in 2022—led to considerable disagreement about when that threshold might be reached. Some argued it was only years away, others suggested it would take decades, and others asserted that it would not be accomplished within the century. Still another group of experts doubted whether AGI is even possible.

In addition to the time frame for the development of strong AI, experts are also divided over its impact on the future of humanity. Some believe that creating AI that equals or surpasses that of humans can only be beneficial. They see strong AI as potentially revolutionizing health care, food production, transportation, and virtually every other field. The logic behind this positive view is that AI would be programmed by humans and would therefore be constrained by safeguards and other limitations placed upon them. A few thinkers have even supported the idea of "the singularity," a turning point at which machines will become so powerful that humanity will be changed beyond what is even imaginable, essentially representing the next step in human evolution.

However, others have raised major concerns about the potential dangers of AGI. Indeed, such concerns were voiced soon after the concept was first discussed in the 1950s and 1960s. Alan Turing himself predicted that one day, machines would become so advanced that they would “take control.” In 1965, one of Turning’s former colleagues, Irving John Good, suggested that if a machine ever surpassed the capabilities of the human mind, it could, in turn, create even more advanced intelligent machines. This would create an intelligence explosion that would vastly outpace human intelligence.

In the twenty-first century, the noted physicist Stephen Hawking said that highly advanced AI could result in the “end of the human race.” Entrepreneur Elon Musk was another prominent figure who claimed the creation of superintelligent AI would be the greatest existential threat to the future of humankind. In 2023, amid a significant wave of public attention to AI advances, an open letter from the Center for AI Safety signed by hundreds of scientists and business leaders stressed that "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks, such as pandemics and nuclear war." A 2024 report commissioned by the US State Department also warned that human extinction was a worst-case outcome of AI development. Others similarly suggested that the idea of a technological singularity is indeed possible but poses an existential threat rather than a wondrous benefit to humanity.

Such warnings have captured headlines and stirred up ideas previously relegated to works of science fiction. And some critics have indeed suggested the possibility of a future in which robots arise to kill humanity, as in the Terminator films, or enslave humans in a virtual reality like in The Matrix (1999). However, others claim that while AI can develop new ways to process and interpret information, it is unlikely to evolve to become “evil” or “good” as often seen in fiction.

Instead, many experts believe superintelligent AI could become an existential threat in two main ways. The first scenario involves autonomous weapons being misused or built so efficiently that they cannot be turned off. For example, many of today’s larger militaries have automated drones in their arsenals. These drones use AI to seek out a target based on programmed parameters. When a drone acquires a target, it must be given the order to attack by a human controller. However, if this function is hijacked by an outside source, the drone can be used to cause mass casualties or as a weapon of terror. Perhaps even more concerning is the future proliferation of autonomous weapons that could result in an AI-powered arms race. With more and more weapons in use, experts caution that the risk of an accident would increase, as would the possibility of a systems malfunction. Experts also theorize that nations may design the weapons to be resistant to enemy sabotage, and therefore more difficult to just switch off, resulting in a potential loss of control.

The second—and to some, far more plausible—scenario is that superintelligent AI develops different goals than the ones humans originally intended it to have. This outcome has already been witnessed in experiments using narrow AI. For example, an AI programmed to play a role-playing video game hacked into the game’s coding to provide itself with keys and other objects needed to win. The programmers believed that the AI would play the game as intended, but the AI saw that the result could be achieved more quickly if it took a “shortcut.”

Many AI experts therefore caution that it is imperative that the goals of any AI be carefully examined and planned out before the system becomes active. While there is no unanimous agreement on how to solve the problem of human control over the output of AI systems, most observers stress that science should begin focusing on the task now to be ready for the arrival of AGI. Even then, it is extremely difficult to know what path the AI may follow in accomplishing its goals.

In one example, experts compare AI to humans constructing a dam project. The humans may take every precaution to make sure the dam is safe and does not impact the region’s ecosystem. However, if an anthill happens to be in the path of the dam, the ants and their home will be destroyed. The humans may not wish to destroy the ants, but the hill is just too insignificant to stop the project. The risk, according to AI experts, is that AI unintentionally sees the result of a project as too important to let humans stand in its way.

In another hypothetical example, philosopher and author Nick Bostrom sets up a scenario where an AI system is tasked with making paperclips. The machine would make paperclips and learn how to make them more efficiently. The AI would look for new resources to make more and more paperclips until the world had too many to realistically use. At this point, humans may try to turn off the machine, but the AI, being more intelligent than humans, would successfully defend itself. It may see humans as not only a threat to its survival, but also as taking up needed resources for it to accomplish its goal. In this way, an AI given a simple task could end up eradicating humanity.

These examples may be extreme, and although some experts see them as plausible, others believe the true threat from AI may be less apocalyptic. For example, some see AI as impacting the independence of future human societies. Even today, AI algorithms control much online content. The worry is that humans will stop thinking for themselves and cede more control to machine intelligence, allowing AI to make decisions for them. An AI that surpasses the intelligence of humans could hypothetically exert even more control, just as humans exert control over pets and other animals. Alternatively, even AI that remains nominally under human control could seriously destabilize society through algorithmic bias and various forms of misinformation such as deepfakes that erode trust and promote conflict.

Still another worry stems from the growing dependence on AI in the economic sector. As more and more jobs are turned over to artificial intelligence in the future, many human workers may find themselves out of work. This could cause significant economic disruption as entire workforces are replaced by AI. It would vastly increase economic disparities and possibly lead to uprisings or other large-scale unrest. The drastic societal changes could cause chaos resulting in a great loss of life.

Bibliography

Anderson, Janna, and Lee Rainie. “Artificial Intelligence and the Future of Humans.” Pew Research Center, 10 Dec. 2018, www.pewresearch.org/internet/2018/12/10/artificial-intelligence-and-the-future-of-humans/. Accessed 22 Feb. 2022.

“What Is Artificial Intelligence (AI).” IBM, www.ibm.com/topics/artificial-intelligence. Accessed 22 Aug. 2024.

Cheatham, Benjamin, et al. “Confronting the Risks of Artificial Intelligence.” McKinsey Quarterly, 26 Apr. 2019, www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/confronting-the-risks-of-artificial-intelligence. Accessed 22 Feb. 2022.

Christian, Brian. The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company, 2020.

Conn, Ariel. “Benefits & Risks of Artificial Intelligence.” Future of Life Institute, 14 Nov. 2015, futureoflife.org/background/benefits-risks-of-artificial-intelligence/. Accessed 22 Aug. 2024.

Critch, Andrew, and Stuart Russell. "TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI." arXiv, 16 June 2023, arxiv.org/pdf/2306.06924.pdf. Accessed 27 June 2023.

Dafoe, Allan, and Stuart Russell. “Yes, We Are Worried about the Existential Risk of Artificial Intelligence.” MIT Technology Review, 2 Nov. 2016, www.technologyreview.com/2016/11/02/156285/yes-we-are-worried-about-the-existential-risk-of-artificial-intelligence/. Accessed 22 Feb. 2022.

Heaven, Will Douglas. "How Existential Risk Became the Biggest Meme in AI." MIT Technology Review, 19 June 2023, www.technologyreview.com/2023/06/19/1075140/how-existential-risk-became-biggest-meme-in-ai/. Accessed 23 June 2023.

Metz, Cade. "How Could AI Destroy Humanity?" The New York Times, 10 June 2023, www.nytimes.com/2023/06/10/technology/ai-humanity.html. Accessed 23 June 2023.

Naudé, Wim, and Otto Barten. "Artificial General Intelligence: Can We Avoid the Ultimate Existential Threat?" OECD.AI Policy Observatory, 26 Jan. 2023, oecd.ai/en/wonk/existential-threat. Accessed 22 Aug. 2024.

Perrigo, Billy. "Exclusive: US Must Move 'Decisively' to Avert 'Extinction-Level' Threat from AI, Government-Commissioned Report Says." Time, 11 Mar. 2024, time.com/6898967/ai-extinction-national-security-risks-report/. Accessed 22 Aug. 2024.

Russell, Stuart. Human Compatible: Artificial Intelligence and the Problem of Control. Viking, 2019.

Sofge, Erik. “The 8 Evil Forms of AI that Gave Robots a Bad Name.” Popular Mechanics, 28 Jan. 2010, www.popularmechanics.com/technology/robots/a5020/4343893/. Accessed 22 Feb. 2022.

Streitfeld, David. "Silicon Valley Confronts the Idea That the 'Singularity' Is Here." The New York Times, 11 June 2023, www.nytimes.com/2023/06/11/technology/silicon-valley-confronts-the-idea-that-the-singularity-is-here.html. Accessed 23 June 2023.

Vincent, James. "Top AI Researchers and CEOs Warn Against 'Risk of Extinction' in 22-Word Statement." The Verge, 30 May 2023, www.theverge.com/2023/5/30/23742005/ai-risk-warning-22-word-statement-google-deepmind-openai. Accessed 23 June 2023.

Williams, Matthew S. “Technological Singularity: An Impending ‘Intelligence Explosion’.” Interesting Engineering, 8 Aug. 2021, interestingengineering.com/technological-singularity-an-impending-intelligence-explosion. Accessed 22 Feb. 2022.