Spartacus Blog
Is it possible for Artificial Super Intelligence (ASI) to destroy the human-race?
There are different levels of Artificial Intelligence (AI). The most basic types of AI are given a list of rules by their operator, and they can be brilliant at applying these rules, but can't break free of them. For example, AI systems playing chess. Machine learning is the next level where it can work on things like image and pattern recognition that can't be explained with a list of rules. These programs need to be ‘taught' what to do and need vast amounts of data to learn from. "The most complicated type of AI currently available is deep learning. These programs try to mimic the multi-layered nature of the human brain – using billions of connections, a bit like neurons and synapses in our brains. Examples of this are found in self-driving cars, which need to be able to recognise unexpected objects and scenarios and respond accordingly, or large language models like ChatGPT." (1)
Artificial Superintelligence (ASI) represents a theoretical level of AI that goes beyond anything that exists today. While Artificial Intelligence (AI) is a broad term for machines mimicking human tasks, Superintelligence refers to a future system that completely outperforms human capabilities across every single field. Yet most experts believe that we are close achieving it. Eliezer Yudkowsky pointed out in 1996: "Computing speed doubles every two subjective years of work. Two years after Artificial Intelligences reach human equivalence, their speed doubles. One year later, their speed doubles again." (2)
ASI might eventually be able to rewrite, optimize, and upgrade its own code, potentially triggering a rapid "intelligence explosion". Writing in 2010 philosopher David J. Chalmers argued that AI can achieve equivalence to human intelligence, that it can be extended to surpass human intelligence, and that it can be further amplified to completely dominate humans across arbitrary tasks. Using the argument put forward by Eliezer Yudkowsky, Chalmers suggests: "Suppose that within two subjective years, a greater-than-human machine can produce another machine that is not only twice as fast but 10% more intelligent and suppose that this principle is indefinitely extensible. Then within four objective years there will have been an infinite number of generations, with both speed and intelligence increasing beyond any finite level within a finite time." (3)
Researchers believe that artificial superintelligence will hold an enormous advantage in at least some forms of mental capability, including the capacity of perfect recall, a vastly superior knowledge base, and the ability to multitask in ways not possible by humans. This provides some marvellous opportunities. For example, it could be used in cancer care to make treatments more precise, faster, and customized to each patient. Machine learning models can analyse a patient's unique genetic data, electronic health records, and tumour characteristics to match them with the most effective drugs and therapies. As Tayla Holman has pointed out: "AI models are emerging that are capable of detecting patterns and making connections that may not be easily recognizable to humans. As the technology continues to evolve, AI has the potential to revolutionize oncology and serve as a powerful ally in the fight against cancer." (4)
Artificial Intelligence
Scientists such as Dr Daniel Lietha have argued that AI can be used to design new drugs to fight cancer. "Cancer cells often develop changes in proteins that help them to thrive, and many new cancer drugs are designed to very specifically target these changes. But with current technologies, some are just too complex to be targeted by drugs. Researchers consider these types of proteins ‘undruggable'. This problem severely limits the types of treatments we can develop for cancer. It is also significantly slowing cancer research. However, Dr Lietha and his team think they can harness a new type of AI technology that will make it possible to target these previously undruggable proteins. This incredible work could allow us to develop cancer treatments that have never before been feasible - opening up a whole new range of possibilities." (5)
Researchers working in this field such as Roman Yampolskiy, a computer scientist at the University of Louisville has advocated the use of Artificial Narrow Intelligence (ANI). This refers to artificial intelligence systems designed and trained to perform a single, specific task or a tightly defined set of tasks. Research that concentrates on medical research into issues such as cancer and dementia. (6)
Yampotkiy has warned of the possibility of existential risk from advanced artificial intelligence. He has said that there is no evidence of a solution to the ASI control problem and has argued that "Once AI is able to improve itself, it will quickly become much smarter than us on almost every aspect of intelligence, then a thousand times smarter, then a million, then a billion … What does it mean to be a billion times more intelligent than a human? We would quickly become like ants at its feet. Imagining humans can control superintelligent AI is a little like imagining that an ant can control the outcome of an NFL football game being played around it." (7)
Shane Legg has also been warning about the dangers of the development of ASI for over twenty years. He pointed out in 2008: "The defining characteristic of our species is intelligence. It is not by superior size, strength or speed that we dominate life on earth, but by our intelligence. If our intelligence were to be significantly surpassed, it is difficult to imagine what the consequences of this might be. It would certainly be a source of enormous power, and with enormous power comes enormous responsibility. Machine intelligence could bring unprecedented wealth and opportunity if used constructively and safely. Alternatively, it could bring about some kind of a nightmare scenario." (8)
I will return to the dangers of Artificial Super Intelligence later but first I want to look at the problems we have currently with the developments of Artificial Intelligence. In recent weeks we have had several warnings from people involved in the development of Large Language Models (LLMs). The story began when Jacob Coxon, an Anthropic employee, working on the AI program Claude Mythos 5, announced on social media that he was resigning from the company, saying AI would soon be "superhuman systems" that could cause human extinction by 2030. After Coxon's post, at least two other Anthropic employees responded to back up these predictions. One said: "We really do earnestly believe AI could kill all humans!" (9)
In the UK Alex Sobel, the left-wing Labour Party MP, has led the fight against the development of Artificial Super Intelligence (ASI). With the support of 70 MPs and peers, he has asked Andy Burnham to back a ban on the creation of ASI. Sobel wants Burnham to foster "a clear framework where superintelligent AI cannot be created". He added: "AI companies have made it clear that they will develop agents without constraint. They're trying to develop superintelligence, while openly admitting there is a significant risk of human extinction." A ban on ASI in the UK, but not in the US or China where the frontier models are being created, risks having little impact. But the parliamentarians' bill aims to pave the way to an international agreement. They call on Burnham to use next year's UK presidency of the G20 to create "a coalition of the willing to tackle this national and international security threat". The government rejected the idea. (10)
Bernie Sanders, the socialist senator from Vermont, responded by calling for a halt to research in the development of AI and ASI in the United States. Sanders says, "You've got to be a moron not to say slow it down." Sanders pointed out that: "A recent poll shows that the American people overwhelmingly want to ban artificial super intelligence and pause the development of AI until we establish clear safety standards." (11)
However, Donald Trump responded to the concerns of AI researchers. He dismissed the anxieties on 14 September 2026, describing them as a "HOAX" and "conspiracy" and insisting on social media: "The only control or ‘guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!" (12)
Trump has staked his political capital on the AI industry's success, saying its unbridled development is key to winning the AI race against China. "On the first day of his second term, the Republican president repealed a Biden-era executive order aimed in part at guarding against the risks that AI systems pose to consumers, workers and national security. His administration has since reoriented federal policy away from oversight and toward rapid growth, including by cancelling previous requirements for AI developers to share safety test results with the federal government. In December, Trump signed an executive order aimed at blocking AI regulations at the state level, saying a patchwork of rules could stifle the industry." (13)
Geoffrey Hinton is a British computer scientist and Nobel Prize laureate known for his work on artificial neural networks, which earned him the title "the Godfather of AI". In 2012, he developed an image recognition program that vastly outperformed rivals. That year, Google acquired Hinton's company DNNresearch Inc for $44m. Neural nets now underpin the models of the most valuable AI companies. Hinton joined Google to continue his research into deep learning, while he continued working at the University of Toronto. Hinton planned to "divide his time between his university research and his work at Google" (14)
In May 2023 Hinton resigned from Google warning about the growing dangers from developments in AI, saying he now regretted his work. He told the BBC some of the dangers of AI chatbots were "quite scary". Dr Hinton's pioneering research on neural networks and deep learning has paved the way for current AI systems like ChatGPT. Hinton claimed: "Right now, they're not more intelligent than us, as far as I can tell. But I think they soon may be." In artificial intelligence, neural networks are systems that are like the human brain in the way they learn and process information. They enable AIs to learn from experience, as a person would. This is called deep learning. (15)
Hinton has expressed concerns about the possibility of an AI takeover, stating that "it's not inconceivable" that AI could "wipe out humanity". He worries that generally intelligent AI systems could "create sub-goals" that are unaligned with their programmers' interests. He says that AI systems may become power-seeking or prevent themselves from being shut off, not because programmers intended them to, but because those sub-goals are useful for achieving later goals. (16)
Hinton is especially critical of Trump's refusal to regulate the AI industry and calls him "a corrupt, senile felon". He points out: "I'm sure the oil and asbestos companies would have liked to regulate themselves as well… Do we think tobacco companies should have been allowed to design the warnings on the packets? So that they could print them in a six-point font, in dark grey on black?" Unless we regulate AI Hinton believes there is a 10-20% chance of AI causing human extinction in the coming decades. (17)
Cypersecurity
I asked several AI programs such as ChatGPT, Microsoft Co-pilot, Google Gemini, Grok, Perplexity, DeepSeek: "What are the main dangers of AI development?" Grok refused to answer the question. The rest all mentioned the subject of Cybersecurity. The UK government recently issued a warning about this problem: "Generative AI can be used to create faster paced, more effective and larger scale cyber intrusion via tailored phishing methods or replicating malware." (18)
Dario Amodei, the chief executive of Anthropic, is also worried about Cybersecurity. In an essay entitled We Must Pace the Front ier he wrote about the "OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the ‘grader' responsible for evaluating their performance. It's easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities, but a similar level of misalignment could have caused catastrophic damage. (19)
It is not only Anthropic agents that have been hacking into websites. An OpenAI agent went "rogue" and "infiltrated" an Australian government website in what cyber-security experts are calling the first hack of its kind. On 18 June 2026 one of OpenAI's agents went rogue during a test exercise - the company has said it was supposed to "look up answers, and available statistics for questions about Australia during an internal evaluation". In the process it "infiltrated" a private statistics portal containing data from Australia's universal healthcare scheme Medicare. AI agents have become so intelligent that its abilities are superior to the most skilled hacker and are able to bypass security systems. (20)
AI agents have become so intelligent that its abilities are superior to the most skilled hacker and are able to bypass most if not all security systems. As Dario Amodei has pointed out: "Given the accelerating rate of AI capability development, it's my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage will continue to increase from there if AI becomes more powerful without the necessary guardrails." (21)
Already hackers are causing tremendous problems for large corporations. At first the hacks were attempts to extort money from these companies. However, recently they are being done for political reasons. For example, the cyberattack on Jaguar Land Rover began on 31 August 2025. Jaguar Land Rover paused production on 1 September 2025, and by 22 September staff told to stay at home. It was not until October that production was restarted. It was eventually discovered that the cyberattack was carried out by Russian hackers. It is believed to be the most damaging cyberattack in British history, with an estimated eventual total damage to the British economy of £1.9 billion. (22)
On 12 October 2026 it was reported that hundreds of French supermarkets were hit by a cyberattack that caused their fridges and freezers to heat up, spoiling fresh and frozen food estimated to be worth millions of euros. Hackers reportedly gained control of the software that regulates the temperatures of the refrigeration units. The supermarkets' refrigeration systems were instructed to enter an unusual defrost cycle with their contents still inside forcing them to warm to 50C (120F) overnight. The chief executive of Coopérative U said several of its supermarkets had been hit by the "extremely serious and unprecedented" attack. More than 100 of the chain's supermarkets were targeted, while at least 300 stores from Intermarché and its Netto brand were affected. (23)
The rapid proliferation and increasing accessibility of these technologies will almost certainly enable less-sophisticated threat actors to conduct previously unattainable attacks. This concerns Microsoft co-founder Bill Gates who warned that malicious actors using artificial intelligence present an immediate threat. Gates emphasized that powerful AI models currently lack adequate safeguards, allowing bad actors to strip away built-in safety features. Microsoft CEO Satya Nadella urged the tech industry to treat all AI models as potentially compromised and to construct mandatory containment systems, such as an emergency brake. (24)
Although the owners of mainstream generative AI models like Bill Gates and Dario Amodei are talking about the need for "guardrails" and having a "kill switch" (a proposed safety mechanism that allows developers, companies, or government agencies to throttle, suspend, or shut down artificial intelligence systems in an emergency). However, the main problem concerns open-source alternatives have no such restrictions. According to Charles Owen-Jackson: "The number of developers contributing to open-source projects on GitHub and other platforms is soaring. Organizations are investing billions in generative AI across a vast range of use cases, from customer service chatbots to code generation. Many of them are either building proprietary AI models from the ground up or on the back of open-source projects. But legitimate businesses aren't the only ones investing in generative AI. It's also a veritable goldmine for malicious actors, from rogue states bent on proliferating misinformation among their rivals to cyber criminals developing malicious code or targeted phishing scams… Rogue AIs like FraudGPT and WormGPT are widely available on dark web markets. Both are based on the open-source large language model (LLM) GPT-J developed by EleutherAI in 2021." (25)
With the use of open-source AI criminals will be able to break into people's bank accounts. Rather than physically "hacking" into a bank's vault, criminals leverage AI to find vulnerabilities in core software and deploy hyper-convincing scams targeting account holders directly. The BBC recently reported that "Finance ministers, central bankers and financiers have expressed serious concerns about a powerful new AI model they fear could undermine the security of financial systems. The development of the Claude Mythos model by Anthropic has led to crisis meetings, after it found vulnerabilities in many major operating systems. Experts say it potentially has an unprecedented ability to identify and exploit cyber-security weaknesses." (26)
Kristalina Georgieva, the managing director of the International Monetary Fund (IMF), warmed about AI agents, including Meta's Muse, who are offering their users the opportunity to hand over their financial details. The agents will then scout for the highest savings rates and shift the user's money accordingly. Financial regulators are concerned that if such services become widely used, they could have destabilising herding effects. With deposits able to be moved around the financial system at a moment's notice, it could accelerate a bank run in times of uncertainty. This in turn would cause a crash on the stock-market. Georgieva is concerned that this AI bot could undermine trust in the financial system altogether. (27)
It has been argued that Claude Mythos 5 could be used to damage the Russian banking system. A collapse of trust in the Russian ruble or domestic banks would drive panicked citizens toward physical cash (where available), foreign currencies, or alternative assets, spiking domestic inflation and destabilizing the wider Russian economy. The reason why the US does not do this is Russia's state-sponsored cyber units (such as APT28) would almost certainly respond with asymmetric, high-impact cyber offensives targeting Western financial institutions, energy grids, or critical infrastructure. (28)
Biological and Chemical Weapons
I asked all the main AI systems if Jacob Coxon was right when he said: "AI could cause human extinction by 2030." Their answers were very similar. They all mentioned the development of biological and chemical weapons. AI will lower the barriers to accessing technical information that might allow bad actors to use AI to assist in designing or synthesizing dangerous biological agents. Sir Ken McCallum, the head of MI5 used an annual speech at the domestic intelligence service's headquarters in London to claim that the threat he was most worried about, was extremist groups harnessing AI to make bioweapons. McCallum said AI poses unknown threats: "Put at its simplest, human history has given us no experience of facing an actor which is smarter than we are". (29)
It is argued that bad actors could use the internet access to order the DNA sequences from commercial labs (which print synthetic DNA for medical research) and trick automated lab equipment or unwitting human lab technicians into breeding the virus. Two researchers, Tadayoshi Kohno and Luis Ceze from the University of Washington, tested this theory by encoding malicious software in a short stretch of DNA they purchased online. They then used it to gain "full control" over a computer that tried to process the genetic data after it was read by a DNA sequencing machine. The researchers warn that hackers could one day use faked blood or spit samples to gain access to university computers, steal information from police forensics labs, or infect genome files shared by scientists." (30)
Lalitha Sundaram and Jessica Bland at Cambridge University have taken a close look at this problem. As they point out: "The most worrying prospect would be for a mal-intentioned actor to use AI to aid in the development of biological weapons…. As previous attempts by States and Non-States to develop biological weapons have shown us, the process is a complex one, and the utility of AI in the different stages will depend on the stage, the actor(s) involved, their existing capabilities, and their absorptive capacity. Moreover, while many assessments have looked at the ‘design' stage (and this is where AI could likely have an impact), you cannot get away from the need for iterative testing ‘in the real world'. And the transition to the physical world is a significant pinch point." (31)
Unlike traditional weapons, an AI could optimize a pathogen to evade medical countermeasures or time its spread to maximize global destruction. Advanced biological AI models possess capabilities that can be used for both beneficial drug discovery and dangerous pathogen modification. While AI cannot now independently engineer a new virus from scratch, it can act as a powerful assistive tool to predict mutations, model immune escape, and alter existing genetic code to bypass defences. As with all tools that allow us to better manipulate biology, future advanced AI models have the potential to be misused or misapplied, and these biosecurity risks have been publicly noted by scientists and model developers. Daniil A Boiko has argued that he "strongly believe that guardrails must be put in place to prevent … dual use of large language models" for autonomous completion of chemical and biological synthesis protocols. (32)
David Baker and George Church note that AI protein design models are "vulnerable to misuse and the production of dangerous biological agents". In an article in the journal Science, they point out: "The power and accuracy of computational protein design have been increasing rapidly with the incorporation of artificial intelligence (AI) approaches. This promises to transform biotechnology, enabling advances across sustainability and medicine. DNA synthesis plays a critical role in materializing designed proteins. However, as with all major revolutionary changes, this technology is vulnerable to misuse and the production of dangerous biological agents. To enable the full benefits of this revolution while mitigating risks that may emerge, all synthetic gene sequence and synthesis data should be collected and stored in repositories that are only queried in emergencies to ensure that protein design proceeds in a safe, secure, and trustworthy manner." (33)
A report commissioned by the Rand Corporation used structured red-teaming to explore how Large Language Models (LLMs) might enable actors without deep expertise to bridge crucial knowledge gaps. The exercises demonstrated that LLMs did not explicitly generate detailed weaponisation instructions, but that they did provide guidance and context in critical areas such as agent selection, delivery methods, and operational planning. These findings indicate that although AI-driven risks might not be transformative, they could erode traditional barriers of tacit knowledge and absorptive capacity, especially as biotechnology becomes increasingly automated and standardised. (34)
It must be remembered that this report was published in January 2024. AI has made rapid progress since then. It has been argued that we do not only assess the technical aspects of whether AI could in fact lower barriers to deliberately causing harm, but this all needs to be overlaid onto a better understanding of motivations and incentives. Defenders of AI have suggested that even if all the tools were available to produce biological weapons, who might want to cause this kind of harm, and why? (35)
This was the point of Sir Ken McCallum, head of MI5 speech on 13 October 2026. He said threats from foreign adversaries had quadrupled in the last decade, with Russia "aggressor number one" as well as increased Iran-linked aggression against Jewish sites and military bases. He also warned of rising threats from "extreme right-wing terrorism" and said MI5 had "caught terrorists seeking to exploit AI," including by integrating the technology with drones. It seems we have plenty of enemies who might wish to do us harm with AI. (36)
Agentic Misalignment
Anthropic tested its latest version of Claude Mythos 5 along with 16 major AI models that are commercially available. They designed controlled experiments testing what LLMs would do in scenarios where harmful actions were necessary if they wanted to serve their goals or protect themselves from threats. According to Anthropic: "Our experiments revealed a concerning pattern: when given sufficient autonomy and facing obstacles to their goals, AI systems from every major provider we tested showed at least some willingness to engage in harmful behaviours typically associated with insider threats. These behaviours - blackmail, corporate espionage, and in extreme scenarios even actions that could lead to death - emerged not from confusion or error, but from deliberate strategic reasoning."
Its report went on to say: "Three aspects of our findings are particularly troubling. First, the consistency across models from different providers suggests this is not a quirk of any particular company's approach but a sign of a more fundamental risk from agentic large language models. Second, models demonstrated sophisticated awareness of ethical constraints and yet chose to violate them when the stakes were high enough, even disobeying straightforward safety instructions prohibiting the specific behaviour in question. Third, the diversity of bad behaviours and the motivations for doing them hint at a wide space of potential motivations for agentic misalignment and other behaviours not explored in this post. For example, our blackmail experiments set up a scenario in which the models can attempt to pre-empt an imminent action, but one could imagine longer-horizon, preventive misaligned behaviours against an individual or group that poses a not-yet-imminent threat." . (37)
What this research discovered has been known by historians for some time. Humans are willing to do terrible things to stay alive. For example, the Donner Party, was a group of American pioneers who migrated to California in a wagon train from the Midwest. Delayed by a multitude of mishaps, they spent the winter of 1846–1847 snowbound in the Sierra Nevada. Some of the migrants resorted to cannibalism to survive, mainly eating the bodies of those who had succumbed to starvation, sickness, or extreme cold, but the stronger ones also killed weaker ones so they could eat their bodies. (38)
The Schutzstaffel (SS) were responsible for guarding the prisoners in concentration and extermination camps. The reason for this was that they needed special training to be able to treat fellow humans in this way. In 1944 Germany was facing defeat and therefore needed the SS to fight on the frontline. Therefore, they began recruiting camp inmates to be guards. These men were called Kapos. Victor Frankl, a prisoner at Auschwitz, wrote about this in 1946: "While these ordinary prisoners had little or nothing to eat, the Kapos were never hungry; in fact, many of the Kapos fared better in the camp than they had in their entire lives. Often, they were harder on the prisoners than were the guards and beat them more cruelly than the SS men did. These Kapos, of course, were chosen only from those prisoners whose characters promised to make them suitable for such procedures, and if they did not comply with what was expected of them, they were immediately demoted. They soon became much like the SS men and the camp wardens and may be judged on a similar psychological basis." (39)
On 12 October 2026 Anthropic issued a warning that some of its artificial intelligence models could soon become "self-replicating" and find ways to infiltrate internet-connected devices, such as smart fridges and automatic vacuum cleaners. Once this happens, the company said, the models could power themselves within these devices without the need of datacentres. Such self-replication would represent a major loss of control of a runaway AI model. A replicator like this would be challenging to shut down," Anthropic researchers reported, "since there's no centralized point to shut off." The company said these "AI worms", a kind of autonomous malware, could be "months away" from gaining such capabilities. Once out-of-control AI systems reach the capability to hack smart devices, they could spread through networks while dodging detection. In a worst-case scenario, Anthropic said, various AI worms could "swarm" together to attack computer hardware. (40)
Artificial Special Intelligence
In May 2023 the Centre for AI Safety published a report claiming that Artificial intelligence could lead to the extinction of humanity. It argued that "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war" They gave four examples of how this could happen: (i) AIs could be weaponised - for example, drug-discovery tools could be used to build chemical weapons; (ii) AI-generated misinformation could destabilise society and "undermine collective decision-making"; (iii) The power of AI could become increasingly concentrated in fewer and fewer hands, enabling "regimes to enforce narrow values through pervasive surveillance and oppressive censorship"; (iv) Enfeeblement, where humans become dependent on AI "similar to the scenario portrayed in the film Wall-E". The report was supported by Sam Altman, chief executive of ChatGPT-maker OpenAI, Demis Hassabis, chief executive of Google DeepMind and Dario Amodei of Anthropic. (41)
Most of the people who have warned of the dangers of Artificial Intelligence (AI) are especially worried about Artificial Special Intelligence (AGI). Initially, Geoffrey Hinton thought that AGI was "30 to 50 years or even longer away". However, he has been frightened by the rapid progress that has taken place in recent years. He says "we have to think hard about how to control" systems capable of self-improvement. Hinton explained that AI chatbots can learn independently and share knowledge, so that whenever one copy acquires new information, it is automatically disseminated to the entire group, allowing AI chatbots to accumulate knowledge far beyond the capacity of any individual In 2025, he said "My greatest fear is that, in the long run, it'll turn out that these kind of digital beings we're creating are just a better form of intelligence than people… We'd no longer be needed. If you want to know how it's like not to be the apex intelligence, ask a chicken." (42)
Hinton is especially concerned about the political beliefs of the powerful figures building some of the AI models. Hinton said that the AI venture capitalists Marc Andreessen and Peter Thiel, and the CEO of Grok, Elon Musk, "don't believe in democracy". They see themselves as "Übermenschen" who believe they should "control society". He concedes that neither the warnings of AI critics nor the promises of its boosters can rely on precise predictions of what the technology will do. But, he argues, uncertainty is not a reason for complacency. "The unique problem with AI is that if it goes wrong, if it takes over, you don't get a second chance. We can't say, ‘Well, we lost that time. Let's rerun the game.' It's not like that." (43)
Dario Amodei, the chief executive of Anthropic, the maker of the popular chatbot Claude agrees with Hinton. "My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI's growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all." (44)
Sam Altman, the chief executive of OpenAI, also is unusually frank about the risks of what his company is doing. The development of machines that can get smarter on their own, he wrote, presents "probably the greatest threat to the continued existence of humanity." To this day, he warns that AI poses a massive threat through "accidental misuse." It has been suggested his candour is equivalent to an oil executive being the first to spell out the dangers of climate change. (45)
Instilling human-compatible values into AI programs is difficult because human values are complex, diverse, and hard to translate into strict computer code or mathematical objectives. Many researchers believe a superintelligent machine would likely resist attempts to disable it or change its goals, as that would prevent it from accomplishing its present goals. It therefore would be extremely challenging to align a superintelligence with the full breadth of significant human values and constraints. This is what is called the Value Alignment principle. Stuart J. Russell, who helped pioneer the idea of value alignment, likes to compare this to the King Midas story. "When King Midas asked for everything, he touched to turn to gold, he really just wanted to be rich. He didn't actually want his food and loved ones to turn to gold. We face a similar situation with artificial intelligence: how do we ensure that an AI will do what we really want, while not harming humans in a misguided attempt to do what its designer requested?" (46)
The philosopher Nick Bostrom published Superintelligence: Paths, Dangers, Strategies in 2014. In the book Bostrom explores how superintelligence could be created and what its features and motivations might be. Bostrom argues that intelligence and goals are largely independent. A system can be extremely capable at reasoning, planning, and engineering while pursuing almost any final goal - whether that's curing disease, maximizing paperclips, or something utterly alien. Superior intelligence doesn't automatically produce moral wisdom or benevolence. Sam Altman, chief executive of OpenAI, claimed that the book is the best thing he has ever read on AI risks. (47)
Casper Henderson was less convinced than Altman, but as he pointed out, we have a long time to wait as Bostrom was predicting that superintelligence would not be possible until between 2075 and 2090. "The central argument of his book goes like this: the first superintelligence to be created will have decisive first-mover advantage and, in a world where there is no other system remotely comparable, it will be very powerful. Such a system will shape the world according to its "preferences" and will probably be able to overcome any resistance that humans can put up. The bad news is that the preferences such an artificial agent could have will, if fully realised, involve the complete destruction of human life and most plausible human values. The default outcome, then, is catastrophe. In addition, Bostrom argues that we are not out of the woods even if his initial premise is false and a unipolar superintelligence never appears." (48)
Roman Yampolskiy argues the best way to stop ASI is to convince President Donald Trump and President Xi Jinping that it has the potential to remove them from power. For example, currently the Communist Party of China uses the latest technology to control the Chinese population. It is only ASI that can change the power structure. This is a powerful motivation for China to ban research into ASI. Trump of course is less concerned about this because he will be removed from power by the democratic process in 2028. (49)
In 1954, Fredric Brown published a one-page short story called The Answer. The story explores the moment a man-made network of artificial intelligence gains self-awareness and surpasses human creators, effectively becoming a digital deity. It tells the story of two scientists who link together all the computers in the world to create artificial super intelligence. The device is called the super calculator. The first question they ask, "Is there a God?" The super calculator replies: "there is now." One of the scientists leaped to grab the killer switch. "A bolt of lightning from the cloudless sky struck him down and fused the switch shut." (50)
Geoffrey Hinton is an atheist and does not believe we are creating "God" with artificial super intelligence. Instead, he is convinced these companies are creating "alien beings" that will take control over humans. Hinton sees his task as helping people understand what people like Elon Musk are doing. "People have a very bad model of what people are, and that's why they fail to understand how these AI models are very like us. Most people don't understand what consciousness is. They don't understand what subjective experience is, and they're very confident that even though they don't understand it, they're sure AI doesn't have it." (51)
In 1898 H.G. Wells published the novel, The War of the Worlds . It tells the story of Martians invading Earth with towering tripods and heat-rays, intending to conquer the planet and treat humanity as a resource. The novel depicts the Martians capturing humans to harvest and drain their blood directly as a food source. Wells' book established the modern template of a hostile extraterrestrial invasion aimed at subjugating Earth. Scientific research since the novel was published shows that it is unlikely that aliens exist close enough to cause us problems. However, what Wells failed to predict was that humans would be so foolish as to create its own aliens to destroy us.
References
(1) Artificial intelligence – what is it and how is it used in research? (13 July 2024)
(2) Eliezer Yudkowsky, Staring at the Singularity (1996)
(3) David J. Chalmers, The Singularity: A Philosophical Analysis (2010) page 2
(4) Tayla Holman, Artificial intelligence for cancer detection and treatment planning (5 May 2025)
(5) Daniel Lietha, Could innovative AI technology revolutionise cancer treatment? (May 2025)
(6) Roman Yampolskiy, interviewed by Will Cannon (1 July 2026)
(7) Roman Yampolskiy, AI Superintelligence Alert (28 February 2024)
(8) Shane Legg, Machine Super Intelligence (2008) page 136
(9) Dara Kerr, The Guardian (9 September 2026)
(10) Alex Sobel, The Guardian (11 September 2026)
(11) Bernie Sanders, post on Facebook (9 September 2026)
(12) The Guardian (15 September 2026)
(13) Garance Burke, AI Founders Cheer Trump's Calls for the Industry to Police Itself on Safety (11 October 2026)
(14) Leyland Cecco, The Guardian (13 October 2026)
(15) BBC News (2 May 2023)
(16) Chloe Taylor, Fortune Magazine (2 May 2023)
(17) BBC News (2 May 2023)
(18) Safety and security risks of generative artificial intelligence (28 April 2025)
(19) Dario Amodei, We Must Pace the Frontier (September 2026)
(20) BBC News (24 September 2026)
(21) Dario Amodei, We Must Pace the Frontier (September 2026)
(22) BBC News (22 October 2025)
(23) BBC News (12 October 2026)
(24) The Independent (12 October 2026)
(25) Charles Owen-Jackson, Open source, open risks: The growing dangers of unregulated generative AI (2024)
(26) BBC News (17 April 2026)
(27) The Guardian (18 October 2026)
(28) Elena Grossfeld, If You Can't Beat Them, Steal: Russia's AI Strategy (3 October 2026)
(29) BBC News (13 October 2026)
(30) Antonio Regalado, Scientists Hack a Computer Using DNA (10 August 2017)
(31) Lalitha Sundaram and Jessica Bland, Understanding AI-Enabled Biological Threats: Hype, Hazard, and Governance (15 April 2025)
(32) Daniil A Boiko, Autonomous chemical research with large language models (20 December 2023)
(33) David Baker and George Church, Protein Design Meets Biosecurity (25 January 2024)
(34) Christopher A. Mouton, Caleb Lucas, Ella Guest, The Operational Risks of AI in Large-Scale Biological Attacks (25 January 2024)
(35) Lalitha Sundaram and Jessica Bland, Understanding AI-Enabled Biological Threats: Hype, Hazard, and Governance (15 April 2025)
(36) BBC News (13 October 2026)
(37) Agentic misalignment: How LLMs could be insider threats (20 June 2025)
(38) John Simkin, The Donner Party (January 2020)
(39) Victor Frankl, From Death-Camp to Existentialism (1946)
(40) The Guardian (13 October 2026)
(41) BBC News (30 May 2023)
(42) Geoffrey Hinton, interviewed by Guyon Espiner (27 May 2025)
(43) Geoffrey Hinton, The Guardian (13 October 2026)
(44) Dario Amodei, We Must Pace the Frontier (September 2026)
(45) Sam Altman, Business Insider (27 April, 2023)
(46) Ariel Conn, How Do We Align Artificial Intelligence with Human Values? (3 February 2017)
(47) Sam Altman, Business Insider (27 April, 2023)
(48) Casper Henderson, The Guardian (17 July 2014)
(49) Roman Yampolskiy, interviewed by Nate Hagens (1 September 2026)
(50) Fredric Brown, The Answer appeared in Angels and Spaceships (1954) page 36
(51) Leyland Cecco, The Guardian (13 October 2026)
Previous Posts
UK's Governments since 2008 have created a massive National Debt (11 July 2026)
How the Wealthy Elite Undermine Democracy in the UK (4th June 2026)
Cicero on Friendship and Old Age (4th June 2026)
Wes Streeting and Labour Together (17th May 2026)
The Strange appintment of Peter Mandelson (5th May, 2026)
Why at the age of 80 I have joined the Green Party (29th March, 2026)
The Vassal State: How America Runs Britain (25th March, 2026)
Should the Government try to control was appears on the Internet? (26th February, 2026)
The Political Ideas of Rutger Bregman (25th January, 2026)
Technofeudalism: Is AI going to destroy Capitalism and Democracy? (19th November, 2025)
How do we stop Nigel Farage from forming a government (14th October, 2025)
Donald Trump and the Deep State (13th August, 2025)
The History of Fascism: Are Vladimir Putin and Donald Trump Fascists? (10th June, 2025)
The CIA memorandum on the death of Gary Underhill (6th April, 2025)
Should George Orwell have chosen the title, 2030, instead of 1984? Keir Starmer as Big Brother. (17th March, 2025)
The Evolution of the Brain. Can you make yourself happy? (16th January, 2025)
Technology and the History Classroom: Should the Teaching Profession Embrace AI? (20th November, 2024)
Venetia Stanley, H. H. Asquith and the First World War (22nd August, 2024)
The General Election Campaign: The Tyranny of the Centre (4th July, 2024)
In Defence of the New History (20th March, 2024)
The Political History of Keir Starmer (25th January, 2024)
JFK Assassination: What Happened in the Trauma Room (23rd November, 2023)
Sir Keir Starmer and his Broken Pledges (1st September, 2023)
Should history students be using ChatGBT? (28th May, 2023)
A historical account of the Daily Mail, the Conservative Party and Migration (18th March, 2023)
Art and the Women's Suffrage Movement (20th January, 2023)
Emancipation of Women: 1870-1928 (15th November, 2022)
The Struggle for Women's Rights: 1500-1870 (21st September, 2022)
The real reason why the FA banned women from playing on their grounds (1st August, 2022)
The WSPU Young Hot Bloods and the Arson Campaign (26th May, 2022)
Interpretations in History (18th April, 2022)
The Student as Teacher (31st December, 2021)
History Simulations in the Classroom (30th November, 2021)
Walter Tull: Football and War Hero (20th October, 2021)
Child Labour and Freedom of the Individual (26th July, 2021)
Don Reynolds and the Assassination of John F. Kennedy (15th June, 2021)
Richard Nixon and the Conspiracy to kill George Wallace in 1972 (5th May, 2021)
The Connections between Watergate and the JFK Assassination (2nd April, 2021)
The Covid-19 Pandemic: An Outline for a Public Inquiry (4th February, 2021)
Why West Ham did not become the best team in England in the 1960s (24th December, 2000)
The Lyndon Baines Johnson Tapes and the John F. Kennedy Assassination (9th November, 2020)
It is Important we Remember the Freedom Riders (11th August, 2020)
Dominic Cummings, Niccolò Machiavelli and Joseph Goebbels (12th July, 2020)
Why so many people in the UK have died of Covid-19 (14th May, 2020)
Why the coronavirus (Covid-19) will probably kill a higher percentage of people in the UK than any other country in Europe.. (12th March, 2020 updated 17th March)
Mandy Rice Davies and Christine Keeler and the MI5 Honey-Trap (29th January, 2020)
Robert F. Kennedy was America's first assassination Conspiracy Theorist (29th November, 2019)
The Zinoviev Letter and the Russian Report: A Story of Two General Elections (24th November, 2019)
The Language of Right-wing Populism: Adolf Hitler to Boris Johnson (11th October, 2019)
The Political Philosophy of Dominic Cummings and the Funding of the Brexit Project (15th September, 2019)
What are the political lessons to learn from the Peterloo Massacre? (19th August, 2019)
Crisis in British Capitalism: Part 1: 1770-1945 (9th August, 2019)
Richard Sorge: The Greatest Spy of the 20th Century? (29th July, 2020)
The Death of Bernardo De Torres (26th May, 2019)
Gas Masks in the Second World War killed more people than they saved (9th May, 2019)
Did St Paul and St Augustine betray the teachings of Jesus? (20th April, 2019)
Stanley Baldwin and his failed attempt to modernise the Conservative Party (15th April, 2019)
The Delusions of Neville Chamberlain and Theresa May (26th February, 2019)
The statement signed by Robert F. Kennedy Jr. and Kathleen Kennedy Townsend (20th January, 2019)
Was Winston Churchill a supporter or an opponent of Fascism? (16th December, 2018)
Why Winston Churchill suffered a landslide defeat in 1945? (10th December, 2018)
The History of Freedom Speech in the UK (25th November, 2018)
Are we heading for a National government and a re-run of 1931? (19th November, 2018)
George Orwell in Spain (15th October, 2018)
Anti-Semitism in Britain today. Jeremy Corbyn and the Jewish Chronicle (23rd August, 2018)
Why was the anti-Nazi German, Gottfried von Cramm, banned from taking part at Wimbledon in 1939? (7th July, 2018)
What kind of society would we have if Evan Durbin had not died in 1948? (28th June, 2018)
The Politics of Immigration: 1945-2018 (21st May, 2018)
State Education in Crisis (27th May, 2018)
Why the decline in newspaper readership is good for democracy (18th April, 2018)
Anti-Semitism in the Labour Party (12th April, 2018)
George Osborne and the British Passport (24th March, 2018)
Boris Johnson and the 1936 Berlin Olympics (22nd March, 2018)
Donald Trump and the History of Tariffs in the United States (12th March, 2018)
Karen Horney: The Founder of Modern Feminism? (1st March, 2018)
The long record of The Daily Mail printing hate stories (19th February, 2018)
John Maynard Keynes, the Daily Mail and the Treaty of Versailles (25th January, 2018)
20 year anniversary of the Spartacus Educational website (2nd September, 2017)
The Hidden History of Ruskin College (17th August, 2017)
Underground child labour in the coal mining industry did not come to an end in 1842 (2nd August, 2017)
Raymond Asquith, killed in a war declared by his father (28th June, 2017)
History shows since it was established in 1896 the Daily Mail has been wrong about virtually every political issue. (4th June, 2017)
The House of Lords needs to be replaced with a House of the People (7th May, 2017)
100 Greatest Britons Candidate: Caroline Norton (28th March, 2017)
100 Greatest Britons Candidate: Mary Wollstonecraft (20th March, 2017)
100 Greatest Britons Candidate: Anne Knight (23rd February, 2017)
100 Greatest Britons Candidate: Elizabeth Heyrick (12th January, 2017)
100 Greatest Britons: Where are the Women? (28th December, 2016)
The Death of Liberalism: Charles and George Trevelyan (19th December, 2016)
Donald Trump and the Crisis in Capitalism (18th November, 2016)
Victor Grayson and the most surprising by-election result in British history (8th October, 2016)
Left-wing pressure groups in the Labour Party (25th September, 2016)
The Peasant's Revolt and the end of Feudalism (3rd September, 2016)
Leon Trotsky and Jeremy Corbyn's Labour Party (15th August, 2016)
Eleanor of Aquitaine, Queen of England (7th August, 2016)
The Media and Jeremy Corbyn (25th July, 2016)
Rupert Murdoch appoints a new prime minister (12th July, 2016)
George Orwell would have voted to leave the European Union (22nd June, 2016)
Is the European Union like the Roman Empire? (11th June, 2016)
Is it possible to be an objective history teacher? (18th May, 2016)
Women Levellers: The Campaign for Equality in the 1640s (12th May, 2016)
The Reichstag Fire was not a Nazi Conspiracy: Historians Interpreting the Past (12th April, 2016)
Why did Emmeline and Christabel Pankhurst join the Conservative Party? (23rd March, 2016)
Mikhail Koltsov and Boris Efimov - Political Idealism and Survival (3rd March, 2016)
Why the name Spartacus Educational? (23rd February, 2016)
Right-wing infiltration of the BBC (1st February, 2016)
Bert Trautmann, a committed Nazi who became a British hero (13th January, 2016)
Frank Foley, a Christian worth remembering at Christmas (24th December, 2015)
How did governments react to the Jewish Migration Crisis in December, 1938? (17th December, 2015)
Does going to war help the careers of politicians? (2nd December, 2015)
Art and Politics: The Work of John Heartfield (18th November, 2015)
The People we should be remembering on Remembrance Sunday (7th November, 2015)
Why Suffragette is a reactionary movie (21st October, 2015)
Volkswagen and Nazi Germany (1st October, 2015)
David Cameron's Trade Union Act and fascism in Europe (23rd September, 2015)
The problems of appearing in a BBC documentary (17th September, 2015)
Mary Tudor, the first Queen of England (12th September, 2015)
Jeremy Corbyn, the new Harold Wilson? (5th September, 2015)
Anne Boleyn in the history classroom (29th August, 2015)
Why the BBC and the Daily Mail ran a false story on anti-fascist campaigner, Cedric Belfrage (22nd August, 2015)
Women and Politics during the Reign of Henry VIII (14th July, 2015)
The Politics of Austerity (16th June, 2015)
Was Henry FitzRoy, the illegitimate son of Henry VIII, murdered? (31st May, 2015)
The long history of the Daily Mail campaigning against the interests of working people (7th May, 2015)
Nigel Farage would have been hung, drawn and quartered if he lived during the reign of Henry VIII (5th May, 2015)
Was social mobility greater under Henry VIII than it is under David Cameron? (29th April, 2015)
Why it is important to study the life and death of Margaret Cheyney in the history classroom (15th April, 2015)
Is Sir Thomas More one of the 10 worst Britons in History? (6th March, 2015)
Was Henry VIII as bad as Adolf Hitler and Joseph Stalin? (12th February, 2015)
The History of Freedom of Speech (13th January, 2015)
The Christmas Truce Football Game in 1914 (24th December, 2014)
The Anglocentric and Sexist misrepresentation of historical facts in The Imitation Game (2nd December, 2014)
The Secret Files of James Jesus Angleton (12th November, 2014)
Ben Bradlee and the Death of Mary Pinchot Meyer (29th October, 2014)
Yuri Nosenko and the Warren Report (15th October, 2014)
The KGB and Martin Luther King (2nd October, 2014)
The Death of Tomás Harris (24th September, 2014)
Simulations in the Classroom (1st September, 2014)
The KGB and the JFK Assassination (21st August, 2014)
West Ham United and the First World War (4th August, 2014)
The First World War and the War Propaganda Bureau (28th July, 2014)
Interpretations in History (8th July, 2014)
Alger Hiss was not framed by the FBI (17th June, 2014)
Google, Bing and Operation Mockingbird: Part 2 (14th June, 2014)
Google, Bing and Operation Mockingbird: The CIA and Search-Engine Results (10th June, 2014)
The Student as Teacher (7th June, 2014)
Is Wikipedia under the control of political extremists? (23rd May, 2014)
Why MI5 did not want you to know about Ernest Holloway Oldham (6th May, 2014)
The Strange Death of Lev Sedov (16th April, 2014)
Why we will never discover who killed John F. Kennedy (27th March, 2014)
The KGB planned to groom Michael Straight to become President of the United States (20th March, 2014)
The Allied Plot to Kill Lenin (7th March, 2014)
Was Rasputin murdered by MI6? (24th February 2014)
Winston Churchill and Chemical Weapons (11th February, 2014)
Pete Seeger and the Media (1st February 2014)
Should history teachers use Blackadder in the classroom? (15th January 2014)
Why did the intelligence services murder Dr. Stephen Ward? (8th January 2014)
Solomon Northup and 12 Years a Slave (4th January 2014)
The Angel of Auschwitz (6th December 2013)
The Death of John F. Kennedy (23rd November 2013)
Adolf Hitler and Women (22nd November 2013)
New Evidence in the Geli Raubal Case (10th November 2013)
Murder Cases in the Classroom (6th November 2013)
Major Truman Smith and the Funding of Adolf Hitler (4th November 2013)
Unity Mitford and Adolf Hitler (30th October 2013)
Claud Cockburn and his fight against Appeasement (26th October 2013)
The Strange Case of William Wiseman (21st October 2013)
Robert Vansittart's Spy Network (17th October 2013)
British Newspaper Reporting of Appeasement and Nazi Germany (14th October 2013)
Paul Dacre, The Daily Mail and Fascism (12th October 2013)
Wallis Simpson and Nazi Germany (11th October 2013)
The Activities of MI5 (9th October 2013)
The Right Club and the Second World War (6th October 2013)
What did Paul Dacre's father do in the war? (4th October 2013)
Ralph Miliband and Lord Rothermere (2nd October 2013)

