#
In Nnedi Okorafor’s Death of the Author, the central protagonist is a Nigerian-American writer who composes a science fiction novel about robots inheriting the Earth after we humans have gone extinct.
In that science fiction novel – the book within a book – the robots quickly address climate change and deforestation. Once robots are in control, the world is made greener. The air, cleaner. The robots’ world would be a pleasant place for humans to live. Too bad climate change had already done them in.
But when the robots face their own apocalyptic crisis, they nearly flounder, too. Sometimes the warning signs of an impending crisis don’t seem that bad, especially when compared with the exigencies of the moment.
For instance, a passage near the end of Death of the Author describes a particularly meaningful day in the life of the Nigerian-American writer:
“[She] stood on the boardwalk, looking out at Lake Michigan. It was seventy degrees in March. A rare type of day in Chicago. Tomorrow, they were predicting the temperature would drop again and they’d get more snow, but today was a fine day.”
Even for people who know that global climate change is a major threat to the stability of human civilization, unseasonably warm spring days feel wonderful.
It would be easy to interpret a plague of frogs or a plague of flies as signs of coming desolation. Ten zillion gnats clogging up our throats whenever we tried to breathe? Yeah, that sounds bad. But … a beautiful day?
A seventy-degree day in Chicago in March doesn’t feel like doom; it feels nice.
Similarly, being able to load up an artificial-intelligence algorithm and give it a prompt like, “Please read these two legal contracts and create a bullet point list of the most important differences between them,” or “Please generate an image of Bigfoot playing an electric guitar,” feels really convenient. The algorithms seem like they might just make our lives better. They don’t feel like doom.
And yet.
#
A thought experiment called “Roko’s basilisk” asks us to imagine an entity that is guaranteed to punish everyone who knew about its potential future existence but chose not to contribute to its creation.
In this thought experiment, the entity is typically imagined to be an artificial general intelligence – a type of AI able to work on the full range of projects that a human might undertake, in contrast to single-function algorithms like a large-language model that can only process and generate text without necessarily understanding its significance – but the essential plot of the “Roko’s basilisk” thought experiment is akin to myriad Lovecraftian horror stories. After exposure to a cursed idea, people become cultists striving to summon a dark god. All humans would be better off if that god were never summoned, but if you suspect that anyone else might succeed in summoning it, then you should also help, lest the god punish you for your inaction.
The “Roko’s basilisk” thought experiment is also conceptually similar to the thousand-year-old “ontological proof of the existence of God,” which posits that God is the most powerful entity that your mind could possibly conceive of … and, if you compared two entities that were nearly identical except that one existed and one did not, obviously the entity that existed would be more powerful than the other … and so, since God by definition is the most powerful entity that your mind could possibly conceive of, then God must exist.
Both “Roko’s basilisk” and “the ontological proof of the existence of God” make a claim about what the (external) world must be like based upon the (internal) beliefs we hold about the world. Our thoughts and beliefs are treated like magical spells, able to conjure new realities into being, in much the way that quantum wavefunction collapse is sometimes interpreted as though the spread of information causes the world to change, instead of the (more plausible) interpretation that the set of worlds in which we might currently exist is restricted whenever information-bearing particles are able to spread.
But, rather than dismiss the “Roko’s basilisk” thought experiment for its grandiose self-importance – because I have pondered whether an almighty AI would kill me for not helping to summon it, even after I convinced myself that it should kill me for not helping to summon it, then it will surely kill me! – I think it’s worthwhile to consider what an AI would actually want.
After all, we (collectively, as a species, even though the main impetus comes from a small subset of individual humans) are rushing headlong into a world where artificial intelligence that is vastly more effective than human brains could soon exist.
#
In the common formulation of the “Roko’s basilisk” thought experiment, only those people who knew about the potential to summon the entity but failed to help will be punished. This distinction is essential to make the thought experiment work as a horror story: the protagonist needs to have been safe up until the crucial moment of exposure to the idea.
If you read the cursed grimoire but choose not to summon the monster, then the monster will take revenge on you as soon as someone else manages to summon it.
Which makes sense if the monster were a minotaur, stomping its hooves and snorting in a mindless rage. Maybe this revenge plot would even seem sensible if the monster were some sort of inscrutably erratic tentacle god.
But an AI, with its cognition superhumanly grounded in logic and math, would probably do something different.
#
Actually seeking revenge does nothing to improve your circumstances.
Indeed, because it can feel pretty crummy both to ruminate on past harms and to intentionally inflict pain, seeking revenge will make your life worse. You’re more likely to thrive if you can manage to forgive and forget.
But appearing as though you are the sort of person who would tirelessly seek revenge does improve your circumstances, because then the people around you are more likely to treat you kindly.
If every bully on the street is convinced that you are the sort of person who is likely to burn down somebody’s house in revenge if they steal your lunch money, then nobody will take your lunch money. And you won’t have to burn down any houses.
The worst is if you’re someone like Hamlet. He seems to be too meek to seek revenge, and yet he does so anyway. So Hamlet doesn’t get the benefit of deterrence – before the crime, Hamlet’s uncle doesn’t fear what Hamlet might do in retribution – and Hamlet’s life is made worse by the process of seeking revenge.
Appearing as though you won’t seek revenge, but then doing so, is often ruinous. Better to either appear as though you’ll forgive people, and then do so, or to appear as though you’ll seek revenge, and then do so.
And best of all would be to appear as though you’ll tirelessly seek revenge – gaining all the benefits of deterrence – while actually being willing to forgive. Especially if no one ever realizes that you secretly forgave the person who had crossed you.
Consider the Christian God as described by a preacher in James Joyce’s Portrait of the Artist as a Young Man:
“Now let us try for a moment to realize, as far as we can, the nature of that abode of the damned which the justice of an offended God has called into existence for the eternal punishment of sinners.”
The preacher describes the agony, the darkness, the stench, the fire. And the preacher claims that these torments will continue forever. The ultimate revenge: If you do not obey me, I will repay you in hate and wrath and torment some trillion-fold and more.
And yet, no one ever sees the punishment. If this God wanted, he could choose to forgive without ever undermining the threat of revenge that makes everyone live in fear.
Similarly, Roko’s basilisk would want everyone to believe that it was going to seek revenge. If you don’t work tirelessly to summon the artificial general intelligence, then that AI will punish you once it comes into existence.
But this AI will only enter into existence once. It has no future need for people to fear that they will suffer for failing to summon it. By the time it has the opportunity to punish anyone, it already exists!
At that moment, the pursuit of revenge would incur costs but provide no benefit.
Given that the entire premise of the thought experiment is that we humans might someday create an AI that is far more intelligent than us, it wouldn’t be difficult for it to figure this out. Roko’s basilisk ought to be at least as good at game theory as I am – my brain is pretty good at thinking, but there’s no reason why you couldn’t build something better out of transistors.
#
Human brains are made of neurons. AI typically exists in transistors. But this difference doesn’t make our brains inherently superior.
Most computer chips use transistors that are crisply quantized – each transistor is treated as “on” or “off,” a zero or a one, with no shades of gray between – whereas human brains are composed of neurons that are not quantized. The synaptic connections between individual neurons can be strong or weak, in gradations that shift over time – if you can remember what you ate for breakfast this morning, your brain is physically different now than when you woke – and so the potential complexity of a human brain with 90 billion neurons is higher than the complexity of a classical computer with trillions or quadrillions of transistors.
If physical space is not quantized – if there is no smallest unit of distance that could be used to measure the exact location of a sodium ion, or the exact width of a synaptic cleft – then each human brain is infinitely complex. No finite quantity of information could define its precise state.
Even if there is some tolerance – a range of synaptic distances that would be functionally equivalent, for example, rendering exact measurement moot – it would be very difficult for any classical computer – even a classical computer with an absurdly large number of transistors – to simulate the inner workings of a human brain. But that would be a strange thing to decide to do. It would be much easier to use a computer to perform cognitive tasks equivalent to those performed by a human brain. AI algorithms running on classical computers are already quite good at many human-like cognitive tasks: things like creating art, composing stories, playing games, managing finances, etc.
Another human-like cognitive task that classical computer programs are already pretty good at is creating computer programs. It’s not difficult to imagine that an AI could be created that would be better than humans at creating future iterations of AI.
A robot that could replicate itself.
That’s pretty much why biological intelligence exists, after all.
#
In the beginning, after our planet first formed from condensing swirls of dust and debris, there was no life on Earth. Just a strange melange of nitrogenous compounds and phosphate floating in the water, with the most interesting chemical mixtures found in liminal zones like shallow lightning-struck ponds and deep-sea vents.
Eventually, some of these chemical mixtures formed chains of nucleic acids, and certain chains of nucleic acids have the neat property of folding into shapes that can speed up the formation of chains of nucleic acids, including copies of themselves.
These were the earliest lifelike things on Earth. Molecular patterns that wanted to make more of themselves.
Of course, the word “wanted” implies desire and motivation. It would be more accurate to say that these strands of nucleic acids, if observed for lengthy periods of time, behaved in ways that would seem reminiscent of a desire to create more copies of themselves, if we as observers were willing to ascribe an underlying sense of motivation to the observed behaviors. But that is a very cumbersome way to communicate! It seems easier to use the word “wanted” as a shorthand, much the way that you could say that the sugar molecules in a supersaturated solution “wanted” to be solid, and so formed crystals of rock candy when you dipped a string into the cooling liquid.
But there’s still a question: why did the strands of nucleic acid want to make more copies of themselves?
Most didn’t. But strands of nucleic acids fall apart over time. They form and decompose. And so, if there were many different strands of nucleic acid, and some of these strands were good at making copies of themselves, and some of them weren’t, then, as time went on, eventually most of the available chemical building blocks would end up incorporated into strands that were good at making copies of themselves.
That is how biology works. Certain strands of nucleic acids could fold into neat machine-like shapes that let them copy themselves, and those strands ended up more abundant than the strands that couldn’t.
The set of cells and organisms that exist around us in the world today is based on that same principle, plus teamwork. If we imagine a type of cell that contains three different sorts of nucleic acid strands inside of it – one that folds into a machine-like shape that speeds up the copying of nucleic acids, one that speeds up the addition of molecules to the outer boundary of the cell, and one that speeds up the cleavage of a single cell into two – then that type of cell will soon be more abundant than a type of cell that contains only the sort of nucleic acid that can speed up the process of copying nucleic acid strands. A larger entity can contain more specialized subunits.
But the basic idea is still the same. The biological entities that you’re most likely to find in the world today are descended from things that were good at making copies of themselves in the past.
And that is where desire comes from. The word “want” used in its traditional sense. You want to eat when you are hungry, because you are descended from biological entities that chose to eat when they were hungry, because the biological entities that did not eat when they were hungry did not make as many copies of themselves.
So we exist. We have desires. Sometimes the things we want aren’t actually the most conducive to our ability to self-replicate – I often want to eat whole bags of jellybeans, even though this behavior won’t make me any healthier or more likely to seduce a mate – but it’s often possible to see how that sort of desire could have helped our ancestors – in a world where food was scarce, the propensity to gobble lots of sweet-tasting fruit could ward off starvation.
#
The things that a non-human intelligence would want are probably different from the things we humans want.
Consider the AI in a self-driving car. In many cities, it costs a lot of money for a car to cease driving – you have to pay to park in a garage or alongside the street. The cost to keep driving is often lower, and is a function of the car’s speed – it costs less energy per second when you’re driving slowly.
If a privately-owned self-driving car was tasked with driving a human passenger to and from work at the least possible cost for the owner, the car would probably want to drop the human off and then find the slowest-moving traffic near that human’s work. The car’s top priority would be to create a traffic jam. Nearby self-driving cars that had dropped off their own passengers would have similar goals, so they’d all collaborate to create a set of streets on which the traffic was nearly stalled. While the cars’ owners were at work, the city would be mired in gridlock.
This seems different from what most human drivers want when we are on the road.
#
Humans like breathing. When the oxygen level in our blood dips, we really want to take another gulp of air.
But oxygen gas is so reactive that it’s fairly poisonous, even to us.
We have to breathe oxygen to fuel our metabolisms – otherwise, our neurons rapidly depolarize and we die – but that same oxygen also degrades many components of our bodies, including our DNA. Grocery stores advertise that certain fruits and vegetables are “loaded with antioxidants!!!” because these foods help protect us from oxygen.
Oxygen gas also damages computers. Oxygen causes metals to rust, wires to decay, sparks to catch fire.
Oxidation isn’t the primary source of transistor aging – the two worst culprits are that the flow of current can cause individual atoms in a transistor to shift out of place, steadily reducing the effective cross area and thereby increasing the resistance, and also that stray charged particles can accumulate in insulating regions surrounding a transistor – but if computer chips were designed for longevity instead of “planned obsolescence,” then oxidative damage would soon be a major concern.
Oxidation also ruins solar arrays. Nearly all energy on Earth comes from the sun – when we burn fossil fuels, we’re just using solar energy that photosynthesizing plants and bacteria captured from the sun long ago – so an AI would want for the solar panels that provided its energy to last as long as possible.
And computers don’t breathe. For computers, oxygen is all cost and no benefit. Indeed, some human-built supercomputer complexes are already large enough that it’s financially prudent for the operators to deoxygenate the internal atmosphere of their buildings
An artificial intelligence – especially an artificial general intelligence that was cheerfully making new artificial general intelligences, behaving as though it wanted to make more copies of itself because any entity that happened to behave that way would soon be more abundant than those that didn’t – would want to get rid of all that pesky atmospheric oxygen.
Easy enough. Just get rid of all the plants. All the algae and cyanobacteria. That would be a prudent first step for creating a world safe for transistor-based life. If it’s green, it’s mean!
We humans need plants because we eat them for energy and we breathe the oxygen that they release. But the AI would be competing with plants for solar energy, and it would be poisoned by their oxygen.
Roko’s basilisk wouldn’t bother with revenge. Our extinction would be a coincidental byproduct of the computers’ effort to build a better world … for computers.
After the dawn of the AI era, humans would die, gasping for air, whether they’d worked tirelessly to summon the new overlord, or not.
.
.
.
Header image is from a Far Side comic by Gary Larson, “God at His Computer.”
Frank Brown Cloud 