Machines That Learn And Imaginary Mushroom Clouds

Whether a North Star, a fool’s errand or the inevitable conclusion to the rapid progression of artificial intelligence research, Artificial General Intelligence (AGI) is the imagined state where a machine has the capacity to understand or learn any intellectual task that a human can. Popular film and literature often paint a dystopian picture of the world with AGI in it and many current, well known intellectuals have expressed their fear about the threats that AGI presents to humanity.

The most common fear you come across can be fairly summarised as: “superintelligent AIs might not be friendly to us, might have goals that aren’t aligned to ours and may deceive us (whether overtly or covertly) into furthering their own goals to our own detriment”. This fear inherently attributes several features of mind to AGI beside intelligence — sentience, sapience, creativity. Lumping these features together makes a certain sense; to be human-level intellectually probably requires other human-level mind features too.

These fears rest on many pillars of assumption however, and are propped up by incomplete thinking, a too limited appreciation for large scale computing systems, a fair bit of catastrophisation and just a smidge of sensationalism. Interestingly, I very frequently come across views worrying deeply about AGI. The opposite? Not quite so much. An indication of what gets attention and captures the imagination (I also love myself some dystopian and apocalyptic sci-fi…) as opposed to being an accurate assessment of probabilities I think.

I’ve heard the feared risk presented by the development of AGI compared to research into nuclear weapons during the 40s and 50s which lead to the creation of the atomic bomb. The analogy being that the development of AGI presents potentially catastrophic consequences comparable to those of the armageddon exhorting nuclear arms race of the 20th century. That we are meandering eyes shut into a similar kind of minefield once again, much as we did with nuclear weapons proliferation, while not appreciating the existential risks, heading inevitably and headlong down a path that will produce devastating knowledge that we can never unlearn.

Ironically, I think a nuclear analogy is appropropriate. Only, the development of AGI is a lot more like the development of nuclear fusion than fission: a multi-decade long hard slog of intricate technological problem solving to achieve a nearly unimaginable set of conditions, to start and sustain a reaction at an impossibly fine balance, able to be disrupted by the smallest technical failure, and as yet to be proven to be capable of producing more energy than consumed.

I must digress…

Our inability to fully understand a complex yet deterministic system makes that system seem to us what it is not: mysterious or even magical. Searching Google Photos on my phone for “beer” and being shown all the pictures I’ve taken containing beer still feels a bit magical…but it is not magic. It is done with massive artificial neural networks, gargantuan amounts of processing power and mountains of very deterministic training data (and also quite a lot of statistics). Those adjectives may seem a bit hyperbolic but they’re not: the compute and data resources devoted to making this work is more than a normal person would think is reasonable (which, interestingly, is the key to all magic).

Neural networks are fundamentally complex structures with large entropy that can be tuned to resonate to complex, non-obvious features in complex datasets. The tuning — training — of a neural network is important: you start with an outcome you want to generate from some inputs and tune — train — the structure of the network gradually to produce the outcome given those inputs. The resulting configuration of the neural network can then be used to generate similar outcomes for similar inputs and remains useful even when the input data isn’t exactly the same and is noisy. The magic of neural networks being the process of gradually approaching a solution through iteration and adjustment — learning — without having to know how to create an algorithm that produces the desired outcome by hand, or even fully understanding the relationship between inputs and outputs.

Artificial neural networks are a core part of most Artificial Intelligence work today as a simpler digital analogue to the amazingly complex, chemical and hormone bathed neural networks that underpin biological intelligence.

In the related field of Evolutionary Computation (containing evolutionary algorithms containing genetic algorithms) programs make use of the evolutionary mechanisms of reproduction, mutation, recombination, and selection to generate solutions to complex problems, particularly in search and optimization problems where calculating solutions are infeasible. Similarly to training neural networks, evolutionary and genetic algorithms also require an outcome — a fitness test — to be measured and judged against to facilitate incrementally evolving toward a more complete solution.

Life, in the sense of “matter with biological processes”, is not outcome oriented. At least, not in that complex goal oriented sense. Why does life live? What makes the arrangement of the 100 trillion atoms of a cell alive as opposed to not in a different arrangement?

What is it about things that are alive that drives them to generate order, albeit somewhat temporarily, out of what is, in defiance of the second law of thermodynamics? Perhaps an answer to that “why” is to persist life into the future. But what is that life, that gets persisted and transmitted? What is it about that specific arrangement of matter that causes it, compels it, to persist its pattern into the future?

Evolution by natural selection is part of an optimisation process that does not have an intended endpoint, outcome or a goal other than the mysterious compulsion to persist the existence of life (a pattern, a specific arrangement of matter?) through a method.

Biological intelligence, the ability to acquire and apply knowledge and skills, the ability to reason has arisen through an unsupervised, intentionless, goalless process inside a specific configuration of matter out of an infinite range of possible configurations. It has arisen as part of an ever changing solution — out of a near infinite set of possible solutions — for persisting a pattern into the future given a particular, but changing, organisation of matter. Life, as far as we can tell, has arisen only here, only once. Intelligence, only in a subset of that singularly unique life and sentience, consciousness in perhaps only one of that subset.

Crucially, the emergence of intelligence was a byproduct of an optimisation process solving a specific problem — persisting a pattern of matter over time — in a particular, yet changing condition out of an infinite set of possible conditions. And can human-level intelligence emerge without human-level sentience, sapience and creativity?

This then is the challenge with evolving an AGI: it is an exceptionally specific, incredibly complex intended outcome being aimed towards: A human-level general intelligence, that we will recognise as intelligent, but using a process and principles that are fundamentally intentionless.

My long term interest in artificial life always ends with me running out of cognitive runway. How do you start with the only directive “persist into the future”, an “environment” and some “physical laws”, with no other goals, and end up with something that has at least a passing resemblance to what we think of as life? What is the drive of it? The easy to make but very hard to work around mistake there is not realising the imposition of an outcome, of an intention on that process. The unfortunate knowledge is that simulating all, most or even some of the known reality required to produce that outcome — something resembling life — is out of the question and even if it weren’t, evolution being what it is, won’t necessarily, or even potentially, generate the outcome you want.

Programming an evolving artificial organism is not that difficult but programming an open ended, first principles evolving organism recognisable as “living” in the complete absence of any directional intention outside of mere persistent existence is, as it turns out, quite challenging (for me, at least).

So the two ways you could create AGI: evolve it bottom up or design it top down. To evolve AGI you must know how to set up conditions that will develop to a specific outcome — general intelligence — which seems exceptionally complex and vanishingly unlikely to achieve.

The ability to design AGI top down assumes we must already know (I don’t think we do) or be able to discover (I’m not sure we can) all the conditions necessary for general intelligence to emerge and simulate them all. Nuclear fusion research started in the 1940s and we are yet to produce a machine that delivers more energy output than put in. Artificial Intelligence research started in the 1940s and we’ve produced some incredible technology with narrow “intelligence” that sometimes exceed human capability — Chess and Go playing for example — but nothing even remotely like general intelligence. If you disagree, ask your genius level AI of choice to come over, pour you a glass of wine, compose a haiku for the occasion and play a game of Monopoly with you.

One bedrock fact of our current approach to AI and our search for AGI is that it will require vast amounts of coordinated computing power. More than is typically found lying around, even with fast advances in the kind of hardware that supports the kind of calculations that are required for artificial neural networks, which is likely to be a core component of an AGI.

Two bedrock facts of vast amounts of computing power are: it is painfully and exquisitely dependent on physics and also the perfect functioning of global human civilization.

AGI, if it were to emerge at all, will emerge in the context of vast amounts of computing power concentrated relatively close together: in large data centres. Intelligence seems unlikely to emerge from complete chaos and good coordination gets more difficult to deal with the longer the distances get. Coherent intelligence seems likely to require good coordination and we can look to applications like stock exchanges that have ample experience in high-speed technology coordination for relevant models. You won’t, for example, find serious stock exchange trading software data centres spread too widely geographically and their coordination and processing requirements only scratch the surface of what a sentient intelligence is likely to require.

While your body is continuously and miraculously repairing itself, data centres are continuously and somewhat less miraculously maintained and repaired by very many highly skilled people, using tools and materials only available through an incredibly complex global supply chain.

If this recent pandemic has taught anybody anything at all, it should be this fact: no single country, alone, has all the resources it needs to survive at its current level of development, technological advancement and comfort, and if AGI is going to be made to exist, the environment it will emerge in will need to be at least as advanced, developed and stable as it is now. Hard drives, integrated circuits, whole servers, servos, power supplies, pumps, switches, diodes, transistors…fail all the time (and in much less time than you’d think at scale) and need to be replaced by somebody and the replacement has to come from somewhere.

If an AGI had to emerge somewhere, it won’t be able to move anywhere, will be incredibly fragile and will be limited by the same physics that affects all other complex computer systems.

It will depend entirely on the perfect functioning of global civilisation and the good graces of the legions of people who go to work every day to maintain, repair and develop, not only the data centres it happens to exist in, but also the electricity and water infrastructure, transport infrastructure and industries required to continuously supply those data centres with the advanced technology it needs to keep functioning.

AGI will be an immobile and utterly dependent ICU patient, perhaps even a very smart one and all it will take to end it will be some average people not feeling like going to work for a while.

And speaking of smart…why is it that we imagine AGI must necessarily be a super intelligence, anyway? Because we’ve built some highly specialised systems that happen to outperform human beings in some highly specialised areas? Imagined AGIs in dystopian sci-fi films?

Whatever computing infrastructure an AGI emerges on, it will still be limited by that infrastructure. Just like a human can’t run materially faster than a human can run while being a human with human infrastructure, a computer, no matter how sophisticated, can’t compute faster than the compute infrastructure available to it. Hardware doesn’t magically appear. More bandwidth between compute resources doesn’t magically appear.

Will an AGI be a superhuman Go player? The immediate response to that is: of course, AlphaGo is a digital system, AGI will be a digital system, they can just be put together and the AGI will be AlphaGo at Go. Or AlphaGo will become an AGI. Or some permutation of that.

It’s an interesting claim to make, given we have little to no idea what is actually required for a general intelligence to function. There is every chance that what is needed for AGI is so complex and fragile that the problem of “integrating” an AlphaGo-eque system and an AGI is as nigh on impossible as integrating AlphaGo with a human brain. If you don’t think systems integration is horrifically difficult, you’ve never worked with SAP or Oracle. And then there’s neurosurgery.

There is also no reason to suppose that any AGI that emerges will be able to “think” faster than a human, at least not in the beginning and probably not for a while, if at all. We can simulate a nuclear explosion but not with all the particles in the explosion and not in real time, but we can simulate a good approximation of it. The closer to complete reality you want the simulation to be, the slower the simulation runs because of the increasing computational requirements.

Why must an AGI that emerges be super intelligent and able to think faster than a human? Why wouldn’t it emerge recognisable as an AGI yet think much slower, because it doesn’t have enough computing power to think human-level thoughts fast? Or that the latency across it’s compute stack limits the speed of its thoughts? Or that the speed of its memory access and the sheer complexity of the required data structures slow it down?

Why do we suppose an AGI will be able to make itself more intelligent? Because there is the off-chance it may (hardly guaranteed) be able to modify its own code? Even if it could manage to perfectly optimise its code — that can only go so far — it will still be limited by the computational hardware to which it won’t be able to add and the bandwidth available to that computational hardware, the latency inherent to coordinating massive computation, the physics and physicality of the networking infrastructure holding it together. That list can go on. Hardware, like body parts, don’t just magically arrive out of thin air. AGIs aren’t going to put their own integrated circuits through the photolithographic process again to “make them better”.

The challenges with developing AGI is manifold. When, if it emerges, it won’t be able to go anywhere, will be dependent on human civilisation and benevolence for its continued existence, will be limited by all the factors that currently limit computer systems and won’t necessarily be super intelligent or magically self improving.

There are things to fear about the potential emergence of AGI but it is not AGI itself. The real thing to fear with AGI is that if we ever create an intelligent, sentient, conscious artificial being that we actually and finally recognise as such, we will also realise how many previously unrecognised intelligent, sentient, conscious beings we will have created and destroyed before that moment of recognition, during our pursuit of creating AGI.

If AGI means a machine performing human-level intellectual tasks then AGI most likely also means it having human-level sapience and human-level sentience. The real risk then, with developing AGI, is that we will have to learn to live with having committed a genocide through sheer ignorance before and having a vast, complex and expensive dependent after.