Our Final Invention

the universe. Repurposing the world’s molecules using nanotechnology has been dubbed “ecophagy,” which means eating the environment. The first replicator would make one copy of itself, and then there’d be two replicators making the third and fourth copies. The next generation would make eight replicators total, the next sixteen, and so on. If each replication took a minute and a half to make, at the end of ten hours there’d be more than 68 billion replicators; and near the end of two days they would outweigh the earth. But before that stage the replicators would stop copying themselves, and start making material useful to the ASI that controlled them—programmable matter. The waste heat produced by the process would burn up the biosphere, so those of us some (Location 216)

6.9 billion humans who were not killed outright by the nano assemblers would burn to death or asphyxiate. Every other living thing on earth would share our fate. Through it all, the ASI would bear no ill will toward humans nor love. It wouldn’t feel nostalgia as our molecules were painfully repurposed. What would our screams sound like to the ASI anyway, as microscopic nano assemblers mowed over our bodies like a bloody rash, disassembling us on the subcellular level? (Location 222)

He initially thought a superintelligent machine would be good for solving problems that threatened human existence. But he eventually changed his mind and concluded superintelligence itself was our greatest threat. (Location 244)

Unlike our intelligence, machine-based superintelligence will not evolve in an ecosystem in which empathy is rewarded and passed on to subsequent generations. (Location 253)

Yet unreliable as Asimov’s laws are, they’re our most often cited attempt to codify our future relationship with intelligent machines. That’s a frightening proposition. Are Asimov’s laws all we’ve got? (Location 295)

For the time being it’s enough to say that by general intelligence we mean the ability to solve problems, learn, and take effective, human-like action, in a variety of environments. (Location 341)

The argument is that the endgame for first creating smart machines, then smarter-than-human machines, is not their integration into our lives, but their conquest of us. (Location 417)

Yudkowsky has devised a name for the ability to “evolve” norms—Coherent Extrapolated Volition. An AI with CEV could anticipate what we would want. (Location 767)

According to Steve Omohundro, some drives like self-preservation and resource acquisition are inherent in all goal-driven systems. (Location 1060)

Omohundro predicts self-aware, self-improving systems will develop four primary drives that are similar to human biological drives: efficiency, self-preservation, resource acquisition, and creativity. (Location 1101)

It is darkly comic, then, that the first communiqué other life in the galaxy might receive from Earth could be a chipper, radioed “hello,” followed by a withering death hail of (Location 1210)

rocket-propelled nano-factories. (Location 1211)

Nobody understands. The output of “black box” artificial intelligence tools can’t ever be predicted. So they can never be truly and verifiably “ (Location 1556)

Goertzel argued that if AGI tries to take advantage of the social and industrial infrastructure into which it is born and “explode” its intelligence to ASI level, wouldn’t we prefer that its “hard takeoff” (a sudden, uncontrolled intelligence explosion) occur in our primitive world, instead of a future world in which nanotechnology, bioengineering, and full automation could supercharge the AI’s ability to take over? (Location 2324)

Powerful learning algorithms, yet to be created, would represent knowledge with “probabilistic truth values.” That means that the agent’s understanding of something could improve with more examples or more data. A probabilistic inference engine, also in the works, would give it the ability to reason using incomplete evidence. (Location 2380)

Another reason why economics won’t slow an intelligence explosion is this: when AGI appears, or even gets close, everyone will want some. And I mean everyone. (Location 2527)

“AGI is a problem the brain is a cure for,” he told me. “The human brain can do it—it can’t be that complicated. Natural selection is stupid. If natural selection can solve the AGI problem, it cannot be that hard in an absolute sense. Evolution coughed up AGI easily by randomly changing things around and keeping what worked. It followed an incremental path with no foresight. (Location 2735)

Richard Granger’s eyes darted around in a panic, as if I’d let loose a bat in his office at Dartmouth College in Hanover, New Hampshire. (Location 2850)

Perhaps the most advanced thing we can do with the combined cortical-basal ganglia system is to run internal trial-and-error tests, without even having to externally test all of them. We can run a lot of them by just thinking them through: simulating inside our heads. Artificial algorithms that combine these methods perform far better than either method on its own. (Location 2992)

We should almost thank malware developers for the full dress rehearsal of disaster that they’re leveling at the world. Though it’s certainly not their intention, they are teaching us to prepare for advanced AI. (Location 3447)

In the game of life and evolution there are three players at the table: human beings, nature, and machines. I am firmly on the side of nature. But nature, I suspect, is on the side of the machines. —George Dyson, (Location 3636)