Many of these arguments contain echoes of Lewis Mumford, the twentieth-century grandee of humanist critiques of techno-industrial society, who warned that any intelligent machines will “bear the stamp of the human mind, partly rational, partly cretinous, partly demonic.” Kurzweil has always reveled in locking horns with modern-day Mumfordians who denounce the Singularity as a terrifying caricature of techno-optimist delusion, and he devoted a long chapter in The Singularity Is Near to grappling with antagonistic theologians, Marxists, Malthusians, and AI-skeptical philosophers of consciousness. (John Searle’s influential “Chinese Room” dismissal of AI alone received 10 pages of point-by-point rebuttal.)
Nearer lacks a full fight card of rematches, but Kurzweil does return at length to the “hard problem of consciousness,” which asks how AI consciousness can ever be “proved” when we don’t even understand how human brains give rise to the subjective experience that philosophers call qualia. Kurzweil’s argument, first laid out in his 2012 book, How to Create a Mind, rests on the assumption that the subjective self is a product of computation. All that is required to create qualia in nonbiological mediums is a sufficient duplication of this computing model. While some AI theorists believe “whole-brain emulation” will require understanding and copying human brains at the quantum level, Kurzweil is not one of them. “Subjective consciousness,” he writes,
likely stems from the complex way information is arranged by our brains, so we needn’t worry that our digital emulation doesn’t include a certain protein molecule from the biological original. By analogy, it doesn’t matter whether your jpeg files are stored on a floppy disk, a CD-ROM, or a USB flash drive—they look the same and work the same as long as the information is represented with the same sequence of 1s and 0s…. One of the major research projects of the next two decades will be figuring out what level of brain emulation is sufficient.
Even if the nature and origin of consciousness can never be settled in a way that satisfies the requirements of scientific definition, Kurzweil believes we know enough to begin desacralizing biological qualia. For him, both carbon- and silicon-based consciousness are intensely complex forms of information that emerged from chaos and thus command reverence. One is conjured by eons of biological evolution, the other by thunderbolts of technological advance; both represent a “fundamental force of the universe.” If the math adds up, outward signs of consciousness should be taken seriously as evidence of a subjective inner life, in both original AI consciousnesses and “replicants” based on existing human brains. “If there’s a plausible chance that an entity you mistreat might be conscious,” he writes, “the safest moral choice is to assume that it is rather than risk tormenting a sentient being.”
