Stross on Singularity

I’ve long enjoyed the science fiction novels of Charlie Stross, so I’m honored that he linked to my Betterness Explosion from his Three arguments against the singularity:

I periodically get email from folks who, having read “Accelerando”, assume I am some kind of fire-breathing extropian zealot who believes in the imminence of the singularity, the uploading of the libertarians, and the rapture of the nerds. … It’s time to set the record straight. … Santa Claus doesn’t exist. …

(Economic libertarianism is based on … reductionist … 19th century classical economics — a drastic over-simplification of human behaviour. … If acted upon, would result in either failure or a hellishly unpleasant state of post-industrial feudalism.) …

I can’t prove that there isn’t going to be a hard take-off singularity in which a human-equivalent AI rapidly bootstraps itself to de-facto god-hood. Nor can I prove that mind uploading won’t work, or that we are or aren’t living in a simulation. … However, … the prospects aren’t good.

First: super-intelligent AI is unlikely because … human-equivalent AI is unlikely. … We’re likely to leave out … needing to sleep for roughly 30% of the time, being lazy or emotionally unstable, and having motivations of its own. … We clearly want machines that perform human-like tasks. … But whether we want them to be conscious and volitional is another question entirely.

Uploading … is not obviously impossible. … Imagine most of the inhabited universe has been converted to a computer network, … programs live side by side with downloaded human minds and accompanying simulated human bodies. … A human mind would lumber about in a massively inappropriate body simulation. … I strongly suspect that the hardest part of mind uploading … [is] the body and its interactions with its surroundings. …

Moving on to the Simulation Argument: … anyone capable of creating an ancestor simulation wouldn’t be focussing their attention on any ancestors as primitive as us. … This is my take on the singularity: we’re not going to see a hard take-off, or a slow take-off, or any kind of AI-mediated exponential outburst. What we’re going to see is increasingly solicitous machines defining our environment … We may eventually see mind uploading, but … our hard-wired biophilia will keep dragging us back to the real world, or to simulations indistinguishable from it. …

The simulation hypothesis … we can’t actually prove anything about it. …. Any way you cut these three ideas, they don’t provide much in the way of referent points for building a good life. … It’s unwise to live on the assumption that they’re coming down the pipeline within my lifetime.

Alas Stross’s post is a bit of a rant – strong on emotion, but weak on argument. Maybe Stross did or will explain more elsewhere, but while he makes clear that he doesn’t want to associate with singularity fans, Stross doesn’t make clear that he actually disagrees much. Most thoughtful singularity fans probably agree that where possible hand-coded AI would be designed to be solicitous and avoid human failings, that simple unmodified upload minds are probably not competitive creatures in the long run, and that only a tiny fraction of our distant descendants would be interested in simulating us. (We libertarian-leaning economists even agree that classical econ greatly simplifies.)

But the fact that hand-coded AIs would differ in many ways from humans says little on the key issues of when AI will appear, how fast they’d improve, how local would be that growth, and how fast the world economy would grow as a result. The fact that eventually unmodified human uploads would not be competitive says little on the key issues of whether uploads come before powerful hand-coded AI, how long nearly unmodified uploads would dominate, or just how far from humans would be the most competitive creatures. And the fact that few descendants would simulate ancestor humans says little on the key question of how that small fraction multiplied by the vast number of descendants compares to the actual number of ancestor humans. (And the fact that classical econ greatly simplifies says little on the pleasantness of libertarian policies.)

Stross seems smart and well-read enough to have interesting things to say on these key questions, if only he can overcome his personal revulsion against affiliating with singularity fans, to directly engage these questions.