PeterWatts_Blindsight

Peter Watts 261 Blindsight
of cosmic-ray metaphors, because those feel right, because they
feel natural, because it can't look at things any other way. But it's
the wrong metaphor. So the system misunderstands everything
about itself. Maybe that's not a grand and glorious evolutionary
leap after all. Maybe it's just a design flaw."
"But you're the biologist. You know Mom was right better'n
anyone. Brain's a big glucose hog. Everything it does costs
through the nose."
"True enough," Cunningham admitted.
"So sentience has gotta be good for something, then. Because
it's expensive, and if it sucks up energy without doing anything
useful then evolution's gonna weed it out just like that."
"Maybe it did." He paused long enough to chew food or suck
smoke. "Chimpanzees are smarter than Orangutans, did you know
that? Higher encephalisation quotient. Yet they can't always
recognize themselves in a mirror. Orangs can."
"So what's your point? Smarter animal, less self-awareness?
Chimpanzees are becoming nonsentient?"
"Or they were, before we stopped everything in its tracks."
"So why didn't that happen to us?"
"What makes you think it didn't?"
It was such an obviously stupid question that Sascha didn't have
an answer for it. I could imagine her gaping in the silence.
"You're not thinking this through," Cunningham said. "We're
not talking about some kind of zombie lurching around with its
arms stretched out, spouting mathematical theorems. A smart
automaton would blend in. It would observe those around it,
mimic their behavior, act just like everyone else. All the while
completely unaware of what it was doing. Unaware even of its
own existence."
"Why would it bother? What would motivate it?"
"As long as you pull your hand away from an open flame, who
cares whether you do it because it hurts or because some feedback
algorithm says withdraw if heat flux exceeds critical T? Natural
selection doesn't care about motives. If impersonating something
increases fitness, then nature will select good impersonators over
bad ones. Keep it up long enough and no conscious being would