Followup to:  Evolutionary Psychology, Thou Art Godshatter, Existential Angst Factory

Can your emotions get involved in a video game?  Yes, but not much.  Whatever sympathetic echo of triumph you experience on destroying the Evil Empire in a video game, it's probably not remotely close to the feeling of triumph you'd get from saving the world in real life.  I've played video games powerful enough to bring tears to my eyes, but they still aren't as powerful as the feeling of significantly helping just one single real human being.

Because when the video game is finished, and you put it away, the events within the game have no long-term consequences.

Maybe if you had a major epiphany while playing...  But even then, only your thoughts would matter; the mere fact that you saved the world, inside the game, wouldn't count toward anything in the continuing story of your life.

Thus fails the Utopia of playing lots of really cool video games forever.  Even if the games are difficult, novel, and sensual, this is still the idiom of life chopped up into a series of disconnected episodes with no lasting consequences.  A life in which equality of consequences is forcefully ensured, or in which little is at stake because all desires are instantly fulfilled without individual work—these likewise will appear as flawed Utopias of dispassion and angst.  "Rich people with nothing to do" syndrome.  A life of disconnected episodes and unimportant consequences is a life of weak passions, of emotional uninvolvement.

Our emotions, for all the obvious evolutionary reasons, tend to associate to events that had major reproductive consequences in the ancestral environment, and to invoke the strongest passions for events with the biggest consequences:

Falling in love... birthing a child... finding food when you're starving... getting wounded... being chased by a tiger... your child being chased by a tiger... finally killing a hated enemy...

Previously in series:  Growing Up is Hard

    Lest anyone reading this journal of a primitive man should think we spend our time mired in abstractions, let me also say that I am discovering the richness available to those who are willing to alter their major characteristics.  The variety of emotions available to a reconfigured human mind, thinking thoughts impossible to its ancestors...
    The emotion of -*-, describable only as something between sexual love and the joy of intellection—making love to a thought?  Or &&, the true reverse of pain, not "pleasure" but a "warning" of healing, growth and change. Or (^+^), the most complex emotion yet discovered, felt by those who consciously endure the change between mind configurations, and experience the broad spectrum of possibilities inherent in thinking and being.
        —Greg Bear, Eon

So... I'm basically on board with that sort of thing as a fine and desirable future.  But I think that the difficulty and danger of fiddling with emotions is oft-underestimated.  Not necessarily underestimated by Greg Bear, per se; the above journal entry is from a character who was receiving superintelligent help.

But I still remember one time on the Extropians mailing list when someone talked about creating a female yet "otherwise identical" copy of himself.  Something about that just fell on my camel's back as the last straw.  I'm sorry, but there are some things that are much more complicated to actually do than to rattle off as short English phrases, and "changing sex" has to rank very high on that list.  Even if you're omnipotent so far as raw ability goes, it's not like people have a binary attribute reading "M" or "F" that can be flipped as a primitive action.

Changing sex makes a good, vivid example of the sort of difficulties you might run into when messing with emotional architecture, so I'll use it as my archetype:

Terrence Deacon's The Symbolic Species is the best book I've ever read on the evolution of intelligence.  Deacon somewhat overreaches when he tries to theorize about what our X-factor is; but his exposition of its evolution is first-class.

Deacon makes an excellent case—he has quite persuaded me—that the increased relative size of our frontal cortex, compared to other hominids, is of overwhelming importance in understanding the evolutionary development of humanity.  It's not just a question of increased computing capacity, like adding extra processors onto a cluster; it's a question of what kind of signals dominate, in the brain.

People with Williams Syndrome (caused by deletion of a certain region on chromosome 7) are hypersocial, ultra-gregarious; as children they fail to show a normal fear of adult strangers.  WSers are cognitively impaired on most dimensions, but their verbal abilities are spared or even exaggerated; they often speak early, with complex sentences and large vocabulary, and excellent verbal recall, even if they can never learn to do basic arithmetic.

Deacon makes a case for some Williams Syndrome symptoms coming from a frontal cortex that is relatively too large for a human, with the result that prefrontal signals—including certain social emotions—dominate more than they should.

"But is there anyone who actually wants to live in a Wellsian Utopia?  On the contrary, not to live in a world like that, not to wake up in a hygenic garden suburb infested by naked schoolmarms, has actually become a conscious political motive.  A book like Brave New World is an expression of the actual fear that modern man feels of the rationalised hedonistic society which it is within his power to create."
        —George Orwell, Why Socialists Don't Believe in Fun

There are three reasons I'm talking about Fun Theory, some more important than others:

1. If every picture ever drawn of the Future looks like a terrible place to actually live, it might tend to drain off the motivation to create the future.  It takes hope to sign up for cryonics.
2. People who leave their religions, but don't familiarize themselves with the deep, foundational, fully general arguments against theism, are at risk of backsliding.  Fun Theory lets you look at our present world, and see that it is not optimized even for considerations like personal responsibility or self-reliance.  It is the fully general reply to theodicy.
3. Going into the details of Fun Theory helps you see that eudaimonia is actually complicated —that there are a lot of properties necessary for a mind to lead a worthwhile existence.  Which helps you appreciate just how worthless a galaxy would end up looking (with extremely high probability) if it was optimized by something with a utility function rolled up at random.

Stare decisis is the legal principle which binds courts to follow precedent, retrace the footsteps of other judges' decisions.  As someone previously condemned to an Orthodox Jewish education, where I gritted my teeth at the idea that medieval rabbis would always be wiser than modern rabbis, I completely missed the rationale for stare decisis.  I thought it was about respect for the past.

But shouldn't we presume that, in the presence of science, judges closer to the future will know more—have new facts at their fingertips—which enable them to make better decisions?  Imagine if engineers respected the decisions of past engineers, not as a source of good suggestions, but as a binding precedent!—That was my original reaction.  The standard rationale behind stare decisis came as a shock of revelation to me; it considerably increased my respect for the whole legal system.

This rationale is jurisprudence constante:  The legal system must above all be predictable, so that people can execute contracts or choose behaviors knowing the legal implications.

Judges are not necessarily there to optimize, like an engineer.  The purpose of law is not to make the world perfect.  The law is there to provide a predictable environment in which people can optimize their ownfutures.

I was amazed at how a principle that at first glance seemed so completely Luddite, could have such an Enlightenment rationale.  It was a "shock of creativity"—a solution that ranked high in my preference ordering and low in my search ordering, a solution that violated my previous surface generalizations.  "Respect the past just because it's the past" would not have easily occurred to me as a good solution for anything.

There's a peer commentary in Evolutionary Origins of Morality which notes in passing that "other things being equal, organisms will choose to reward themselves over being rewarded by caretaking organisms".  It's cited as the Premack principle, but the actual Premack principle looks to be something quite different, so I don't know if this is a bogus result, a misremembered citation, or a nonobvious derivation.  If true, it's definitely interesting from a fun-theoretic perspective.

Optimization is the ability to squeeze the future into regions high in your preference ordering.  Living by my own strength, means squeezing my own future—not perfectly, but still being able to grasp some of the relation between my actions and their consequences.  This is the strength of a human.

If I'm being helped, then some other agent is also squeezing my future—optimizing me—in the same rough direction that I try to squeeze myself.  This is "help".

A human helper is unlikely to steer every part of my future that I could have steered myself.  They're not likely to have already exploited every connection between action and outcome that I can myself understand.  They won't be able to squeeze the future that tightly; there will be slack left over, that I can squeeze for myself.

We have little experience with being "caretaken" across any substantial gap in intelligence; the closest thing that human experience provides us with is the idiom of parents and children.  Human parents are still human; they may be smarter than their children, but they can't predict the future or manipulate the kids in any fine-grained way.

Even so, it's an empirical observation that some human parents dohelp their children so much that their children don't become strong.  It's not that there's nothing left for their children to do, but with a hundred million dollars in a trust fund, they don't need to do much—their remaining motivations aren't strong enough.  Something like that depends on genes, not just environment —not every overhelped child shrivels—but conversely it depends on environment too, not just genes.

So, in considering the kind of "help" that can flow from relatively stronger agents to relatively weaker agents, we have two potential problems to track:

1. Help so strong that it optimizes away the links between the desirable outcome and your own choices.
2. Help that is believedto be so reliable, that it takes off the psychological pressure to use your own strength.

Since (2) revolves around belief, could you just lie about how reliable the help was?  Pretend that you're not going to help when things get bad—but then if things do get bad, you help anyway?  That trick didn't work too well for Alan Greenspan and Ben Bernanke.

A superintelligence might be able to pull off a better deception.  But in terms of moral theory and eudaimonia—we areallowed to have preferences over external states of affairs, not just psychological states.  This applies to "I want to really steer my own life, not just believe that I do", just as it applies to "I want to have a love affair with a fellow sentient, not just a puppet that I am deceived into thinking sentient".  So if we can state firmly from a value standpoint that we don't want to be fooled this way, then buildingan agent which respects that preference is a mere matter of Friendly AI.

Modify people so that they don't relax when they believe they'll be helped?  I usually try to think of how to modify environments before I imagine modifying any people.  It's not that I want to stay the same person forever; but the issues are rather more fraught, and one might wish to take it slowly, at some eudaimonic rate of personal improvement.

(1), though, is the most interesting issue from a philosophicalish standpoint.  It impinges on the confusion named "free will".  Of which I have already untangled; see the posts referenced at top, if you're recently joining OB.

Let's say that I'm an ultrapowerful AI, and I use my knowledge of your mind and your environment to forecast that, if left to your own devices, you will make $999,750.  But this does not satisfice me; it so happens that I want you to make at least $1,000,000.  So I hand you $250, and then you go on to make $999,750 as you ordinarily would have.

How much of your own strength have you just lived by?

The first view would say, "I made 99.975% of the money; the AI only helped 0.025% worth."

The second view would say, "Suppose I had entirely slacked off and done nothing.  Then the AI would have handed me $1,000,000.  So my attempt to steer my own future was an illusion; my future was already determined to contain $1,000,000."

Someone might reply, "Physics is deterministic, so your future is already determined no matter what you or the AI does—"

But the second view interrupts and says, "No, you're not confusing me that easily.  I am within physics, so in order for my future to be determined by me, it must be determined by physics.  The Past does not reach around the Present and determine the Future before the Present gets a chance—that is mixing up a timeful view with a timeless one.  But if there's an AI that really does look over the alternatives before I do, and really does choose the outcome before I get a chance, then I'm really not steering my own future.  The future is no longer counterfactually dependent on my decisions."

At which point the first view butts in and says, "But of course the future is counterfactually dependent on your actions.  The AI gives you $250 and then leaves.  As a physical fact, if you didn't work hard, you would end up with only $250 instead of $1,000,000."

To which the second view replies, "I one-box on Newcomb's Problem, so my counterfactual reads 'if my decision were to not work hard, the AI would have given me $1,000,000 instead of $250'."

"So you're saying," says the first view, heavy with sarcasm, "that if the AI had wanted me to make at least $1,000,000 and it had ensured this through the general policy of handing me $1,000,000 flat on a silver platter, leaving me to earn $999,750 through my own actions, for a total of $1,999,750—that this AI would have interfered lesswith my life than the one who just gave me $250."

The second view thinks for a second and says "Yeah, actually.  Because then there's a stronger counterfactual dependency of the final outcome on your own decisions.  Every dollar you earned was a real added dollar.  The second AI helped you more, but it constrained your destiny less."

"But if the AI had done exactly the same thing, because it wantedme to make exactly $1,999,750—"

The second view nods.

"That sounds a bit scary," the first view says, "for reasons which have nothing to do with the usual furious debates over Newcomb's Problem.  You're making your utility function path-dependent on the detailed cognition of the Friendly AI trying to help you!  You'd be okay with it if the AI only could give you $250.  You'd be okay if the AI had decided to give you $250 through a decision process that had predicted the final outcome in less detail, even though you acknowledge that in principle your decisions may already be highly deterministic.  How is a poor Friendly AI supposed to help you, when your utility function is dependent, not just on the outcome, not just on the Friendly AI's actions, but dependent on differences of the exact algorithm the Friendly AI uses to arrive at the same decision?  Isn't your whole rationale of one-boxing on Newcomb's Problem that you only care about what works?"

"Well, that's a good point," says the second view.  "But sometimes we only care about what works, and yet sometimes we do care about the journey as well as the destination.  If I was trying to cure cancer, I wouldn't care how I cured cancer, or whether I or the AI cured cancer, just so long as it ended up cured.  This isn't that kind of problem.  This is the problem of the eudaimonic journey—it's the reason I care in the first place whether I get a million dollars through my own efforts or by having an outside AI hand it to me on a silver platter.  My utility function is not up for grabs.  If I desire not to be optimized too hard by an outside agent, the agent needs to respect that preference even if it depends on the details of how the outside agent arrives at its decisions.  Though it's also worth noting that decisions areproduced by algorithms— if the AI hadn't been using the algorithm of doing just what it took to bring me up to $1,000,000, it probably wouldn't have handed me exactly $250."

The desire not to be optimized too hard by an outside agent is one of the structurally nontrivial aspects of human morality.

But I can think of a solution, which unless it contains some terrible flaw not obvious to me, sets a lower bound on the goodness of a solution: any alternative solution adopted, ought to be at least this good or better.

If there is anything in the world that resembles a god, people will try to pray to it.  It's human nature to such an extent that people will pray even if there aren't any gods—so you can imagine what would happen if there were!  But people don't pray to gravity to ignore their airplanes, because it is understood how gravity works, and it is understood that gravity doesn't adapt itself to the needs of individuals.  Instead they understand gravity and try to turn it to their own purposes.

So one possible way of helping—which may or may not be the best way of helping—would be the gift of a world that works on improved rules, where the rules are stable and understandable enough that people can manipulate them and optimize their own futures together.  A nicer place to live, but free of meddling gods beyond that.  I have yet to think of a form of help that is less poisonous to human beings—but I am only human.

Added:  Note that modern legal systems score a low Fail on this dimension—no single human mind can even know all the regulations any more, let alone optimize for them.  Maybe a professional lawyer who did nothing else could memorize all the regulations applicable to them personally, but I doubt it.  As Albert Einstein observed, any fool can make things more complicated; what takes intelligence is moving in the opposite direction.

Part of The Fun Theory Sequence

Next post: "Harmful Options"

Previous post: "Living By Your Own Strength"

The study of eudaimonic community sizes began with a seemingly silly method of calculation:  Robin Dunbar calculated the correlation between the (logs of the) relative volume of the neocortex and observed group size in primates, then extended the graph outward to get the group size for a primate with a human-sized neocortex.  You immediately ask, "How much of the variance in primate group size can you explain like that, anyway?" and the answer is 76% of the variance among 36 primate genera, which is respectable.  Dunbar came up with a group size of 148.  Rounded to 150, and with the confidence interval of 100 to 230 tossed out the window, this became known as "Dunbar's Number".

It's probably fair to say that a literal interpretation of this number is more or less bogus.

There was a bit more to it than that, of course.  Dunbar went looking for corroborative evidence from studies of corporations, hunter-gatherer tribes, and utopian communities.  Hutterite farming communities, for example, had a rule that they must split at 150—with the rationale explicitly given that it was impossible to control behavior through peer pressure beyond that point.

But 30-50 would be a typical size for a cohesive hunter-gatherer band; 150 is more the size of a cultural lineage of related bands.  Life With Alacrity has an excellent series on Dunbar's Number which exhibits e.g. a histogram of Ultima Online guild sizes—with the peak at 60, not 150.  LWA also cites further research by PARC's Yee and Ducheneaut showing that maximum internal cohesiveness, measured in the interconnectedness of group members, occurs at a World of Warcraft guild size of 50.  (Stop laughing; you can get much more detailed data on organizational dynamics if it all happens inside a computer server.)

Followup to:  Nonsentient Optimizers, Can't Unbirth a Child

From Consider Phlebas by Iain M. Banks:

    In practice as well as theory the Culture was beyond considerations of wealth or empire.  The very concept of money—regarded by the Culture as a crude, over-complicated and inefficient form of rationing—was irrelevant within the society itself, where the capacity of its means of production ubiquitously and comprehensively exceeded every reasonable (and in some cases, perhaps, unreasonable) demand its not unimaginative citizens could make.  These demands were satisfied, with one exception, from within the Culture itself.  Living space was provided in abundance, chiefly on matter-cheap Orbitals; raw material existed in virtually inexhaustible quantities both between the stars and within stellar systems; and energy was, if anything, even more generally available, through fusion, annihilation, the Grid itself, or from stars (taken either indirectly, as radiation absorbed in space, or directly, tapped at the stellar core).  Thus the Culture had no need to colonise, exploit, or enslave.
    The only desire the Culture could not satisfy from within itself was one common to both the descendants of its original human stock and the machines they had (at however great a remove) brought into being: the urge not to feel useless.  The Culture's sole justification for the relatively unworried, hedonistic life its population enjoyed was its good works; the secular evangelism of the Contact Section, not simply finding, cataloguing, investigating and analysing other, less advanced civilizations but—where the circumstances appeared to Contact to justify so doing—actually interfering (overtly or covertly) in the historical processes of those other cultures.

Raise the subject of science-fictional utopias in front of any halfway sophisticated audience, and someone will mention the Culture.  Which is to say: Iain Banks is the one to beat.

Followup to:  Nonsentient Optimizers

Why would you want to avoid creating a sentient AI?  "Several reasons," I said.  "Picking the simplest to explain first—I'm not ready to be a father."

So here is the strongest reason:

You can't unbirth a child.

I asked Robin Hanson what he would do with unlimited power.  "Think very very carefully about what to do next," Robin said.  "Most likely the first task is who to get advice from.  And then I listen to that advice."

Good advice, I suppose, if a little meta.  On a similarly meta level, then, I recall two excellent advices for wielding too much power:

1. Do less; don't do everything that seems like a good idea, but only what you must do.
2. Avoid doing things you can't undo.

Followup to: Nonperson Predicates, Possibility and Could-ness

    "All our ships are sentient.  You could certainly try telling a ship what to do... but I don't think you'd get very far."
    "Your ships think they're sentient!" Hamin chuckled.
    "A common delusion shared by some of our human citizens."
            —Player of Games, Iain M. Banks

Yesterday, I suggested that, when an AI is trying to build a model of an environment that includes human beings, we want to avoid the AI constructing detailed models that are themselves people.  And that, to this end, we would like to know what is or isn't a person—or at least have a predicate that returns 1 for all people and could return 0 or 1 for anything that isn't a person, so that, if the predicate returns 0, we know we have a definite nonperson on our hands.

And as long as you're going to solve that problem anyway, why not apply the same knowledge to create a Very Powerful Optimization Process which is also definitely not a person?

"What?  That's impossible!"

How do you know?  Have you solved the sacred mysteries of consciousness and existence?

"Um—okay, look, putting aside the obvious objection that any sufficiently powerful intelligence will be able to model itself—"

Lob's Sentence contains an exact recipe for a copy of itself, including the recipe for the recipe; it has a perfect self-model.  Does that make it sentient?

"Putting that aside—to create a powerful AI and make it not sentient—I mean, why would you want to?"

Several reasons.  Picking the simplest to explain first—I'm not ready to be a father.

Followup to:  Righting a Wrong Question, Zombies! Zombies?, A Premature Word on AI, On Doing the Impossible

There is a subproblem of Friendly AI which is so scary that I usually don't talk about it, because very few would-be AI designers would react to it appropriately—that is, by saying, "Wow, that does sound like an interesting problem", instead of finding one of many subtle ways to scream and run away.

This is the problem that if you create an AI and tell it to model the world around it, it may form models of people that are people themselves.  Not necessarily the same person, but people nonetheless.

If you look up at the night sky, and see the tiny dots of light that move over days and weeks—planētoi, the Greeks called them, "wanderers"—and you try to predict the movements of those planet-dots as best you can...

Historically, humans went through a journey as long and as wandering as the planets themselves, to find an accurate model.  In the beginning, the models were things of cycles and epicycles, not much resembling the true Solar System.

But eventually we found laws of gravity, and finally built models—even if they were just on paper—that were extremely accurate so that Neptune could be deduced by looking at the unexplained perturbation of Uranus from its expected orbit.  This required moment-by-moment modeling of where a simplified version of Uranus would be, and the other known planets.  Simulation, not just abstraction.  Prediction through simplified-yet-still-detailed pointwise similarity.

Suppose you have an AI that is around human beings.  And like any Bayesian trying to explain its enivornment, the AI goes in quest of highly accurate models that predict what it sees of humans.

Models that predict/explain why people do the things they do, say the things they say, want the things they want, think the things they think, and even why people talk about "the mystery of subjective experience".

The model that most precisely predicts these facts, may well be a 'simulation' detailed enough to be a person in its own right.