Kaj makes the efficiency argument in favor of full-fledged AI, but what good is efficiency when you have fully surrendered your power?

What good is being the president of a corporation any more, when you've just pressed a button that makes a full-fledged AI run it?

Forget any leadership role in a situation where an AI comes to life. Except in the case that it is completely uninterested in us and manages to depart into outer space without totally destroying us in the process.

Why build an AI at all?

That is, why build a self-optimizing process?

Why not build a process that accumulates data and helps us find relationships and answers that we would not have found ourselves? And if we want to use that same process to improve it, why not let us do that ourselves?

Why be locked out of the optimization loop, and then inevitably become subjects of a God, when we can make ourselves a critical component in that loop, and thus 'be' gods?

I find it perplexing why anyone would ever want to build an automatic self-optimizing AI and switch it to "on". No matter how well you planned things out, not matter how sure you are of yourself, by turning the thing on, you are basically relinquishing control over your future to... whatever genie it is that pops out.

Why would anyone want to do that?

My earlier comment is not to imply that I think "maximization of human happiness" is the most preferred goal.

An easily obvious one, yes. But faulty; "human" is a severely underspecified term.

In fact, I think that putting in place a One True Global Goal would require ultimate knowledge about the nature of being, to which we do not have access currently.

Possibly, the best we can do is come up with plausible global goal that suits us for medium run, while we try to find out more.

That is, after all, what we have always done as human beings.

Eliezer: You have perhaps already considered this, but I think it would be helpful to learn some lessons from E-Prime when discussing this topic. E-Prime is a subset of English that bans most varieties of the verb "to be".

I find sentences like "murder is wrong" particularly underspecified and confusing. Just what, exactly, is meant by "is", and "wrong"? It seems like agreeing on a definition for "murder" is the easy part.

It seems the ultimate confusion here is that we are talking about instrumental values (should I open the car door?) before agreeing on terminal values (am I going to the store?).

If we could agree on some well-defined goal, e.g. maximization of human happiness, we could much more easily theorize on whether a particular case of murder would benefit or harm that goal.

It is, however, much harder to talk about murders in general, and infeasible to discuss this unless we have agreed on a terminal value to work for.

frelkins: Should I apologize, then, for not yet having developed sufficient wit to provide pleasure with style to those readers who are not pleased by the thought?

Cynicism is warranted to the extent that it leads to a realistic assessment and a predictive model of the world.

Cynicism is exaggerated when it produces an unrealistic, usually too pessimistic, model of the world.

But to the extent that cynicism is a negative evaluation of "what is", I am not being a cynic in this topic.

I am not saying, bitterly, how sad it is that most people are really motivated by their selfishness, and how sad the world is because of this, etc.

What I am saying is that selfishness is okay. That recognizing your selfishness is the healthiest state. I am saying not that people who are selfish are corrupting the world. I am saying that people who are self-righteous are.

I understand people who want to reshape the world because they want it to be different, and are honest about this selfish preference and endeavor. I respect that.

What I don't respect is people who are self-righteous in thinking that they know how to reshape the world to make other people happy, and do not see how self-anchored their motivation is. They are trying to do the same thing as those people who want to reshape the world selfishly. But the self-righteous ones, they sell what they are doing as being "higher on a moral ladder", because, obviously, they know what is good for everyone.

I think that sort of behavior is just pompous, arrogant, and offensive.

Be honest. Do things because of you. Don't do things because of others. Then, we can all come together and agree sensibly on how to act as to not step on each other's toes.

But don't be running around "healing" the world, pretending like you're doing it a favor.

Phillip Huggan - let me just say that I think you are an arrogant creature that does much less good to the world than he thinks. The morality you so highly praise only appears to provide you with a reason to smugly think of yourself as "higher developed" than others. Its benefit to you, and its selfish motivation, is plainly clear.

Phillip Huggan: "Denis, are you claiming there is no way to commit acts that make others happy?"

Why the obsession with making other people happy?

Phillip Huggan: "Or are you claiming such an act is always out of self-interest?"

Such acts are. Stuff just is. Real reasons are often unknowable; and if known, would be trivial, technical, mundane.

In general, I wouldn't say self-interest. It is not in your self interest to cut off your penis and eat it, for example. But some people desire it and act on it.

Desire. Not necessarily logical. Does not necessarily make sense. But drives people's actions.

Reasons for desire? Unknowable.

And if known?

Trivial. Technical. Mundane.

Phillip Huggan: "The former position is absurd, the latter runs into the problem that people who jump on grenades, die."

I can write a program that will erase itself.

Doesn't mean that there's an overarching morality of what programs should and should not do.

People who jump on grenades do so due to an impulse. That impulse comes from cached emotions and thoughts. You prime yourself that it's romantic to jump on a grenade, you jump on a grenade. Poof.

Stuff is. Fitting stuff that happens into a moral framework? A hopeless endeavor for misguided individuals seeking to fulfil the romantic notion that things should make sense.

Phillip Huggan: "A middle class western individual, all else equal, is morally better by donating conspicious consumption income to charity, than by exercising the Libertarian market behaviour of buying luxury goods."

Give me a break. You gonna contribute to a charity to take care of all the squid in the ocean? The only justification not to is if you invent an excuse why they are not worth caring about. And if not the squid, how about gorillas, then? Baboons, and chimpanzees?

If we're going to intervene because a child in Africa is dying of malaria or hunger - both thoroughly natural causes of death - then should we not also intervene when a lion kills an antelope, or a tribe of chimpanzees is slaughtered by their neighbors?

You have to draw a line somewhere, or else your efforts are hopeless. Most people draw the line at homo sapiens. I say that line is arbitrary. I draw it where it makes sense. With people in my environment.

Thanks for the link to The People's Romance!

Disagreeing with Mr. Huggan, I'd say Obert is the one without a clue.

Obert seems to be trying to find some external justification for his wants, as if it's not sufficient that they are his wants; or as if his wants depend on there being an external justification, and his mental world would collapse if he were to acknowledge that there isn't an external justification.

I would compare morality to patriotism in the sense of the Onion article that Robin Hanson recently linked to. Much like patriotism, morality is something adopted by people who like to believe in Great Guiding Ideas. Their intellect drives them to recognize that the idea of a god is ridiculous, but the religious need remains, so they try to replace it with a principle. A self-generated principle which they try to think is independent and universal and not self-generated at all. They create their own illusion as a means of providing purpose for their existence.

Unknown: "For all those who have said that morality makes no difference to them, I have another question: if you had the ring of Gyges (a ring of invisibility) would that make any difference to your behavior?"

What sort of stupid question is this? :-) But of course! If I gave you a billion dollars, would it make any difference to your behavior? :-)