I actually thought Eliezer that you did rather poorly in this dialogue. You and your logic reached their limits. The tools you drew on were from too narrow a scope and didn't match Lanier's breadth. I am surprised (and worried) that all the other comments "take your side". I think this "event" requires som updating by you, Eliezer. Person to person - you lost. And I think this phrase "I was a little too reluctant to interrupt" is an example of cognitive dissonance and not the truth of the matter at all.

@Caledonian

How can you ask this question "What is actual evil?"? You must know. Perhaps not definitively, but surely well enough, that we others would recognize it and agree.

What is the function of this false barrier that is no barrier?

May I suggest that the asking of this question is an example of the answer, (whilst I accept, that evil done, does not have to be intended)?

Vassar - your English is encrypted - more an assumption of intelligence than a sign.

EY - I admire your work. Along with Robin this is the best Show in Town and I will miss it, when it stops.

I actually doubt whether you are accomplishing anything - but this does not seem so important to me, because the effort itself is worthwhile. And we are educated along the way.

This is a youthful blog with youthful worries. From the vantage point of age worrying about intelligence seems like a waste of time and unanswerable to boot.

But those are the stones in your shoes.

Marshall I think that's a bit of a cop-out.

Why wouldn't a PM cheat? Why would it ever remain inside the frame of the game?

Would two so radically different agents even recognize the same pay-off frame?

"The different one" will have different pay-offs - and I will never know them and am unlikely to benefit fra any of them.

In my world a PM is chaotic, just as I am chaotic in his. Thus we are each other's enemy and must hide from the other.

No interaction because otherwise the number of crying mothers and car dealerships will always be higher.

I think you guys are calculating too much and talking too much.

Regardless of the "intelligence" of a PM, in my world that is a pretty stupid thing to do. I would expect such a "stupid" agent to do chaotic things indeed evil things. Things I could not predict and things I could not understand.

In an interactioin with a PM I would not expect to win, regardless of how clever and intelligent I am. Maybe they only want to make paperclips (and play with puppies), but such an agent will destroy my world.

I have worked with such PM's.

I would never voluntarily choose to interact with them.

I think this is your most important post.

I wonder - did we all understand this parable in the same way? I doubt it!

I think Eliezer is saying: We know on average what's right and what's wrong. It is part of being human. There are different versions of being human and thus our rights and wrongs are embedded in time and place. It is in the "Thickness" of living with others we know what and how to do. Mostly it is easy. Because morality is human. Stopping up and thinking about all this gives what Michael Vassar calls "Aack!!! Too... many... just so stories... bad evolutionary psychology... comment moderation... failing."

Marshall, how is your "usefulness" not isomorphic to the word "good"? Useful for what?

I suppose I just want to avoid the preachiness of the word good. It is unfortunately coherent to die for goodness. It is not very useful to die for usefullness.

Useful for what? This doesn't seem like a useful question. Usefulness is obvious and thus no need to ask.

I do not wish to lose my way or be carried away by the bigness of the nominalisation "morality". Occam's Razor should also be applied here - in a pleasant and gentle way.

Why keep on about "morality"? Isn't this just a type of con-word used by ministers of religion, teachers and politicians to impress on us the need to be good and improve (in ways which they decide)? Can't we just abolish this word and tune it out? We all drive on the correct side of the road, because it is useful and it is seen to be useful by all. This is morality - small useful rules for getting along. There is no mystery about where they come from. We find ways to avoid bumping into things. We are brought up with this implicit usefulness and maintain it. In the luxury of our affluence it is no longer useful to boil the cat and we need our dogs to understand our language. Thus our circles of "utility" expand.