Marshall I think that's a bit of a cop-out.

Why wouldn't a PM cheat? Why would it ever remain inside the frame of the game?

Would two so radically different agents even recognize the same pay-off frame?

"The different one" will have different pay-offs - and I will never know them and am unlikely to benefit fra any of them.

In my world a PM is chaotic, just as I am chaotic in his. Thus we are each other's enemy and must hide from the other.

No interaction because otherwise the number of crying mothers and car dealerships will always be higher.

I think you guys are calculating too much and talking too much.

Regardless of the "intelligence" of a PM, in my world that is a pretty stupid thing to do. I would expect such a "stupid" agent to do chaotic things indeed evil things. Things I could not predict and things I could not understand.

In an interactioin with a PM I would not expect to win, regardless of how clever and intelligent I am. Maybe they only want to make paperclips (and play with puppies), but such an agent will destroy my world.

I have worked with such PM's.

I would never voluntarily choose to interact with them.

I think this is your most important post.

I wonder - did we all understand this parable in the same way? I doubt it!

I think Eliezer is saying: We know on average what's right and what's wrong. It is part of being human. There are different versions of being human and thus our rights and wrongs are embedded in time and place. It is in the "Thickness" of living with others we know what and how to do. Mostly it is easy. Because morality is human. Stopping up and thinking about all this gives what Michael Vassar calls "Aack!!! Too... many... just so stories... bad evolutionary psychology... comment moderation... failing."

Marshall, how is your "usefulness" not isomorphic to the word "good"? Useful for what?

I suppose I just want to avoid the preachiness of the word good. It is unfortunately coherent to die for goodness. It is not very useful to die for usefullness.

Useful for what? This doesn't seem like a useful question. Usefulness is obvious and thus no need to ask.

I do not wish to lose my way or be carried away by the bigness of the nominalisation "morality". Occam's Razor should also be applied here - in a pleasant and gentle way.

Hopefully: My suggestion is, that it is the use of the metaphor "illusion", which is unfortunate. In the process - Search Find (have a beer) Execute there is no room for illusion. Just as David Copperfield works very hard whilst making illusions but is himself under no illusion. In other words "illusion" is an out of process perspective. It is in the "I" of the beholder. You hope(fully) expect that the search process can be improved by the intervention of "I". Why should that be? Would the search process be improved by a coin flip? In other words dissolving "free will" dissolves the "user-illusion" of "Eye". And this plays havoc with your persistence odds. You want your illusion to persist?

Clearly, clear thinking is opaquely difficult. The future is not random, Roland, the future is just unknown. The future will have about as much structure as the now has, but we do not yet know all the details. This can possibly be construed as "randomness" in our thinking, but it is not randomness in the world. Another POV would call this "randomness in our thinking" as uncertainty. Roland "don't fall into this trap" means don't let determinism dertermine you - or worse - let Roland determine what you are determined to do. Maybe the Don't Panic button would be better and thusly your future was.

How can I think a thought? The river that flows without a drop.

Am I thinking the next thought? Chemicals, doing what they ought.

With time an illusion. The I that says it's me, is a figment too.

I struggle to choose to do what must be done.

Don't ask who I am. But observe what was done.