"Does my current utility function contain a global maximum at the case where I wish for and receive death right now?" is a really scary question to ask a genie.
I would prefer "I wish for the world to changed in such a manner as my present actual utility function, (explicitly distinct from my perception of my utility function), is at the global maximum possible without altering my present actual utility function."
Or in colloquial language "Give me what I want, not what I think I want."
That sounds pretty scary too. I don't think I am close enough to being an agent to have a well-defined utility function. If I do (paradoxical as it sounds), it's probably not something I would reflectively like. For example, I think I have more empathy for things I am sexually attracted to. But the idea of a world where everyone else (excluding me and a few people I really like) is a second-class citizen to hot babes horrifies me. But with the wrong kind of extrapolation, I bet that could be said to be what I want.
I can't easily describe any procedure I kn...
Here's the new thread for posting quotes, with the usual rules: