Being around here has made me think that I know everything interesting about the world and suppressed my excitement and joy from many minor things I could do. I also feel like my sense of wonder diminished. As I write this, I am a little unhappy, and in a period of depression, but I had similar feelings, if less intense, even before this period.

I was wondering whether you have any advice on how to restore this; or even better, how to "forget" as much rationality and transhumanism as possible (if not actually forgetting, then at least "to think and feel as I did before I read the Sequences")?

Uhm, it's evident I've not made my argument very clear (which is not a surprise, since I've wrote that stub in less than a minute).

Let me rephrase it in a way that addresses your point:

"Value alignment is very hard, because two people, both very intelligent and with a mostly coherent epistemology, can radically differ in their values because of a very tiny difference.
Think for example to neoreactionaries or rationalists converted to religions: they have a mostly coherent set of beliefs, often diverging from atheist or progressive rationalists in very few points."

Are we still talking about an AI that can be programmed at will?

I feel we are talking past each other, because reading the comment above I'm totally confused about what question you're answering...

Let me rephrase my question: if I substitute one of the parts of an AI with an inequivalent part, say a kidney with a solar cell, will its desires change or not?

A quick and dirty inspiration: that value alignment is very hard is showed by very intelligent people going neoreactionary or convert to a religion from atheism. They usually base their 'move' on a coherent epistemology, with just some tiny components that zag instead of zigging.

Comment at will, I'll expand with more thoughts.

http://lesswrong.com/r/discussion/lw/nwu/reality_is_arational/

-8 Yet still no one being able to refute my arguments.

In the ancient world it was very common to predict the imminent end of the world.

How so?

But that doesn't just reduce to a will of survival? I know that extracting certain salts from my blood is essential to my survival, so I want that my parts that do exactly that continue to do so. But I do not have any specific attachment to that functions just because a sub-part of me executes this. If I were in a simulation, say, even if I knew that my simulated kidneys worked in the same way I know I could continue to exist even without that function.
From the wording of your previous comments, it seemed that an AI conscious of its parts should have isomorphic desires, but the problem is that there could be many different isomorphisms, some of which are ridiculous.

When it feels those tendencies, it will feel desires that have nothing to do with paperclips.

Maybe, but they could still operate in harmony to reduce the world to a giant paperclips.

Because as I said, most humans would never even think of the doomsday argument. So the argument can't apply to them. In order to get the mathematical guarantee that 90% of people who use the argument will be correct, you need to restrict your reference class only to people familiar with the argument.

More generally, the copernican principle is that there is nothing particularly special about this exact moment in time. But we know there is something special. The modern world is very different than the ancient world. The probability of these ideas occurring to an ancient person, are very different than to a modern person. And so any anthropic reasoning should adjust for that probability.