I'm going to stop going point-for-point on this, and this will probably be my final post on the matter. But the gist of my argument is this:
You say that it's reasonable to "look away", to consciously try to disconnect your emotions from reality. This is essentially sacrificing emotional epistemic rationality for emotional instrumental rationality. In that sense, I consider it theoretically reasonable: epistemic rationality is ultimately only a sub-goal of instrumental rationality.
But unless you're a perfect rationalist, it is extremely dangerous to have a policy of favoring instrumental rationality over epistemic rationality. It's virtually impossible to lie to yourself in a way that is not contagious. Unless you have complete information about the universe and total knowledge of how to apply it, you can never be sure that the lie you told to cover up one unfortunate truth won't catch you somewhere else -- and when it's a lie you've told yourself, a false thing you've willed yourself into believing, you can't even keep the truth at the back of your mind to make sure you maintain correspondence with reality.
Yours is the logic of conversion, the argument that says you should abandon truth for religion if it seems likely to make you happier. Maybe this is the case -- but only if you can be sure that reality will never come back and bite you in the ass. Because once you've given up that instinct for truth, you can't get it back. A lie you tell to yourself is self-reinforcing and can't be isolated. Most likely you will never be able to dig it out.
If you were perfect, you could entirely disjoin the emotional state you wished to feel from the emotional valuation you wished to decide with -- making one conscious and keeping the other deep inside your head. But you're not perfect, you're human -- and humans can't do that. One who tries to do so will find that their real, underlying, motivating emotions change to match the ones they consciously desire to feel -- and in doing so alter their actions.
So your choice is this: either change your emotions to match reality, with all the suffering that entails; or ignore reality for the sake of your emotions, and sacrifice your moral code in doing so.
Sacrificing epistemology is not something you can do once you've awakened as a rationalist.
This is essentially sacrificing emotional epistemic rationality for emotional instrumental rationality.
One thing that you're overlooking here is that the kind of self-modification Dan is talking about can't be done unless you actually have strong epistemic rationality with respect to your emotions -- strong enough to understand the judgment by which you arrived at the emotions in the first place.
...If you were perfect, you could entirely disjoin the emotional state you wished to feel from the emotional valuation you wished to decide with -- making one co
Related to: I'm Scared; Purchase utilons and fuzzies separately
Expanded from this comment.
You have awakened as a rationalist, discarded your false beliefs, and updated on new evidence. You understand the dangers of UFAI, you do not look away from death or justify it. You realize your own weakness, and the Vast space of possible failures.
And understanding all this, you feel bad about it. Very bad, in fact. You are afraid of the dangers of the future, and you are horrified by the huge amounts of suffering. You have shut up and calculated, and the calculation output that you should feel 3^^^3 times as bad as over a stubbed toe. And a stubbed toe can be pretty bad.
But this reaction of yours is not rational. You should consider the options of choosing not to feel bad about bad things happening, and choosing to feel good no matter what.
Your bad feelings, whether of fear, empathetic suffering, or something else, are probably counterproductive. Not only do you feel bad - a loss of utility in itself - but such feelings probably hurt, rather than help, your efforts to change the world for the better.
You may believe that your emotional outlook must be "rational": that it must correspond to your conscious estimates of the present or the future. Perhaps you expect to die of old age, or perhaps you are aware of people being tortured in secret prisons. You are forcing your emotions to match the future you foresee, and so you feel unhappy and afraid.
I suggest that you allow your emotions to become disconnected from your conscious long-term predictions. Stop trying to force yourself to be unhappy because you predict bad things. Say to yourself: I choose to be happy and unafraid no matter what I predict!
Emotions are not a a tool like rational thought, which you have to use in a way that corresponds to the real world. You can use them in any way you like. It's rational to feel happy about a bleak future, because feeling happy is a good thing and there is no point in feeling unhappy!
Being happy or not, afraid or not, does not have to be determined by your conscious outlook. The only things that force your mind to be unhappy are things like pain, hunger, loneliness, and the immediate expectation of these. If you accept that your goal is to be happy and unafraid as a fact independent of the future you foresee, you can find various techniques to achieve this.
Unfortunately such techniques vary for different people. This post doesn't discuss any: it is about the prerequisite decision to be happy.
Expecting to die of cancer in fifty years does not, in itself, cause negative emotions like fear. Imagining the death in your mind, and dwelling on it, does cause fear. In the first place, avoid thinking about any future problem that you are not doing anything about.
Use your natural defensive mechanisms, such as of not acknowledging unsolved problems, or compartmentalizing different beliefs. Don't dismiss them as biases or irrational practices. They exist for a good reason and have their proper use.
This does not mean that you should ignore problems on the conscious level. It is possible to decouple the two things, with practice. You can take long-term strategic actions (donate to SIAI, research immortality) without acutely fearing the result of failure by not imagining that result.
When you're faced with something terrible and you're not doing anything about it anyway, just look away. Defeat the implicit LW conditioning that tells you looking away from the suffering of others is wrong. It's wrong only if it affects your actions, not your emotions.