Note that this makes the struck-through text very difficult to search.

But remember that it's not just your own rationality that benefits you.

Are you saying that improving epistemic rationality is important because it benefits others as well as myself? This is true, but there are many other forms of self-improvement that would also have knock-on effects that benefit others.

I have actually read most of the relevant sequences, epistemic rationality really isn't low-hanging fruit anymore for me, although I wish I had known about cognitive biases years ago.

Hello everyone, Like many people, I come to this site via an interest in transhumanism, although it seems unlikely to me that FAI implementing CEV can actually be designed before the singularity (I can explain why, and possibly even what could be done instead, but it suddenly occurred to me that it seems presumptuous of me to criticize a theory put forward by very smart people when I only have 1 karma...).

Oddly enough, I am not interested in improving epistemic rationality right now, partially because I am already quite good at it. But more than that, I am trying to switch it off when talking to other people, for the simple reason (and I'm sure this has already been pointed out before) that if you compare three people, one who estimates the probability of an event at 110%, one who estimates it at 90%, and one who compensates for overconfidence bias and estimates it at 65%, the first two will win friends and influence people, while the third will seem indecisive (unless they are talking to other rationalists). I think I am borderline asperger's (again, like many people here) and optimizing social skills probably takes precedence over most other things.

I am currently doing a PhD in "absurdly simplistic computational modeling of the blatantly obvious" which better damn well have some signaling value. In my spare time, to stop my brain turning to mush, among other things I am writing a story which is sort of rationalist, in that some of the characters keep using science effectively even when the world is going crazy and the laws of physics seem to change dependent upon whether you believe in them. On the other hand, some of the characters are (a) heroes/heroines (b) awesomely successful (c) hippies on acid who do not believe in objective reality (not that I am implying that all hippies/people who use lsd are irrational). Maybe the point of the story is that you need more than just rationality? Or that some people are powerful because of rationality, while others have imagination, and that friendship combines their powers in a my little pony like fashion? Or maybe its all just an excuse for pretentious philosophy and psychic battles?

I must conclude one (or more) of a few things from this post, none of them terribly flattering.

1. You do not actually believe this argument.
2. You have not thought through its logical conclusions.
3. You do not actually believe that AI risk is a real thing.
4. You value the plus-votes (or other social status) you get from writing this post more highly than you value marginal improvements in the likelihood of the survival of humanity.

I find it rather odd to be advocating self-censorship, as it's not something I normally do. However, I think in this case it is the only ethical action that is consistent with your statement that the argument "might work", if I interpret "might work" as "might work with you as the gatekeeper". I also think that the problems here are clear enough that, for arguments along these lines, you should not settle for "might" before publicly posting the argument. That is, you should stop and think through its implications.

But it turns out that tasks that seem easy to us can in fact require such a specialized region.

In a way, this really shouldn't be surprising at all. Any common mental task which has its own specialised region will of course seem easy to us, because it doesn't make use of the parts of the brain we are consciously aware of.

I think the "just make sure everyone agrees on everything" idea is a good one, but quite difficult in practice.

With this line of argument, there's literally no point at which one can sit back and say, "I've fulfilled my duty to charity - there's nothing more to do".

That reminds me very strongly of something I read in a Jewish prayerbook, or possibly the Talmud, a long time ago. I can't find it with google (translation being what it is), but here's my best recollection:

"It is a command we are given repeatedly in Torah. But what does it really mean to 'love your neighbour as yourself'? ... Never would a man say 'I have fulfilled my obligation to myself'. In the same way, you have never fulfilled your obligation to your neighbour."

Taking the comparison back the other way raises what I think is an interesting question. People have no issues with the idea that your obligations to yourself are unbounded, so why does having unbounded obligations to others pose a problem?

There's literally no point at which one can sit back and say, "I've fulfilled my duty to myself - there's nothing more to do".

Oh jeez, just screw it. Seems that I can't say even a slightly, tangentially ideological thing without fucking up.

I think the criticism of 6 is a misunderstanding. It doesn't say "the world resembles the ancestral savanna", it says "the world resembles the ancestral savanna more than say a windowless office". The best environment is unlikely to be anything like the ancestral savanna, but it's likely to be closer to that than to a windowless office, in terms of sensory experience. The point I think is not the specifics of the environment, but that it engages with our bodies and senses in a way that we, as evolved creatures, find satisfying, and in a way that the purely mental stimulation available in the office does not.

That's what I took away from the linked post.