I feel this should not be in featured posts, as amusing as it was at the time

I don't think anyone denies that brain states have a strong influence on conscious experience, which is the only thing that Phineas Gage proved. The real question is how mechanistic matter can create subjective experience. For example, someone who was completely colorblind from birth could never understand what it felt like to see the color green, no matter how much neuroscience that person knew, i.e., you could never convey the sensation of "green" through a layout of a connectome or listing wavelengths of light.

However, this doesn't mean that there must be some magical substance which produces experience, and it does not mean that Whole Brain Emulation and AGI is impossible, which is the hasty conclusion reached by many non-materialists. Rather, it only poses problems for those who say that brain states are the same thing as conscious experience.

Lately I've noticed, both here and the wider LW-sphere, a trend towards rationalizing the status quo. For example, pointing out how seemingly irrational behavior might actually be rational when taking into account various factors. Has anyone else noticed the same?

At any rate I'm not sure if this represents an evolution (taking into account more subtleties) or regression (genuine change is too hard so let's rationalize) in the discourse.

A Christian proverb says: “The Church is not a country club for saints, but a hospital for sinners”. Likewise, the rationalist community is not an ivory tower for people with no biases or strong emotional reactions, it’s a dojo for people learning to resist them.

SlateStarCodex

I find it takes a great deal of luminosity in order to be honest with someone. If I am in a bad mood, I might feel that its my honest opinion that they are annoying when in fact what is going on in my brain has nothing to do with their actions. I might have been able to like the play in other circumstances, but was having a bad day so flaws I might have been otherwise able to overlook were magnified in my mind. etc.

This is my main fear with radical honesty, since it seems to promote thinking that negative thoughts are true just because they are negative. The reasoning going 'I would not say this if I were being polite, but I am thinking it, therefore it is true' without realizing that your brain can make your thoughts be more negative from the truth just as easily as it can make them more positive than the truth.

In fact, saying you enjoyed something you didnt enjoy, and signalling enjoyment with appropriate facial muscles (smiling etc) can improve your mood by itself, especially if it makes the other person smile.

Many intelligent people get lots of practice pointing out flaws, and it is possible that this trains the brain into a mode where one's first thoughts on a topic will be critical regardless of the 'true' reaction. If your brain automatically looks for flaws in something and then a friend asks your honest opinion you would tell them the flaws; but if you look for things to compliment your 'honest' opinion might be different.

tl;dr honesty is harder than many naively think, because our brains are not perfect reporters of their state, and even if they were good luck explaining your inner feelings about something across the inferential distance. Better to just adjust all your reactions slightly in the positive direction to reap the benefits of happier interactions (but only slightly, don't say you liked activities you loathed otherwise you'll be asked back, say they were ok but not your cup of tea etc)

he puts 2.72 on the table, and you put 13.28 on the table.

I'm confused...if the prediction does not come true (which you estimated as being 33 percent likely) you only gain $2.72? and if the most probable outcome does come true you lose 13.28?

I don't see the analogy. Paperclipping doesn't have to be an ineffable value for a paperclipper, and paperclippers don't have to be motivated by anything qualia-like.

Eliezer, thanks for clarifying. This is how I originally conceived you viewed the threat from superintelligent paperclip-maximisers, i.e. nonconscious super-optimisers. But I was thrown by your suggestion above that such a paperclipper could actually understand first-person phenomenal states, i.e, it's a hypothetical "full-spectrum" paperclipper. If a hitherto non-conscious super-optimiser somehow stumbles upon consciousness, then it has made a momentous ontological discovery about the natural world. The conceptual distinction between the conscious and nonconscious is perhaps the most fundamental I know. And if - whether by interacting with sentients or by other means - the paperclipper discovers the first-person phenomenology of the pleasure-pain axis, then how can this earth-shattering revelation leave its utility function / world-model unchanged? Anyone who is isn't profoundly disturbed by torture, for instance, or by agony so bad one would end the world to stop the horror, simply hasn't understood it. More agreeably, if such an insentient paperclip-maximiser stumbles on states of phenomenal bliss, might not clippy trade all the paperclips in the world to create more bliss, i.e revise its utility function? One of the traits of superior intelligence, after all, is a readiness to examine one's fundamental assmptions and presuppositions - and (if need be) create a novel conceptual scheme in the face of surprising or anomalous empirical evidence.

Taking this post in the way it was intended i.e. 'are there any reasons why such a policy would make people more likely to attribute violent intent to LW' I can think of one:

The fact that this policy is seen as necessary could imply that LW has a particular problem with members advocating violence. Basically, I could envision the one as saying: 'LW members advocate violence so often that they had to institute a specific policy just to avoid looking bad to the outside world'

And, of course, statements like 'if a proposed conspiratorial crime were in fact good you shouldn't talk about it on the internet' make for good out-of-context excerpts.