Dennis Bider: A BASIC and ESSENTIAL though these days largely forgotten principle of liberal society is that it can be the case, and often is, that behavior X is NOT OK but that banning behavior X would also be NOT OK.

Pete, if you do that then being a casual decision theorist won't, you know, actually Win in the one shot case. Note that evolution doesn't produce organisms that cooperate in one shot prisoners dilemmas.

But Eliezer, you can't assume that Clippy uses the same decision making process that you do unless you know that you both unfold from the same program with different utility functions or something. If you have the code that unfolds into Clippy and Clippy has the code that unfolds into you it may be that you can look at Clippy's code and see that Clippy defects if his model of you defects regardless of what he does and cooperates if his model of you cooperates if your model of him cooperates, but you don't have his code. You can't say much about all possible minds or about all possible paperclip maximizing minds.

I would say that in my experience evangelicals and traditional Christians max out at in the low 140s IQ wise.

Nick: Do you imagine that they would tell you so? Also, you are a) young, and b) haven't been in any setting where people come from a large variety of social backgrounds.

Highly intelligent Christians, Dyson for instance, are likely to believe roughly the same things you do but frame them differently. Tegmark 4 = Spinoza's god, for instance.

Hopefully: You and may others. I will if I ever pull together the emotional resources to, which seem unusually high for me. It's very demanding of effort for me to address a large group, most of whom will fail to "get it" whatever I do.

Seconding Hopefully Anonymous on this point. Also emphasizing that not infrequently, when the accuracy of beliefs that a nerd can promote is low due to inferential distance or to gaps in intelligence, nerds tend to give true statements without bridging the inferential distance, predictably promoting less than maximally accurate beliefs, primarily, I would say, motivated by the self-righteous feeling they get from telling the truth and the feeling of superior righteous indignation they get from their perceived inferiors showing ignorance by not understanding. If you are an atheist and a Christian wants to know if you believe in God, there IS NO Bayesian level honest answer, but there never is when dealing with people who aren't at least ancient art rationalists. The most honest answer is "Yes", which will be interpreted as something like "I believe in not eating babies" or "I believe in expressing my loyalty towards people and groups I care about" or the like.

Seconding Scott Aaronson (especially on the tragedy of the US getting nukes too late) and Michael G R, though with the suggestion that the "demo" bomb could have been Trinity.

Eliezer: I think that the hypocrisy of the US is mostly in our maintaining a large arsenal but telling others that they can't have any, not in having used nukes. I don't see a case for .1% probability increase for nuclear war without using the bomb which is stronger than the case for the inverse, but I do see a million dead Japanese. Also, in terms of catastrophic risks, a couple more years of total war to encourage ultimately disastrous cultural changes in the US.

What I really want to know about WWII is why Hitler wasn't assassinated before it was too late. Surely if the situation with Archduke Ferdinand teaches us anything it's that Assassination can solve all of one's geo-political problems ;-(

Eliezer: I think that you misunderstand Roko, but that doesn't really matter, as he seems to understand you fairly well right now and to be learning effectively.

Unknown: Not at all. Utility maximization is very likely to lead to counterintuitive actions, and might even lead to humanly useless ones, but the particular actions it leads to are NOT whatever salient actions you wish to justify but are rather some very specific set of actions that have to be discovered. Seriously, you NEED to stop reasoning with rough verbal approximations of the math and actually USE the math instead. Agreement emerges from Bayes, but the mere fact that you call something agreement doesn't strongly suggest that it is Bayesian. Seriously I have tried to communicate with you, Carl has tried, and Nick has tried. You aren't interested in figuring out the actual counter-intuitive consequences of your beliefs, as you are too afraid that you would have to act, you know, counter-intuitively, which in actuality you don't do at all. As far as I can tell you aren't worth any of us wasting any more time on.

Lara: invertebrates are, I believe, generally believed to have MUCH simpler sets of neurotransmitters than vertebrates. Think how few genes fruit flies have.

Great comments thread! Thanks all!

Seconding Roko, Carl, HA, Nick T, etc.

Eliezer or Robin: Can you cite evidence for "we can more persuasively argue, for what we honestly believe". My impression is that it has been widely assumed in evolutionary psychology and fairly soundly refuted in the general psychology of deception, which tells us that the large majority of people detect lies at about chance and that similar effort seems to enable the development of the fairly rare skill of the detection of lies and evasion of such detection.

Carl: Unknown and Utilitarian could be distinct but highly correlated (we're both here after all). In principle we could see them as both unpacking the implications of some fairly simple algorithm. Have you noticed them both making the same set of mistakes in their efforts to understand Bayesian reasoning, anthropics, decision theory, etc? Still could be the same program running on different sets of wetware.

I second Hopefully on criticism of the strawman postmodernist. Honestly, I think that academic disciplines, or even schools, where everyone is completely full of it are extremely rare. There are thoughtful, intelligent, and honest people who frame important and fairly novel ideas in the terminology of sociology, academic feminism, Freudianism, behaviorism, even, math-help-us, Jung. Perfect intellectual honest and intelligence among humans are a chimera, and different disciplines aspire to different approximations thereof by establishing different sorts of standards for esteem and for publication.

As a general summary, I would say that the Enlightenment was, to a large extent, a set of proscriptions regarding what types of questions should and should not be asked, argumentative styles should and should not be used, and hand waves are and are not allowed. Some set of standards is necessary for functional discourse under imperfect honesty, and the Enlightenment standards enabled fruitful conversation among people of merely, perhaps, 70th percentile honesty and 96th percentile intelligence, an AMAZING standard that reshaped history. Sadly, each set of standards disables productive discussion of certain subject matter, and places one's beliefs on ultimately unsound foundations, so re-foundation is ultimately necessary. The more dysfunctional academic disciplines are largely those which try to deal, possibly prematurely, with subject matter for which we lack good honesty enforcing protocols, including subject matter foundational to the Enlightenment subject matter, but for individuals in those disciplines insights are indeed possible and low-hanging fruit are even abundant.