There's so much that can go wrong with such reasoning, given that intelligence (even at the size of a galaxy of Dyson spheres) is not a perfect God, as to render such arguments irrelevant and entirely worthless. Furthermore there's enough ways how the non-orthogonality can hold, such as e.g. almost all intelligences with wrong moral systems crashing or failing to improve, that are not covered by 'converges'.

meta: Tendency to talk seriously about products of very bad reasoning really puts an upper bracket on the sanity of newcomers to LW. As is the idea that very bad argument trumps authority (when it comes to the whole topic).

You can represent any form of agency with utility function that is 0 for doing what agency does not want to do, and 1 for doing what agency want to do. This looks like a special case of such triviality, as true as it is irrelevant. Generally one of the problems with insufficient training in math is the lack of training for not reading extra purpose into mathematical definitions.

I have no grounding in cogsci/popular rationality, but my initial impression of LW was along the lines of "hmm, this seems interesting, but nothing seems that new to me..." I stuck around for a while and eventually found the parts that interested me (hitting rocky ground around the time I reached the /weird/ parts), but for a long while the impression was that this site had too high a rhetoric to actual content ratio, and presented itself as more revolutionary than its content justifies.

My (better at rationality than me) OH had a more extreme first impression of approximately "These people are telling me nothing new, or vaguely new things that aren't actually useful, in a tone that suggests that it's going to change my life. They sound like a bunch of pompous idiots." He also stuck around though, and enjoyed reading the sequences as consolidating his existing ideas into concrete lumps of usefulness.

From these two limited points of evidence, I timidly suggest that although LW is pitched at generic rational folk, and contain lots of good ideas about rationality, the way things are written over-represent the novelty and importance of some of the ideas here, and may actively put off people who have good ideas about philosophy and rationality but treat them as "nothing big".

Another note - jumping straight into the articles helped neither of us, so it's probably a good idea to simplify navigation, as has already been mentioned, and make the "About" page more prominent, since that gives a good idea to someone new as to what actually happens on this site - something that is quite non-obvious.

Popularizing ideas from contemporary cognitive science and naturalized philosophy seems like a pretty worthy goal in and of itself. I wonder to what extent the "Less Wrong" identity helps this (by providing a convenient label and reference point), and to what extend it hurts (by providing an opportunity to dismiss ideas as "that Less Wrong phyg"). I suspect the former dominates, but the latter might be heard from more.

honest people can't stay self deluded for very long.

This is surely not true. Lots of wrong ideas last a long time beyond when they are, in theory, recognizably wrong. Humans have tremendous inertia to stick with familiar delusion, rather than replace them with new notions.

Consider any long-lived superstition, pseudoscience, etc. To pick an uncontroversial example, astrology. There were very powerful arguments against it going back to antiquity, and there are believers down to the present. There are certainly also conscious con artists propping up these belief structures -- but they are necessarily the minority of purported believers. You need more victims than con artists for the system to be stable.

People like Newton and Kepler -- and many eminent scientists since -- were serious sincere believers in all sorts of mystical nonsense -- alchemy, numerology, and so forth. I's possible for smart careful people to persistently delude themselves -- even when the same people, in other contexts, are able to evaluate evidence accurately and form correct conclusions.

I am a counterexample. I think Eliezer is a self-important wanker, but I have a favorable view of LW as a whole. I agree that I might be rare. I also wouldn't describe myself as a "part of the LW community." I think I attended a total of 1 meetup.

My first impression of lesswrong was of a community devoted to pop science, sci-fi, and futurism. Also, around that time singulartian was getting a bad name for good reasons (but it was the Kurzweil kind d'oh), and so I closed the tab thinking I wasn't missing anything interesting. It wasn't until years later when I was getting tired of the ignorance and arrogance of the skeptic community that I found my way back to lesswrong with some linked post that showed careful, honest thinking.

It would be a good idea to put up a highly visible link on the front page addressing new visitors' immediate criticisms. For example:

• Is this place a Kurzweil fanclub?
• Are you guys pulling most of the content on this site out of your ass?
• Why should I care what you people have to say? The people here seem weird.
• I think most of what you people believe is bullshit, am I not welcome here?

Another thing, the layout of this site will take people more than ten seconds to grok which is enough to have most people just leave. For instance, I'd rename 'discussion' to 'forum' and 'main' to 'rationality blog' or just 'blog'.

You know, an uncharitable reading of this would almost sort-of kinda maybe construe it as a rebuke of the LW community. Almost.

Some of the rationality may to significant extent be a subset of standard, but it has important omissions - in the areas of game theory for instance - and much more importantly significant miss-application such as taking the theoretically ideal approaches given infinite computing power as the ideal, and seeing as the best try the approximations to them which are grossly sub-optimal on the limited hardware where different algorithms have to be employed instead. One has to also understand that in practice computations have cost, and any form of fuzzy reasoning (anything other than very well verified mathematical proof) accumulates errors with each step, regardless of whenever it is 'biased' or not.

Choosing such a source for self education is definitely not common. As is the undue focus on what is 'wrong' about thinking (e.g. lists of biases) rather than on more effective alternatives to biases; if you remove the biases that won't in itself give you extra powers of rational thinking; your reasoning will be as sloppy as before and you'll simply be wrong in an unusual way (for instance you'll end up believing in unfalsifiable unjustified propositions other than God; it seems to me that this has occurred in practice)

edit: Note: he asked a question, I'm answering why it is seen as fringe, it may sound like unfair critique but I am just explaining what it looks like from outside. The world is not fair; if you use dense non-standard jargon, that raises the costs, and lowers the expected utility of reading what you wrote (because most people using non-standard jargon don't really have anything new to say). Processing has non zero utility cost, that must be understood, if the mainstream rationalists don't instantly see you as worth reading, they won't read you, that's only rational on their part. You must allow for other agents to act rationally. It is not always rational to even read an argument.

Actually, given that one could only read some small fraction of rationality related material, it is irrational to read anything but known best material, where you have some assurance that the authors have good understanding of the topic, including those parts that are not exciting, or seem too elementary, or go counter to the optimism - the sort of assurance you get when the authors of the material have advanced degrees.

edit: formatting, somewhat expanded.

large objects tend to violate 'laws of quantum mechanics' as we know them (the violation is known as gravity)

I cannot agree with this assertion. Except for the mysterious "measurement" thing, where only a single outcome is seen where many were possible (I'm intentionally use the word "seen" to describe our perception, as opposed to "occurs", which may irk the MWI crowd), the quantum world gracefully turns classical as the objects get larger (the energy levels bunch tighter together, the tunneling probabilities vanish exponentially, the interaction with the environment, resulting in decoherence, gets stronger, etc.).

This has not been shown to have anything to do with gravity, though Roger Penrose thinks that gravity may limit the mass of quantum objects, and I am aware of some research trying to test this assertion.

For all I know, someone might be writing a numerical code to trace through decoherence all the way to the microscopic level as we speak, based on the standard QM/QFT laws.