Note:  After writing this post, I realized there's a lot I need to learn about this subject.  I've been thinking a lot about how I use the word "elitism" and what it meant to me.  I was unaware that there are a large number of people who use the word to describe themselves and mean something totally different from the definition that I had.  This resulted in my perception that people who were using the word to describe themselves were being socially inept.  I now realize that it's not a matter of social ineptness, that it may be more of a matter of political sides.  I also realized that mind-kill reactions may be influencing us here (myself included).  So, now my goal is to make sure I understand both sides thoroughly to transcend these mind-kill reactions and explain to others how I accomplished this so that none of us has to have them.  I think these sides can get along better.  That is what I ultimately want - for the gifted population and the rest of the world to understand one another better, for the privileged and the disadvantaged to understand one another better, and for the tensions between those groups to be reduced so that we can work together effectively.  I realize that this is not a simple undertaking, but this is a very important problem to me.  I see this being an ongoing project in my life.  If I don't seem to understand your point of view on this topic, please help me update.  I want to understand it.

TLDR: OMG a bunch of people seem to want to use the word "elitist" to describe LessWrong but I know that this can provoke hatred.  I don't want to be smeared as an elitist.  I can't fathom why it would be necessary for us to call ourselves "elitists".

I have noticed a current of elitism on LessWrong.  I know that not every person here is an elitist, but there are enough people here who seem to believe elitism is a good thing (13 upvotes!?) that it's worth addressing this conflict.  In my experience, the word "elitism" is a triggering word - it's not something you can use easily without offending people.  Acknowledging intellectual differences is a touchy subject also, very likely to invite accusations of elitism.  From what I've seen, I'm convinced that using the word "elitism" casually is a mistake, and referring to intellectual differences incautiously is also risky.

Here, I analyze the motives behind the use of the word elitism, make a suggestion for what the main conflict is, mention a possible solution, talk about whether the solution is elitist, what elitism really means, and what the consequences may be if we allow ourselves to be seen as elitists.

The theme I am seeing echoed throughout the threads where elitist comments surfaced is "We want quality" and "We want a challenging learning environment".  I agree that quality goals and a challenging environment are necessary for refining rationality, but I disagree that elitism is needed.

I think the problem comes in at the point where we think about how challenging the environment should be.  There's a conflict between the website's main vision: spreading rationality (detailed in: Rationality: Common Interest of Many Causes) and striving for the highest quality standards possible (detailed in Well-Kept Gardens Die By Pacifism).

If the discussions are geared for beginners, advanced people will not learn.  If the discussions are geared for advanced people, beginners are frustrated.  It's built into our brains.  Psychologist Mihaly Csikszentmihalyi, author of "Flow: The psychology of optimal experience" regards flow, the feeling of motivation and pleasure you get when you're appropriately challenged, to be the secret to happiness and he explains that if you aren't appropriately challenged, you're either going to feel bored or frustrated depending on whether the challenge is too small or too great for your ability level.

Because our brains never stop rewarding and punishing us with flow, boredom and frustration, we strive for that appropriate challenge constantly.  Because we're not all at the same ability level, we're not all going to flow during the same discussions.  We can't expect this to change, and it's nobody's fault.

This is a real conflict, but we don't have to choose between the elitist move of blocking everyone that's not at our level vs. the flow killing move of letting the challenge level in discussions decrease to the point where it results in everyone's apathy - we can solve this.

Why bother to solve it?  If your hope is to raise the sanity waterline, you cannot neglect those who are interested in rational thought but haven't yet gotten very far.  Doing so would limit your impact to a small group, failing to make a dent in overall sanity.  If you neglect the small group of advanced rationalists, then you've lost an important source of rational insights that people at every level might learn from and you will have failed to attract the few and precious teachers who will assist the beginners in developing further faster.

And there is a solution; summarized in one paragraph:  Make several areas divided by their level of difficulty.  Advanced learners can learn in the advanced area, beginners in the beginner area.  That way everyone learns.  Not every advanced person is a teacher, but if you put a beginner area and an advanced area on the same site, some people from the advanced area will help get the beginners further.  One-on-one teaching isn't the only option - advanced people might write articles for beginners and get through to thousands at once.  They might write practice quizzes for them to do (not hard to implement from a web developer's perspective).  There are other things.  (I won't get into them here.)

This brings me to another question: if LessWrong separates the learning levels, would the separation qualify as elitism?

I think we can all agree that people don't learn well in classes that are too easy for them.  If you want advanced people to improve, it's an absolute necessity to have an advanced area.  I'm not questioning that.  I'm questioning whether it qualifies under the definition of elitism:

(dictionary.com)

Spreading rationality empowers people. If you wanted to take power over them, you'd horde it.  By posting our rational insights in public, we share them.  We are not hoarding them and demanding to be made rulers because of our power.  We are giving them away and hoping they improve the world. 

Using rationality as a basis for rule makes no sense anyway.  If you have a better map of the territory, people should update because you have a better map (assuming you overcome inferential distances).  Forcing an update because you want to rule would only amount to an appeal to authority or coercion.  That's not rational.  If you show them a more complete map and they update, that isn't about you - you should be updating your map when the time comes, too.  It's the territory that rules us all.  You are only sharing your map.

For the second definition, there are two pieces.  "Consciousness of or pride in" and "select or favored group".  I can tell you one thing for certain: if you form a group of intellectual elitists, they will not be considered "select or favored" by the general population.  They will be treated as the scum on the bottom of scum's shoe.

For that reason, any group of intellectual elitists will quickly become an oxymoron.  First, they'll have to believe that they are "select and favored" when they are not, and perhaps justify this with "we are so deserving of being select and favored that no one can see it but us" (which may make them hopelessly unable to update).  Second, the attitude of superiority is likely to provoke such anti-intellectual counter-prejudice that the resulting oppression could make them ineffectual.  Powerless to get anywhere because they are so hated, their "superiority" will make them into second class citizens.  You don't achieve elite status by being an intellectual elitist.

In the event that LessWrong was considered "select" or "favored" by the outside population, would "consciousness" of that qualify the members as elitists?  If you use the literal definition of "consciousness", you can claim a literal "yes" - but it would mean that simply acknowledging a (hypothetical) fact (independent market research surveys, we'll say) should be taken as automatic proof that you're an arrogant scumbag.  That would be committing Yvain's "worst argument in the world", guilt by association.  We can't assume that everyone who acknowledges popularity or excellence is guilty of wrongdoing.

So let's ask this: Why does elitism have negative connotations? What does it REALLY mean when people call a group of intellectuals "elitists"?

I think the answer to this is in Jane Elliot's brown eyes, blue eyes experiment.  If you're not familiar with it, a school teacher named Jane Elliot, horrified by the assassination of Martin Luther King, Jr. decided to teach her class a lesson about prejudice.  She divided the class into two groups - brown eyes and blue eyes.  She told them things like brown eyed kids are smarter and harder-working than blue eyed kids.  The children reacted dramatically:

"When several of the brown-eyed kids who had problems reading went to their primer that morning, they whizzed through sentences"

"A smart blue-eyed girl, who had never had problems with her multiplication tables, started making all kinds of mistakes."

"During afternoon recess, the girl came running back to Room 10, sobbing. Three brown-eyed girls had ganged up on her, and one had hit her, warning, “You better apologize to us for getting in our way because we’re better than you are."

When people complain of elitism, what they seem to be reacting to is a concern that feeling "better than others" will be used as an excuse for abuse - either via coercion, or by sabotaging their sense of self-worth and intellectual performance.

The goal of LessWrong is to spread rationality in order to make a bigger difference in the world.  This has nothing to do with abusing people.  Just because some people with advanced abilities choose to use them as an excuse to abuse other people, it doesn't mean that anybody here has to do that.  Just because some of us might have advanced abilities and are also aware of them does not mean we need to commit Yvain's "the worst argument in the world" by assuming the guilt that comes with elitism.  We can reject this sort of thinking.  If people tell you that you're an elitist because you want a challenging social environment to learn in, or because you want to make the project that is the LessWrong blog as high quality as it can be, you can refuse to be labeled guilty.

Refusing to be guilty by association takes more work than accepting the status quo but what would happen if we allowed ourselves to be disrespected for challenging ourselves and striving for quality?  If we agree with them, we're viewing positive character traits as part of a problem.  That encourages people to shoot themselves in the foot - and they can point that same gun at all of humanity's potential, demanding that nobody seeks the challenging social environment they need to grow, that nobody sets any learning goals to strive for because quality standards are elitist.  To allow a need for challenges and standards to be smeared as elitist will only hinder the spread of rationality.

How many may forgo refining rationality because they worry it will make them look like an elitist?

These are the reasons I choose to be non-abusive and to send a message to the world that non-abusive intellectuals exist.

What do you think of this?

In the Muehlhauser-Hibbard Dialogue on AGI, Hibbard states it will be "impossible to decelerate AI capabilities" but Luke counters with "Persuade key AGI researchers of the importance of safety ... If we can change the minds of a few key AGI scientists, it may be that key insights into AGI are delayed by years or decades." and before I read that dialogue, I had come up with three additional ideas on Heading off a near-term AGI arms race. Bill Hibbard may be right that "any effort expended on that goal could be better applied to the political and technical problems of AI safety" but I doubt he's right that it's impossible.

How do you prove something is impossible?  You might prove that a specific METHOD of getting to the goal does not work, but that doesn't mean there's not another method.  You might prove that all the methods you know about do not work.  That doesn't prove there's not some other option you don't see.  "I don't see an option, therefore it's impossible." is only an appeal to ignorance.  It's a common one but it's incorrect reasoning regardless.  Think about it.  Can you think of a way to prove that a method that does work isn't out there waiting to be discovered without saying the equivalent of "I don't see any evidence for this." We can say "I don't see it, I don't see it, I don't see it!" all day long. 

I say: "Then Look!"

How often do we push past this feeling to keep thinking of ideas that might work?  For many, the answer is "never" or "only if it's needed".  The sense that something is impossible is subjective and fallible.  If we don't have a way of proving something is impossible, but yet believe it to be impossible anyway, this is a belief.  What distinguishes this from bias? 

I think it's a common fear that you may waste your entire life on doing something that is, in fact, impossible.  This is valid, but it's completely missing the obvious:  As soon as you think of a plan to do the impossible, you'll be able to guess whether it will work.  The hard part is THINKING of a plan to do the impossible.  I'm suggesting that if we put our heads together, we can think of a plan to make an impossible thing into a possible one.  Not only that, I think we're capable of doing this on a worthwhile topic.  An idea that's not only going to benefit humanity, but is a good enough idea that the amount of time and effort and risk required to accomplish the task is worth it.

Here's how I am going to proceed: 

Step 1: Come up with a bunch of impossible project ideas. 

Step 2: Figure out which one appeals to the most people. 

Step 3: Invent the methodology by which we are going to accomplish said project. 

Step 4: Improve the method as needed until we're convinced it's likely to work.

Step 5: Get the project done.

Impossible Project Ideas

• Decelerate AI Capabilities Research: If we develop AI before we've figured out the political and technical safety measures, we could have a disaster.  Luke's Ideas (Starts with "Persuade key AGI researchers of the importance of safety").  My ideas.
• Solve Violent Crime: Testosterone may be the root cause of the vast majority of violent crime, but there are obstacles in treating it. 
• Syntax/static Analysis Checker for Laws: Automatically look for conflicting/inconsistent definitions, logical conflicts, and other possible problems or ambiguities. 

• Understand the psychology of money

• Rational Agreement Software:  If rationalists should ideally always agree, why not make an organized information resource designed to get us all to agree?  This would track the arguments for and against ideas in such a way where each piece can be verified logically and challenged, make the entire collection of arguments available in an organized manner where none are repeated and no useless information is included, and it would need to be such that anybody can edit it like a wiki, resulting in the most rational outcome being displayed prominently at the top.  This is especially hard because it would be our responsibility to make something SO good, it convinces one another to agree, and it would have to be structured well enough that we actually manage to distinguish between opinions and facts. Also, Gwern mentions in a post about critical thinking that argument maps increase critical thinking skills.
• Discover unrecognized bias:  This is especially hard since we'll be using our biased brains to try and detect it.  We'd have to hack our own way of imagining around the corners, peeking behind our own minds.

• Logic checking AI: Build an AI that checks your logic for logical fallacies and other methods of poor reasoning.

Add your own ideas below (one idea per comment, so we can vote them up and down), make sure to describe your vision, then I'll list them here.

Figure out which one appeals to the most people.

Assuming each idea is put into a separate comment, we can vote them up or down.  If they begin with the word "Idea" I'll be able to find them and put them on the list.  If your idea is getting enough attention obviously, it will at some point make sense to create a new discussion for it.

Non-causal models

Non-causal models are quite common in many fields, and can be quite accurate. Here predictions are made, based on (a particular selection of) past trends, and it is assumed that these trends will continue in future. There is no causal explanation offered for the trends under consideration: it's just assumed they will go on as before. Non-causal models are thus particularly useful when the underlying causality is uncertain or contentious. To illustrate the idea, here are three non-causal models in computer development:

1. Moore's laws about the regular doubling of processing speed/hard disk size/other computer related parameter.
2. Robin Hanson's model where the development of human brains, hunting, agriculture and the industrial revolution are seen as related stages of accelerations of the underlying economic rate of growth, leading to the conclusion that there will be another surge during the next century (likely caused by whole brain emulations or AI).
3. Ray Kurzweil's law of time and chaos, leading to his law of accelerating returns. Here the inputs are the accelerating evolution of life on earth, the accelerating 'evolution' of technology, followed by the accelerating growth in the power of computing across many different substrates. This leads to a consequent 'singularity', an explosion of growth, at some point over the coming century.

Before anything else, I should thank Moore, Hanson and Kurzweil for having the courage to publish their models and put them out there where they can be critiqued, mocked or praised. This is a brave step, and puts them a cut above most of us.

That said, though I find the first argument quite convincing, I find have to say I find the other two dubious. Now, I'm not going to claim they're misusing the outside view: if you accuse them of shoving together unrelated processes into a single model, they can equally well accuse you of ignoring the commonalities they have highlighted between these processes. Can we do better than that? There has to be a better guide to the truth that just our own private impressions.

This is based on a discussion in #lesswrong a few months back, and I am not sure how to resolve it.

Setup: suppose the world is populated by two groups of people, one just wants to be left alone (labeled Jews), the other group hates the first one with passion and want them dead (labeled Nazis). The second group is otherwise just as "good" as the first one (loves their relatives, their country and is known to be in general quite rational). They just can't help but hate the other guys (this condition is to forestall the objections like "Nazis ought to change their terminal values"). Maybe the shape of Jewish noses just creeps the hell out of them, or something. Let's just assume, for the sake of argument, that there is no changing that hatred.

Is it rational to exterminate the Jews to improve the Nazi's quality of life? Well, this seems like a silly question. Of course not! Now, what if there are many more Nazis than Jews? Is there a number large enough where exterminating Jews would be a net positive utility for the world? Umm... Not sure... I'd like to think that probably not, human life is sacred! What if some day their society invents immortality, then every death is like an extremely large (infinite?) negative utility!

Fine then, not exterminating. Just send them all to concentration camps, where they will suffer in misery and probably have a shorter lifespan than they would otherwise. This is not an ideal solutions from the Nazi point of view, but it makes them feel a little bit better. And now the utilities are unquestionably comparable, so if there are billions of Nazis and only a handful of Jews, the overall suffering decreases when the Jews are sent to the camps.

This logic is completely analogous to that in the dust specks vs torture discussions, only my "little XML labels", to quote Eliezer, make it more emotionally charged. Thus, if you are a utilitarian anti-specker, you ought to decide that, barring changing Nazi's terminal value of hating Jews, the rational behavior is to herd the Jews into concentration camps, or possibly even exterminate them, provided there are enough Nazi's in the world who benefit from it.

This is quite a repugnant conclusion, and I don't see a way of fixing it the way the original one is fixed (to paraphrase Eliezer, "only lives worth celebrating are worth creating").

EDIT: Thanks to CronoDAS for pointing out that this is known as the 1000 Sadists problem. Once I had this term, I found that lukeprog has mentioned it on his old blog. 

This article should really be called "Patching the argumentative flaw in the Sequences created by the Quantum Physics Sequence".

There's only one big thing wrong with that Sequence: the central factual claim is wrong. I don't mean the claim that the Many Worlds interpretation is correct; I mean the claim that the Many Worlds interpretation is obviously correct. I don't agree with the ontological claim either, but I especially don't agree with the epistemological claim. It's a strawman which reduces the quantum debate to Everett versus Bohr - well, it's not really Bohr, since Bohr didn't believe wavefunctions were physical entities. Everett versus Collapse, then.

I've complained about this from the beginning, simply because I've also studied the topic and profoundly disagree with Eliezer's assessment. What I would like to see discussed on this occasion is not the physics, but rather how to patch the arguments in the Sequences that depend on this wrong sub-argument. To my eyes, this is a highly visible flaw, but it's not a deep one. It's a detail, a bug. Surely it affects nothing of substance.

However, before I proceed, I'd better back up my criticism. So: consider the existence of single-world retrocausal interpretations of quantum mechanics, such as John Cramer's transactional interpretation, which is descended from Wheeler-Feynman absorber theory. There are no superpositions, only causal chains running forward in time and backward in time. The calculus of complex-valued probability amplitudes is supposed to arise from this.

The existence of the retrocausal tradition already shows that the debate has been represented incorrectly; it should at least be Everett versus Bohr versus Cramer. I would also argue that when you look at the details, many-worlds has no discernible edge over single-world retrocausality:

• Relativity isn't an issue for the transactional interpretation: causality forwards and causality backwards are both local, it's the existence of loops in time which create the appearance of nonlocality.
• Retrocausal interpretations don't have an exact derivation of the Born rule, but neither does many-worlds.
• Many-worlds finds hope of such a derivation in a property of the quantum formalism: the resemblance of density matrix entries to probabilities. But single-world retrocausality finds such hope too: the Born probabilities can be obtained from the product of ψ with ψ*, its complex conjugate, and ψ* is the time reverse of ψ. 
• Loops in time just fundamentally bug some people, but splitting worlds have the same effect on others.

I am not especially an advocate of retrocausal interpretations. They are among the possibilities; they deserve consideration and they get it. Retrocausality may or may not be an element of the real explanation of why quantum mechanics works. Progress towards the discovery of the truth requires exploration on many fronts, that's happening, we'll get there eventually. I have focused on retrocausal interpretations here just because they offer the clearest evidence that the big picture offered by the Sequence is wrong.

It's hopeless to suggest rewriting the Sequence, I don't think that would be a good use of anyone's time. But what I would like to have, is a clear idea of the role that "the winner is ... Many Worlds!" plays in the overall flow of argument, in the great meta-sequence that is Less Wrong's foundational text; and I would also like to have a clear idea of how to patch the argument, so that it routes around this flaw.

In the wiki, it states that "Cleaning up the old confusion about QM is used to introduce basic issues in rationality (such as the technical version of Occam's Razor), epistemology, reductionism, naturalism, and philosophy of science." So there we have it - a synopsis of the function that this Sequence is supposed to perform. Perhaps we need a working group that will identify each of the individual arguments, and come up with a substitute for each one.

Meta-discussion is nasty. Allegedly, troll-feeding was flooding the comments. Verifiably, meta-discussion is flooding the comments. Keep it simple stupid!

A robot is going on a one-shot mission to a distant world to collect important data needed to research a cure for a plague that is devastating the Earth. When the robot enters hyperspace, it notices some anomalies in the engine's output, but it is too late to get the engine fixed. The anomalies are of a sort that, when similar anomalies have been observed in other engines, 25% of the time it indicates a fatal problem, such that the engine will explode virtually every time it tries to jump. 25% of the time, it has been a false positive, and the engine exploded only at its normal negligible rate. 50% of the time it has indicated a serious problem, such that each jump was about a 50/50 chance of exploding. Anyway, the robot goes through the ten jumps to reach the distant world, and the engine does not explode. Unfortunately, the jump coordinates for the mission were a little off, and the robot is in a bad data-collecting position. It could try another jump - if the engine doesn't explode, the extra data it collects could save lives. If the engine does explode, however, Earth will get no data from the distant world at all. (The FTL radio is only good for one use, so he can't collect data and then jump.) So how did you program your robot? Did you program your robot to believe that since the engine worked 10 times, the anomaly was probably a false positive, and so it should make the jump? Or did you program your robot to follow the "Androidic Principle" and disregard the so-called "evidence" of the ten jumps, since it could not have observed any other outcome? People's lives are in the balance here. A little girl is too sick to leave her bed, she doesn't have much time left, you can hear the fluid in her lungs as she asks you "are you aware of the anthropic principle?" Well? Are you?

Very soon, Eliezer is supposed to start posting a new sequence, on "Open Problems in Friendly AI". After several years in which its activities were dominated by the topic of human rationality, this ought to mark the beginning of a new phase for the Singularity Institute, one in which it is visibly working on artificial intelligence once again. If everything comes together, then it will now be a straight line from here to the end.

I foresee that, once the new sequence gets going, it won't be that easy to question the framework in terms of which the problems are posed. So I consider this my last opportunity for some time, to set out an alternative big picture. It's a framework in which all those rigorous mathematical and computational issues still need to be investigated, so a lot of "orthodox" ideas about Friendly AI should carry across. But the context is different, and it makes a difference.

Begin with the really big picture. What would it take to produce a friendly singularity? You need to find the true ontology, find the true morality, and win the intelligence race. For example, if your Friendly AI was to be an expected utility maximizer, it would need to model the world correctly ("true ontology"), value the world correctly ("true morality"), and it would need to outsmart its opponents ("win the intelligence race").

Now let's consider how SI will approach these goals.

The evidence says that the working ontological hypothesis of SI-associated researchers will be timeless many-worlds quantum mechanics, possibly embedded in a "Tegmark Level IV multiverse", with the auxiliary hypothesis that algorithms can "feel like something from inside" and that this is what conscious experience is.

The true morality is to be found by understanding the true decision procedure employed by human beings, and idealizing it according to criteria implicit in that procedure. That is, one would seek to understand conceptually the physical and cognitive causation at work in concrete human choices, both conscious and unconscious, with the expectation that there will be a crisp, complex, and specific answer to the question "why and how do humans make the choices that they do?" Undoubtedly there would be some biological variation, and there would also be significant elements of the "human decision procedure",  as instantiated in any specific individual, which are set by experience and by culture, rather than by genetics. Nonetheless one expects that there is something like a specific algorithm or algorithm-template here, which is part of the standard Homo sapiens cognitive package and biological design; just another anatomical feature, particular to our species.

Having reconstructed this algorithm via scientific analysis of human genome, brain, and behavior, one would then idealize it using its own criteria. This algorithm defines the de-facto value system that human beings employ, but that is not necessarily the value system they would wish to employ; nonetheless, human self-dissatisfaction also arises from the use of this algorithm to judge ourselves. So it contains the seeds of its own improvement. The value system of a Friendly AI is to be obtained from the recursive self-improvement of the natural human decision procedure.

Finally, this is all for naught if seriously unfriendly AI appears first. It isn't good enough just to have the right goals, you must be able to carry them out. In the global race towards artificial general intelligence, SI might hope to "win" either by being the first to achieve AGI, or by having its prescriptions adopted by those who do first achieve AGI. They have some in-house competence regarding models of universal AI like AIXI, and they have many contacts in the world of AGI research, so they're at least engaged with this aspect of the problem.

Upon examining this tentative reconstruction of SI's game-plan, I find I have two major reservations. The big one, and the one most difficult to convey, concerns the ontological assumptions. In second place is what I see as an undue emphasis on the idea of outsourcing the methodological and design problems of FAI research to uploaded researchers and/or a proto-FAI which is simulating or modeling human researchers. This is supposed to be a way to finesse philosophical difficulties like "what is consciousness anyway"; you just simulate some humans until they agree that they have solved the problem. The reasoning goes that if the simulation is good enough, it will be just as good as if ordinary non-simulated humans solved it.

I also used to have a third major criticism, that the big SI focus on rationality outreach was a mistake; but it brought in a lot of new people, and in any case that phase is ending, with the creation of CFAR, a separate organization. So we are down to two basic criticisms.

First, "ontology". I do not think that SI intends to just program its AI with an apriori belief in the Everett multiverse, for two reasons. First, like anyone else, their ventures into AI will surely begin with programs that work within very limited and more down-to-earth ontological domains. Second, at least some of the AI's world-model ought to be obtained rationally. Scientific theories are supposed to be rationally justified, e.g. by their capacity to make successful predictions, and one would prefer that the AI's ontology results from the employment of its epistemology, rather than just being an axiom; not least because we want it to be able to question that ontology, should the evidence begin to count against it.

For this reason, although I have campaigned against many-worlds dogmatism on this site for several years, I'm not especially concerned about the possibility of SI producing an AI that is "dogmatic" in this way. For an AI to independently assess the merits of rival physical theories, the theories would need to be expressed with much more precision than they have been in LW's debates, and the disagreements about which theory is rationally favored would be replaced with objectively resolvable choices among exactly specified models.

The real problem, which is not just SI's problem, but a chronic and worsening problem of intellectual culture in the era of mathematically formalized science, is a dwindling of the ontological options to materialism, platonism, or an unstable combination of the two, and a similar restriction of epistemology to computation.

Any assertion that we need an ontology beyond materialism (or physicalism or naturalism) is liable to be immediately rejected by this audience, so I shall immediately explain what I mean. It's just the usual problem of "qualia". There are qualities which are part of reality - we know this because they are part of experience, and experience is part of reality - but which are not part of our physical description of reality. The problematic "belief in materialism" is actually the belief in the completeness of current materialist ontology, a belief which prevents people from seeing any need to consider radical or exotic solutions to the qualia problem. There is every reason to think that the world-picture arising from a correct solution to that problem will still be one in which you have "things with states" causally interacting with other "things with states", and a sensible materialist shouldn't find that objectionable.

What I mean by platonism, is an ontology which reifies mathematical or computational abstractions, and says that they are the stuff of reality. Thus assertions that reality is a computer program, or a Hilbert space. Once again, the qualia are absent; but in this case, instead of the deficient ontology being based on supposing that there is nothing but particles, it's based on supposing that there is nothing but the intellectual constructs used to model the world.

Although the abstract concept of a computer program (the abstractly conceived state machine which it instantiates) does not contain qualia, people often treat programs as having mind-like qualities, especially by imbuing them with semantics - the states of the program are conceived to be "about" something, just like thoughts are. And thus computation has been the way in which materialism has tried to restore the mind to a place in its ontology. This is the unstable combination of materialism and platonism to which I referred. It's unstable because it's not a real solution, though it can live unexamined for a long time in a person's belief system.

An ontology which genuinely contains qualia will nonetheless still contain "things with states" undergoing state transitions, so there will be state machines, and consequently, computational concepts will still be valid, they will still have a place in the description of reality. But the computational description is an abstraction; the ontological essence of the state plays no part in this description; only its causal role in the network of possible states matters for computation. The attempt to make computation the foundation of an ontology of mind is therefore proceeding in the wrong direction.

But here we run up against the hazards of computational epistemology, which is playing such a central role in artificial intelligence. Computational epistemology is good at identifying the minimal state machine which could have produced the data. But it cannot by itself tell you what those states are "like". It can only say that X was probably caused by a Y that was itself caused by Z.

Among the properties of human consciousness are knowledge that something exists, knowledge that consciousness exists, and a long string of other facts about the nature of what we experience. Even if an AI scientist employing a computational epistemology managed to produce a model of the world which correctly identified the causal relations between consciousness, its knowledge, and the objects of its knowledge, the AI scientist would not know that its X, Y, and Z refer to, say, "knowledge of existence", "experience of existence", and "existence". The same might be said of any successful analysis of qualia, knowledge of qualia, and how they fit into neurophysical causality.

It would be up to human beings - for example, the AI's programmers and handlers - to ensure that entities in the AI's causal model were given appropriate significance. And here we approach the second big problem, the enthusiasm for outsourcing the solution of hard problems of FAI design to the AI and/or to simulated human beings. The latter is a somewhat impractical idea anyway, but here I want to highlight the risk that the AI's designers will have false ontological beliefs about the nature of mind, which are then implemented apriori in the AI. That strikes me as far more likely than implanting a wrong apriori about physics; computational epistemology can discriminate usefully between different mathematical models of physics, because it can judge one state machine model as better than another, and current physical ontology is essentially one of interacting state machines. But as I have argued, not only must the true ontology be deeper than state-machine materialism, there is no way for an AI employing computational epistemology to bootstrap to a deeper ontology.

In a phrase: to use computational epistemology is to commit to state-machine materialism as your apriori ontology. And the problem with state-machine materialism is not that it models the world in terms of causal interactions between things-with-states; the problem is that it can't go any deeper than that, yet apparently we can. Something about the ontological constitution of consciousness makes it possible for us to experience existence, to have the concept of existence, to know that we are experiencing existence, and similarly for the experience of color, time, and all those other aspects of being that fit so uncomfortably into our scientific ontology.

It must be that the true epistemology, for a conscious being, is something more than computational epistemology. And maybe an AI can't bootstrap its way to knowing this expanded epistemology - because an AI doesn't really know or experience anything, only a consciousness, whether natural or artificial, does those things - but maybe a human being can. My own investigations suggest that the tradition of thought which made the most progress in this direction was the philosophical school known as transcendental phenomenology. But transcendental phenomenology is very unfashionable now, precisely because of apriori materialism. People don't see what "categorial intuition" or "adumbrations of givenness" or any of the other weird phenomenological concepts could possibly mean for an evolved Bayesian neural network; and they're right, there is no connection. But the idea that a human being is a state machine running on a distributed neural computation is just a hypothesis, and I would argue that it is a hypothesis in contradiction with so much of the phenomenological data, that we really ought to look for a more sophisticated refinement of the idea. Fortunately, 21st-century physics, if not yet neurobiology, can provide alternative hypotheses in which complexity of state originates from something other than concatenation of parts - for example, entanglement, or from topological structures in a field. In such ideas I believe we see a glimpse of the true ontology of mind, one which from the inside resembles the ontology of transcendental phenomenology; which in its mathematical, formal representation may involve structures like iterated Clifford algebras; and which in its biophysical context would appear to be describing a mass of entangled electrons in that hypothetical sweet spot, somewhere in the brain, where there's a mechanism to protect against decoherence.

Of course this is why I've talked about "monads" in the past, but my objective here is not to promote neo-monadology, that's something I need to take up with neuroscientists and biophysicists and quantum foundations people. What I wish to do here is to argue against the completeness of computational epistemology, and to caution against the rejection of phenomenological data just because it conflicts with state-machine materialism or computational epistemology. This is an argument and a warning that should be meaningful for anyone trying to make sense of their existence in the scientific cosmos, but it has a special significance for this arcane and idealistic enterprise called "friendly AI". My message for friendly AI researchers is not that computational epistemology is invalid, or that it's wrong to think about the mind as a state machine, just that all that isn't the full story. A monadic mind would be a state machine, but ontologically it would be different from the same state machine running on a network of a billion monads. You need to do the impossible one more time, and make your plans bearing in mind that the true ontology is something more than your current intellectual tools allow you to represent.

Eliezer's post How To Convince Me That 2 + 2 = 3 has an interesting consideration - if putting two sheep in a field, and putting two more sheep in a field, resulted in three sheep being in the field, would arithmetic hold that two plus two equals three?

I want to introduce another question.  What exactly are you counting?

Imagine one sheep in one field, and another sheep in another.  Now put them together.  Do you now have two sheep?

"Of course!"

Ah, but is that -all- you have?

"What?"

Two sheep are more than twice as complex as a single sheep.  It takes more than twice as many bits to describe two sheep than it takes to describe a single sheep, because, in addition to those two sheep, you now also have to describe their relationship to one another.

Or, to phrase it slightly differently, does 1+1=2?

Well, the answer is, it depends on what you're counting.

If you're counting the number of discrete sheep, 1+1=2.  However, why is the number of discrete sheep meaningful?

If you're a hunter counting, not herded sheep, but prey - two sheep is, roughly, twice as much meat as one sheep.  1+1=2.  If you're a herder, however, two sheep could be a lot more valuable than one - two sheep can turn into three sheep, if one is female and one is male.  The value of two sheep can be more than twice the value of a single sheep.  And if you're a hypercomputer running Solomonoff Induction to try to describe sheep positional vectors, two sheep will have a different complexity than twice the complexity of a single sheep.

Which is not to say that one plus one does not equal two.  It is, however, to say that one plus one may not be meaningful as a concept outside a very limited domain.

Would an alien intelligence have arrived at arithmetic?  Depends on what it counts.  Is arithmetic correct?

Well, does a set of two sheep contain only two sheep, or does it also contain their interactions?  Depends on your problem domain; 1+1 might just equal 2+i.

Celia Green is a figure who should interest some LW readers. If you can imagine Eliezer, not as an A.I. futurist in 2000s America, but as a parapsychologist in 1960s Britain - she must have been a little like that. She founded her own research institute in her mid-20s, invented psychological theories meant to explain why the human race was walking around resigned to mortality and ignorance, felt that her peers (who got all the research money) were doing everything wrong... I would say that her two outstanding books are The Human Evasion and Advice to Clever Children. The first book, while still very obscure, has slowly acquired a fanbase online; but the second book remains thoroughly unknown.

For a synopsis of what the books are about, I think something I wrote in 1993 (I've been promoting her work on the Internet for years) remains reasonable. They contain an analysis of the alleged deficiencies and hidden motivations of normal human psychology, description of an alternative outlook, and an examination of various topics from that new perspective. There is some similarity to the rationalist ideal developed in the Sequences here, in that her alternative involves existential urgency, deep respect for uncertainty, and superhuman aspiration.

There are also prominent differences. Green's starting point is not Bayesian calculation, it's Humean skepticism. Green would agree that one should aspire to "think like reality", but for her this would mean, above all, being mindful of "total uncertainty". It's a fact that I don't know what comes next, that I don't know the true nature of reality, that I don't know what's possible if I try; I may have habitual opinions about these matters, but a moment's honest reflection shows that none of these opinions are knowledge in any genuine sense; even if they are correct, I don't know them to be correct. So if I am interested in thinking like reality, I can begin by acknowledging the radical uncertainty of my situation. I exist, I don't know why, I don't know what I am, I don't know what the world is or what it has planned for me. I may have my ideas, but I should be able to see them as ideas and hold them apart from the unknown reality.

If you are like me, you will enjoy the outlook of open-ended striving that Green develops in this intellectual context, but you will be jarred by her account of ordinary, non-striving psychology. Her answer to the question, why does the human race have such petty interests and limited ambitions, is that it is sunk in an orgy of mutual hatred, mostly disguised, and resulting from an attempt to evade the psychology of striving. More precisely, to be a finite human being is to be in a desperate and frustrating situation; and people attempt to solve this problem, not by overcoming their limitations, but by suppressing their reactions to the situation. Other people are central to the resulting psychological maneuvers. They are a way for you to distract yourself from your own situation, and they are a safe target if the existential frustration and desperation reassert themselves.

Celia Green's psychological ideas are the product of her personal confrontation with the mysterious existential situation, and also her confrontation with an uncomprehending society. I've thought for some time that her portrayal of universal human depravity results from overestimating the potential of the average human being; that in effect she has asked herself, if I were that person, how could I possibly lead the life I see them living, and say the things I hear them saying, unless I were that twisted up inside? Nonetheless, I do think she has described an aspect of human psychology which is real and largely unexamined, and also that her advice on how to avoid the resentful turning-away from reality, and live in the uncertainty, is quite profound. One reason I'm promoting these books is in the hope that some small part of the culture at large is finally ready to digest their contents and critically assess them. People ought to be doing PhDs on the thought of Celia Green, but she's unknown in that world.

As for Celia Green herself, she's still alive and still going. She has a blog and a personal website and an organization based near Oxford. She's an "academic exile", but true to her philosophy, she hasn't compromised one iota and hopes to start her own private university. She may especially be of interest to the metaphysically inclined faction of LW readers, identified by Yvain in a recent blog post.