Two objections to this: Firstly you have to extrapolate from the chimp-to-human range and into superintelligence range. The gradient may not be the same in the two. Second, it seems to me that the more intelligent humans are, the more "the other humans in my tribe" becomes the dominant part of your environment; this leads to increased returns to intelligence, and consequently you do get an increasing optimisation pressure.

Hum. Suppose that increasing the intelligence of an AI requires a series of insights, or searches through design space, or however you want to phrase it. The FOOM then seems to assume that each insight is of roughly equal difficulty, or at least that the difficulty does not increase as rapidly as does the intelligence. But it does not seem obvious that the jump from Arbitrary Intelligence Level 2 to 3 requires an insight of equal difficulty as the jump from 3 to 4. In fact, intuitively it seems that jumps from N to N+1 are easier than jumps from N+1 to N+2. (It is not immediately obvious to me what the human intelligence distribution implies about this. We don't even know, strictly speaking, that it's a bell curve, although it does seem to have a fat middle.) If, to take a hypothetical example, each jump doubles in difficulty but gives a linear increase in intelligence, then the process won't FOOM at all - it'll go asymptotic horizontally, albeit perhaps at a level much above a genius human's. Even if the difficulty increases only linearly while granting a linear increase in intelligence, that keeps the time required for each jump constant. That doesn't rule out arbitrarily intelligent AIs, but it does mean the increase doesn't show an asymptote. (Depending on the time constant, it could even be uninteresting. If it takes the AI ten years to generate the insights to increase its IQ by one point, and it starts at 100, then we'll be waiting a while.)

Now, neither of those possibilities is especially likely. But if we take the increase in difficulty per level as x, and the increase in intelligence per level as y, and the time to the next insight as proportional to (x/y), then what reason do we have to believe that x < y? (Or, if they're roughly equal, that the constant of proportionality is small.)

Took the survey.

Did anyone else find the IQ questions to be either boringly easy or impossibly hard? I don't think I answered anything that required more than 15s of thought.

Keeping it selected and ctrl-c/cmd-c ing it works.

I took the survey! Karma, please!

Never done an IQ test before. I thought it was fun! Now I want to take one of the legitimate ones.

I did not intend to explain how i arrived at this conclusion. I'm just stating my answer to the question.

Do you think the statement "If X is just, then do X" is wrong?

The statement that the world is just is a lie. There exist possible worlds that are just - for instance, these worlds would not have children kidnapped and forced to kill - and ours is not one of them.

Thus, justice is a meaningful concept. Justice is a concept defined in terms of the world (pinned-down causal links) and also irreducibly normative statements. Normative statements do not refer to "the world". They are useful because we can logically deduce imperatives from them. "If X is just, then do X." is correct, that is:

Do the right thing.

I'll do 10.

What is the error-checking process? Will we fix any mistakes in our verdicts via an LW discussion after they have been gathered?

I recently read the wiki article on criticality accidents, and it seems relevant here. "A criticality accident, sometimes referred to as an excursion or a power excursion, is the unintentional assembly of a critical mass of a given fissile material, such as enriched uranium or plutonium, in an unprotected environment."

Assuming Eliezer's analysis is correct, we cannot afford even 1 of these in the domain of self-improving AI. Thankfully, its harder to accidentally create a self-improving AI than it is to drop a brick in the wrong place at the wrong time.