Took the survey, except for the digit ratio part.

You have mentioned the weakened reflection principle as being the following: ∀φ∈L'. ∀a,b∈Q. a≤P(φ)≤b ⇒ P(a<P('φ')<b)=1

This seems to be a typo, it should be ∀φ∈L'. ∀a,b∈Q. a<P(φ)<b ⇒ P(a<P('φ')<b)=1

On the other hand, there must be some downside to pain asymbolia, or we'd all have it. (Plainly the mutation exists; why isn't it selected for?)

Took the survey!

"Buddhism IS different. It's the followers who aren’t."

-- A Dust Over India.

Commentary: Reading this made me realize that many religions genuinely are different from each other. Christianity is genuinely different from Judaism, Islam is genuinely different from Christianity, Hinduism is genuinely different from all three. It's religious people who are the same everywhere; not the same as each other, obviously, but drawn from the same distribution. Is this true of atheistic humanists? Of transhumanists? Could you devise an experiment to test whether it was so, would you bet on the results of that experiment? Will they say the same of LessWrongers, someday? And if so, what's the point?

Now that I think on it, though, there might be a case for scientists being drawn from a different distribution, or computer programmers, or for that matter science fiction fans (are those all the same distributions as each other, I wonder?). It's not really hopeless.

But you're being vague otherwise. Name a crazy or unfounded belief.

Holden asked me something similar today via mail. Here is what I replied:

You wrote in 'Other objections to SI's views':

Unlike the three objections I focus on, these other issues have been discussed a fair amount, and if these other issues were the only objections to SI's arguments I would find SI's case to be strong (i.e., I would find its scenario likely enough to warrant investment in).

It is not strong. The basic idea is that if you pull a mind at random from design space then it will be unfriendly. I am not even sure if that is true. But it is the strongest argument they have. And it is completely bogus because humans do not pull AGI's from mind design space at random.

Further, the whole case for AI risk is based on the idea that there will be a huge jump in capability at some point. Which I think is at best good science fiction, like faster-than-light propulsion, or antimatter weapons (when in doubt that it is possible in principle).

The basic fact that an AGI will most likely need something like advanced nanotechnology to pose a risk, which is itself an existential risk, hints at a conjunction fallacy. We do not need AGI to then use nanotechnology to wipe us out, nanotechnology is already enough if it is possible at all.

Anyway, it feels completely ridiculous to talk about it in the first place. There will never be a mind that can quickly and vastly improve itself and then invent all kinds of technological magic to wipe us out. Even most science fiction books avoid that because it sounds too implausible.

I have written thousands of words about all this and never got any convincing reply. So if you have any specific arguments, let me know.

They say what what I write is unconvincing. But given the amount of vagueness they use to protect their beliefs, my specific criticisms basically amount to a reductio ad absurdum. I don't even need to criticize them, they would have to support their extraordinary beliefs first or make them more specific. Yet I am able to come up with a lot of arguments that speak against the possibility they envision, without any effort and no knowledge of the relevant fields like complexity theory.

Here is a comment I received lately:

…in defining an AGI we are actually looking for a general optimization/compression/learning algorithm which when fed itself as an input, outputs a new algorithm that is better by some multiple. Surely this is at least an NP-Complete if not more problem. It may improve for a little bit and then hit a wall where the search space becomes intractable. It may use heuristics and approximations and what not but each improvement will be very hard won and expensive in terms of energy and matter. But no matter how much it tried, the cold hard reality is that you cannot compute an EXPonential Time algorithm in polynomial time unless (P=EXPTIME :S). A no self-recursive exponential intelligence theorem would fit in with all the other limitations (speed, information density, Turing, Gödel, uncertainties etc) the universe imposes.

If you were to turn IBM Watson gradually into a seed AI, at which point would it become an existential risk and why? They can't answer that at all. It is pure fantasy.

END OF EMAIL

For more see the following posts:

• Is an Intelligence Explosion a Disjunctive or Conjunctive Event?
• Risks from AI and Charitable Giving
• Why I am skeptical of risks from AI
• Implicit constraints of practical goals (including the follow-up comments that I posted.)

Some old posts:

• Should I believe what the SIAI claims?
• What I would like the SIAI to publish
• SIAI’s Short-Term Research Program

See also:

• Interview series on risks from AI
• We are SIAI. Argument is futile.

If you believe I don't understand the basics, see:

• A Primer On Risks From AI

Also:

• Open Problems in Ethics and Rationality
• Objections to Coherent Extrapolated Volition

There is a lot more, especially in the form of comments where I talk about specifics.