No, because not all norms are moral norms.

Intuition is perception.

That's actually a pretty contentious claim.

As far as I know, yes it would.

Citation is still needed :-) Do you have a link to that study?

Have you ever noticed that we have lots of evidence that slimmer people tend to be healthier, but not that losing weight makes you healthier?

Let me see if I read your position correctly. We know that slim people are healthier than fat people, right? We know that getting fat worsens your health -- I believe this is fairly uncontroversial -- do you wish to contest that? But you are saying that this is a ratchet, losing fat will not make your health any better. In other words, once you gained weight there is no path back to health ever?

That seems a rather strong statement to me and I haven't seem much support for that being true in reality. Why did you pick this position as your prior?

We perhaps shouldn't give much weight to PZ Myers' viewpoint. See this Slatestarcodex article.

Yes, this does reduce the benefit of getting Trump to support mosquito eradication.

If you believe in G-d then you believe in a being that can change reality just by willing it

OK, so by that definition...if you instead believe in a perfect rationalist that has achieved immortality, lived longer than we can meaningfully express, and now operates technology that is sufficiently advanced to be indistinguishable from magic, including being involved in the formation of planets, then - what label should you use instead of 'G-d'?

Yes, I noticed he overlooked the distinction between "I know I am conscious because it's my direct experience" and "I know I am conscious because I say 'I know I am conscious because it's my direct experience.'" And those are two entirely different things.

"I don't trust my ability to set limits on the abilities of Bayesian superintelligences."

Limits? I can think up few on the spot already.

Environment: CPU power, RAM capacity etc. I don't think even you guys claim something as blatant as "AI can break laws of physics when convenient".

Feats:

• Win this kind of situation in chess. Sure, AI would not allow occurence of that situation in first place during game, but that's not my point.

• Make human understand AI. Note: uplifting does not count, since human then ceases to be human. As a practice, try teaching your cat Kant's philosophy.

• Make AI understand itself fully and correctly. This one actually works on all levels. Can YOU understand yourself? Are you even theoretically capable of that? Hint: no.

• Related: survive actual self-modification, especially without any external help. Transhumanist fantasy says AIs will do it all the time. Reality is that any self-preserving AI will be as eager to preform self-modification as you to get randomized extreme form of lobotomy (transhumanist version of Russian roulette, except with all bullets in every gun except one in gazilion).

I guess some people are so used to think about AI as magic omnipotent technogods they don't even notice it. Sad.

You only fight over things that are valuable.

Sentience is exactly just the ability to feel. If it can feel joy, it is sentient.

I would play lotto: if I win more than 10M$, I take the black box and leave. Otherwise I'd look in the black box: if it is full, I also take the small one. If not, I leave with just the empty black box. As this should be inconsistent, assuming a time traveling Omega, it would either make him not choose me for his experiment or let me win for sure (assuming time works in similar ways as in HPMOR). If I get nothing, it would prove the Omega wrong (and tell me quite a bit about how the Omega (and time) works). If his prediction was correct though, I win 11.000.000$, which is way better than either 'standard' variant.