This is incidental to the topic, but what do you mean by “controlled shutdown”, as distinct from “shutdown”?
My guess: the now-malfunctioning AGI is in charge of critical infrastructure upon which the lives of O(3^^^3) humans depend at the time it detects that it is about to self-deceive. Presumably a "controlled shutdown" would be some kind of orderly relinquishing of its responsibilities so that as few of those 0(3^^^3) humans are harmed in the process.
Of course, that assumes that such a shutdown is actually possible at that time. What guarantees could be provided to ensure that a non-future-destroying controlled shutdown of an AGI would be feasible at any point in time?
>How exactly can you construct a disproof of X without using
>premises that rule out X? That's what disproving is.
Sure, a mathematical proof proceeds from its premises and therefore any results achieved are entailed in those premises. I am not sure we are really in the real of pure mathematics here but I probably should have been more precise in my statement. In a non-mathematical discussion, a slightly longer chain of reasoning is generally preferred -- starting with the premise that dualism is false is a little uncomfortably close to starting with ...
Caledonian:
In an argument that is basically attempting to disprove the existence of God, it seems a little disingenuous to me to include premises that effectively rule out God's existence. If you aren't willing to at least allow the possibility of dualism for the sake of argument, then why bother talking about God at all?
Also, I am not sure what your notion of "infinite" mathematics is about. Can you elaborate or point me to some relevant resources?
"Chad: if you seriously think that Turing-completeness does not imply the possibility of sentience, then you're definitely in the wrong place indeed."
gwern: The implication is certainly there and it's one I am sympathetic with, but I'd say its far from proven. The leap in logic there is one that will keep the members of the choir nodding along but is not going to win over any converts. A weak argument is a weak argument, whether you agree with the conclusion reached by that argument -- it's better for the cause if the arguments are held to higher standards.
"Conway's Life has been proven Turing-complete, so it would be possible to build a sentient being in the Life universe"
Bit of a leap in logic here, no?
The leap is that the Church–Turing thesis applies to human (“sentient”) cognition. Many theists deny this.
Worst case, our laws of physics seem to be turing-computable.
Isn't there quite a bit of selection bias involved here? Perhaps some level of native intelligence is required to be in the elite (or strongly correlated to it at least) but don't you have to see how many equally intelligent people are toiling away outside the elite to determine if greater intelligence is sufficient? Wouldn't Bayes be a little disappointed with thispost? Also, you observe that people who are in the elite seem happier and more fulfilled than the average. That doesn't really seem that surprising to me -- after all they are already pretty far...
Catperson: I agree with your reading.
Additionally, it seems Akon is unaware of the historic asymmetry of gender roles in non-consensual sex.
Either that, or in this impossible possible world there was a reversal from our present gender dynamics at some point in the past.