This is incidental to the topic, but what do you mean by “controlled shutdown”, as distinct from “shutdown”?
My guess: the now-malfunctioning AGI is in charge of critical infrastructure upon which the lives of O(3^^^3) humans depend at the time it detects that it is about to self-deceive. Presumably a "controlled shutdown" would be some kind of orderly relinquishing of its responsibilities so that as few of those 0(3^^^3) humans are harmed in the process.
Of course, that assumes that such a shutdown is actually possible at that time. What guarantees could be provided to ensure that a non-future-destroying controlled shutdown of an AGI would be feasible at any point in time?
>How exactly can you construct a disproof of X without using
>premises that rule out X? That's what disproving is.
Sure, a mathematical proof proceeds from its premises and therefore any results achieved are entailed in those premises. I am not sure we are really in the real of pure mathematics here but I probably should have been more precise in my statement. In a non-mathematical discussion, a slightly longer chain of reasoning is generally preferred -- starting with the premise that dualism is false is a little uncomfortably close to starting with the premise that God doesn't exist for my taste.
>If you can give us an example of a physical phenomenon that
>cannot be generated by a Turing Machine's output -- just
>one example -- then I will grant that we have no grounds
>for presuming human cognition can be so constructed.
For the record (not that I have any particular standing worthy of note): I am not a dualist and I believe 100% that human cognition is a physical phenomenon that could be captured by a sufficiently complex Turing Machine. Can I prove this to be the case? No I can't and I don't really care to try -- and it's likely "above my level". The only reason I piped up at all is because I think strawman arguments are unconvincing and do a disservice to everyone.
>Also, you'll win several Nobel Prizes and go down in history
>as one of the greatest scientist-thinkers ever.
>I'm not holding my breath.
Don't worry -- I have no such aspiration, so you can comfortably continue with your respiration.
Caledonian:
In an argument that is basically attempting to disprove the existence of God, it seems a little disingenuous to me to include premises that effectively rule out God's existence. If you aren't willing to at least allow the possibility of dualism for the sake of argument, then why bother talking about God at all?
Also, I am not sure what your notion of "infinite" mathematics is about. Can you elaborate or point me to some relevant resources?
"Chad: if you seriously think that Turing-completeness does not imply the possibility of sentience, then you're definitely in the wrong place indeed."
gwern: The implication is certainly there and it's one I am sympathetic with, but I'd say its far from proven. The leap in logic there is one that will keep the members of the choir nodding along but is not going to win over any converts. A weak argument is a weak argument, whether you agree with the conclusion reached by that argument -- it's better for the cause if the arguments are held to higher standards.
"Conway's Life has been proven Turing-complete, so it would be possible to build a sentient being in the Life universe"
Bit of a leap in logic here, no?
Isn't there quite a bit of selection bias involved here? Perhaps some level of native intelligence is required to be in the elite (or strongly correlated to it at least) but don't you have to see how many equally intelligent people are toiling away outside the elite to determine if greater intelligence is sufficient? Wouldn't Bayes be a little disappointed with thispost? Also, you observe that people who are in the elite seem happier and more fulfilled than the average. That doesn't really seem that surprising to me -- after all they are already pretty far up Maslow's pyramid by the time you can identify them as elite (although I have met a few billionaires and many millionaires who don't seem particularly happy or fulfilled with their lives).
WRT VCs: amongst VCs, Jurvetson is clearly outside the norms, given how quickly he became a partner at a major firm. Better study more typical VCs to draw your conclusion about the class.
Catperson: I agree with your reading.
Additionally, it seems Akon is unaware of the historic asymmetry of gender roles in non-consensual sex.
Either that, or in this impossible possible world there was a reversal from our present gender dynamics at some point in the past.