Will_Newsome comments on Journal of Consciousness Studies issue on the Singularity - Less Wrong

14 Post author: lukeprog 02 March 2012 03:56PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (85)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 02 March 2012 06:14:59PM *  23 points [-]

Tipler paper

Wow, that's all kinds of crazy. I'm not sure how much as I'm not a mathematical physicist - MWI and quantum mechanics implied by Newton? Really? - but one big flag for me is pg187-188 where he doggedly insists that the universe is closed, although as far as I know the current cosmological consensus is the opposite, and I trust them a heck of a lot more than a fellow who tries to prove his Christianity with his physics.

(This is actually convenient for me: a few weeks ago I was wondering on IRC what the current status of Tipler's theories were, given that he had clearly stated they were valid only if the universe were closed and if the Higgs boson was within certain values, IIRC, but I was feeling too lazy to look it all up.)

And the extraction of a transcendent system of ethics from a Feynman quote...

A moment’s thought will convince the reader that Feynman has described not only the process of science, but the process of rationality itself. Notice that the bold-faced words are all moral imperatives. Science, in other words, is fundamentally based on ethics. More generally, rational thought itself is based on ethics. It is based on a particular ethical system. A true human level intelligence program will thus of necessity have to incorporate this particular ethical system. Our human brains do, whether we like to acknowledge it or not, and whether we want to make use of this ethical system in all circumstances. When we do not make use of this system of ethics, we generate cargo cult science rather than science.

This is just too wrong for words. This is like saying that looking both ways before crossing the street is obviously a part of rational street-crossing - a moment's thought will convince the reader (Dark Arts) - and so we can collapse Hume's fork and promote looking both ways to a universal meta-ethical principal that future AIs will obey!

An AI program must incorporate this morality, otherwise it would not be an AI at all.

Show me this morality in the AIXI equation or GTFO!

After all, what is a computer program but a series of imperative sentences?

A map from range to domain, a proof in propositional logic, or a series of lambda equations and reductions all come to mind...

In fact, I claim that an ethical system that encompasses all human actions, and more generally, all actions of any set of rational beings (in particular, artificial intelligences) can be deduced from the Feynman axioms. In particular, note that destroying other rational beings would make impossible the honestly Feynman requires.

One man's modus ponens is another man's modus tollens. That the 'honestly' requires other entities is proof that this cannot be an ethical system which encompasses all rational beings.

Hence, they will be part of the community of intelligent beings deciding whether to resurrect us or not. Do not children try to see to their parents’ health and well-being? Do they not try and see their parent survive (if it doesn’t cost too much, and it the far future, it won’t)? They do, and they will, both in the future, and in the far future.

Any argument that rests on a series of rhetorical questions is untrustworthy. Specifically, sure, I can in 5 seconds come up with a reason they would not preserve us: there are X mind-states we can be in while still maintaining identity or continuity; there are Y (Y < X) that we would like or would value; with infinite computing power, we will exhaust all Y. At that point, by definition, we could choose to not be preserved. Hence, I have proven we will inevitably choose to die even if uploaded to Tipler's Singularity.

(Correct and true? Dunno. But let's say this shows Tipler is massively overreaching...)

What a terrible paper altogether. This was a peer-reviewed journal, right?

Comment author: Will_Newsome 03 March 2012 07:32:40AM *  1 point [-]

An AI program must incorporate this morality, otherwise it would not be an AI at all.

Show me this morality in the AIXI equation or GTFO!

The "AIXI equation" is not an AI in the relevant sense.

Comment author: gwern 03 March 2012 07:49:36AM 0 points [-]

Fine, 'show me this morality in a computable implementation of AIXI using the speed prior or GTFO' (what was it called, AIXI-tl?).

Comment author: Will_Newsome 03 March 2012 08:04:55AM 1 point [-]

That also isn't an AI in the relevant sense, as it doesn't actually exist. Tipler would simply deny that such an AI would be able to anything for Searlian reasons. You can't prove that an AIXI-style AI will ever work, and it's presumably part of Tipler's argument that it won't work, so simply asserting that it will work is sort of pointless. I'm just saying that if you want to engage with his argument you'll have to get closer to it 'cuz you're not yet in bowshot range. If your intention was to repeat the standard counterargument rather than show why it's correct then I misinterpreted your intention; apologies if so.

Comment author: gwern 03 March 2012 08:25:29AM 1 point [-]

Tipler would simply deny that such an AI would be able to anything for Searlian reasons. You can't prove that an AIXI-style AI will ever work, and it's presumably part of Tipler's argument that it won't work, so simply asserting that it will work is sort of pointless.

The AIXI proofs seem pretty adequate to me. They may not be useful, but that's different from not working.

More to the point, nothing in Tipler's paper gave me the impression he had so much as heard of AIXI, and it's not clear to me that he does accept Searlian reasons - what is that, by the way? It can't be Chinese room stuff since Tipler has been gung ho on uploading for decades now.

Comment author: Will_Newsome 03 March 2012 08:34:41AM 3 points [-]

The AIXI proofs seem pretty adequate to me. They may not be useful, but that's different from not working.

It's really not obvious that if you run an AIXI-like AI it will actually do anything other than self-destruct, no matter how much juice you give it. There have been various papers on this theme recently and it's a common LW meme ("AIXI drops an anvil on its head").

By "Searlian reasons" I mean something like emphasizing the difference between syntax and semantics and the difficulty of the grounding problem as representative of this important dichotomy between narrow and general intelligence that philosophers of mind get angry with non-philosophers of mind for ignoring.

I don't think Tipler's not having heard of AIXI is particularly damning, even if true.

Comment author: gwern 08 March 2012 03:12:13AM 1 point [-]

It's really not obvious that if you run an AIXI-like AI it will actually do anything other than self-destruct, no matter how much juice you give it. There have been various papers on this theme recently and it's a common LW meme ("AIXI drops an anvil on its head").

I don't think it's obvious it would self-destruct - any more than it's obvious humans will not self-destruct. (And that anvil phrase is common to Eliezer.) The papers you allude to apply just as well to humans.

I don't think Tipler's not having heard of AIXI is particularly damning, even if true.

I believe you are the one who is claiming AIXI will never work, and suggesting Tipler might think like you.