Eliezer_Yudkowsky comments on So You Want to Save the World - Less Wrong

41 Post author: lukeprog 01 January 2012 07:39AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (146)

You are viewing a single comment's thread. Show more comments above.

Comment author: lukeprog 27 December 2011 09:44:04PM *  4 points [-]

Stuart Armstrong's explanation of the 5-and-10 problem is:

The five-and-ten problem (sometimes known as the heavy ghost problem) is a problem in certain types of [updateless decision theory]-like decision theories, when the fact that a counterfactual is known to be false makes the algorithm implement it.

Specifically, let there be a decision problem which involves the choice between $5 and $10, a utility function that values the $10 more than the $5, and an algorithm A that reasons something like:

"Look at all proposition of the type '(A decides to do X) implies (Utility=y)', and find the X that maximises y, then do X."

When faced with the above problem, certain types of algorithm can reason:

"The utility of $10 is greater than the utility of $5. Therefore I will never decide to choose $5. Therefor (A decides to do 'choose $5') is a false statement. Since a false statement implies anything, (A decides to do 'choose $5') implies (Utility=y) for any, arbitrarily high, value of y. Therefore this is the utility maximising decision, and I should choose $5."

That is the informal, natural language statement of the problem. Whether the algorithm is actually vulnerable to the 5-and-10 problem depends on the details of what the algorithm is allowed to deduce about itself.

However, some think Drescher's explanation is more accurate. Somebody should write a short paper on the problem so I can cite that instead. :)

Comment author: [deleted] 28 December 2011 03:36:55AM 7 points [-]

Clearly, if the algorithm concludes that it will certainly not choose the $5, and then does choose the $5, it concluded wrong. But the reasoning seems impeccable, and there don't seem to be any false premises here. It smacks of the unexpected hanging paradox.

Ooh, but wait. Expanding that reasoning a bit, we have...

The utility of $10 is greater than the utility of $5. Therefore, an algorithm whose axioms are consistent will never decide to choose $5. I am an algorithm whose axioms are consistent. Therefore, I will never decide to choose $5.

The assumption "I am an algorithm whose axioms are consistent" is one that we already know leads to a contradiction, by Löb's theorem. If we can avoid the wrath of Löb's theorem, can we also avoid the five-and-ten problem?

(Granted, this probably isn't the best place to say this.)

Comment author: Eliezer_Yudkowsky 28 December 2011 04:55:45PM 7 points [-]

If we can avoid the wrath of Löb's theorem, can we also avoid the five-and-ten problem?

Very likely yes. Now ask if I know how to avoid the wrath of Löb's theorem.

Comment author: wedrifid 28 December 2011 05:02:38PM 7 points [-]

Very likely yes. Now ask if I know how to avoid the wrath of Löb's theorem.

Do you know how to avoid the wrath of Lob's theorem?

Comment author: Eliezer_Yudkowsky 28 December 2011 05:04:36PM 4 points [-]

Not yet.

Comment author: Manfred 28 December 2011 07:21:45PM 3 points [-]

What kind of powers are you hoping for beyond this sort of thing?

Comment author: Anubhav 17 January 2012 08:29:19AM -1 points [-]

Now ask if I know how to avoid the wrath of Löb's theorem

For someone making a desperate effort to not be a cult leader, you really do enjoy arbitrarily ordering people around, don't you?

</humour possibly subject to Poe's law>