Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Nebu comments on The Number Choosing Game: Against the existence of perfect theoretical rationality - Less Wrong

-1 Post author: casebash 29 January 2016 01:04AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (151)

You are viewing a single comment's thread. Show more comments above.

Comment author: Nebu 05 January 2016 07:56:46AM 1 point [-]

Alternately, letting "utility" back in, in a universe of finite time, matter, and energy, there does exist a maximum finite utility which is the sum total of the time, matter, and energy in the universe.

Why can't my utility function be:

  • 0 if I don't get ice cream
  • 1 if I get vanilla ice cream
  • infinity if I get chocolate ice cream

?

I.e. why should we forbid a utility function that returns infinity for certain scenarios, except insofar that it may lead to the types of problems that the OP is worrying about?

Comment author: Usul 05 January 2016 08:07:09AM 0 points [-]

I was bringing the example into the presumed finite universe in which we live, where Maximum Utility = The Entire Universe. If we are discussing a finite-quantity problem than infinite quantity is ipso facto ruled out.

Comment author: casebash 05 January 2016 11:52:44PM 1 point [-]

I think Nebu was making the point that while we normally use utility to talk about a kind of abstract gain, computers can be programmed with an arbitrary utility function. We would generally put certain restraints on it so that the computer/robot would behave consistently, but those are the only limitation. So even if there does not exist such a thing as infinite utility, a rational agent may still be required to solve for these scenarios.

Comment author: Nebu 24 January 2016 10:44:33PM *  0 points [-]

I guess I'm asking "Why would a finite-universe necessarily dictate a finite utility score?"

In other words, why can't my utility function be:

  • 0 if you give me the entire universe minus all the ice cream.
  • 1 if you give me the entire universe minus all the chocolate ice cream.
  • infinity if I get chocolate ice cream, regardless of how much chocolate ice cream I receive, and regardless of whether the rest of the universe is included with it.