Comment author:jimrandomh
25 February 2011 10:12:36PM
5 points
[-]

But why do beliefs need to pay rent in anticipated experiences? Why canâ€™t they pay rent in utility?

They can. They just do so very rarely, and since accepting some inaccurate beliefs makes it harder to determine which beliefs are and aren't beneficial, in practice we get the highest utility from favoring accuracy. It's very hard to keep the negative effects of a false belief contained; they tend to have subtle downsides. In the example you gave, Joe's belief that he's already smart and beautiful might be stopping him from pursuing self-improvements. But there definitely are cases where accurate beliefs are definitely detrimental; Nick Bostrom's Information Hazards has a partial taxonomy of them.

Comment author:HonoreDB
26 February 2011 01:47:39AM
0 points
[-]

I don't think it's possible for a reflectively consistent decision-maker to gain utility from self-deception, at least if you're using an updateless decision theory. Hiding an unpleasant fact F from yourself is equivalent to deciding never to know whether F is true or false, which means fixing your belief in F at your prior probability for it. But a consistent decision-maker who loses 10 utilons from believing F with probability ~1 must lose p*10 utilons for believing F with probability p.

Comment author:jimrandomh
26 February 2011 03:04:19AM
*
2 points
[-]

A consistent decision-maker who loses 10 utilons from believing F with probability ~1 must lose p*10 utilons for believing F with probability p.

No, this is not true. Many of the reasons why true beliefs can be bad for you are because information about your beliefs can leak out to other agents in ways other than through your actions, and there is is no particular reason for this effect to be linear. For example, blocking communications from a potential blackmailer is good because knowing with probability 1.0 that you're being blackmailed is more than 5 times worse than knowing with probability 0.2 that you will be blackmailed in the future if you don't.

Comment author:jimrandomh
26 February 2011 05:20:27PM
0 points
[-]

I don't think it's linear in the average Joe story, either; if there's one threshold level of belief which changes his behavior, then utility is constant for levels of belief on either side of that threshold and discontinuous in between.

Comment author:HonoreDB
26 February 2011 05:47:07PM
1 point
[-]

A rational agent can have its behavior depend on a threshold crossing of belief, but if there's some belief that grants it utility in itself (e.g. Joe likes to believe he is attractive), the utility it gains from that belief has to be linear with the level of belief. Otherwise, Joe can get dutch-booked by a Monte Carlo plastic surgeon.

Comment author:jimrandomh
26 February 2011 05:58:54PM
0 points
[-]

Otherwise, Joe can get dutch-booked by a Monte Carlo plastic surgeon.

This doesn't sound right. Could you describe the Dutch-booking procedure explicitly? Assume that believing P with probability p gives me utility U(p)=p^2+C.

Comment author:HonoreDB
26 February 2011 07:33:13PM
*
0 points
[-]

An additive constant seems meaningless here: if Joe gets C utilons no matter what p is, then those utilons are unrelated to p or to P--Joe's behavior should be identical if U(p)=p^2, so for simplicity I'll ignore the C.

Now, suppose Joe currently believes he is not attractive. A surgery has a .5 chance of making him attractive and a .5 chance of doing nothing. This surgery is worth U(.5)-U(0)=.25 utilons to Joe; he'll pay up to that amount for it.

Suppose instead the surgeon promises to try again, once, if the first surgery fails. Then Joe's overall chance of becoming attractive is .75, so he'll pay U(.75)-U(0)=.75^2=0.5625 for the deal.

Suppose Joe has taken the first deal, and the surgeon offers to upgrade it to the second. Joe is willing to pay up to the difference in prices for the upgrade, so he'll pay .5625-.25=.3125 for the upgrade.

Joe buys the upgrade. The surgeon performs the first surgery. Joe wakes up and learns that the surgery failed. Joe is entitled to a second surgery, thanks to that .3125-utility purchase of the upgrade. But the second surgery is now worth only .25 utility to him! The surgeon offers to buy that second surgery back from him at a cost of .26 utility. Joe accepts. Joe has spent a net of .0525 utility on an upgrade that gave him no benefit.

As a sanity check, let's look at how it would go if Joe's U(p)=p. The single surgery is worth .5. The double surgery is worth .75. Joe will pay up to .25 utility for the upgrade. After the first surgery fails, the upgrade is worth .5 utility. Joe does not regret his purchase.

Comment author:jimrandomh
26 February 2011 08:25:59PM
*
2 points
[-]

You're missing the fact that how much Joe values the surgery depends on whether or not he expects to be told whether it worked afterward. If Joe expects to have the surgery but to never find out whether or not it worked, then its value is U(0.5)-U(0)=0.25. On the other hand, if he expects to be told whether it worked or not, then he ends up with a belief-score or either 0 or 1, not 0.5, so its value is (0.5*U(1.0) + 0.5*U(0)) - U(0) = 0.5.

Suppose Joe is uncertain whether he's attractive or not - he assigns it a probability of 1/3. Someone offers to tell him the true answer. If Joe's utility-of-belief function is U(p)=p^2, then being told the answer is worth ((1/3)*U(1) + (2/3)*U(0)) - U(1/3) = ((1/3)*1 + (2/3)*0) - (1/9) = 2/9, so he takes the offer. If on the other hand his utility-of-belief function were U(p)=sqrt(p), then being told the information would be worth ((1/3)*sqrt(1) + (2/3)*sqrt(0)) - sqrt(1/3) = -0.244, so he plugs his ears.

Comment author:HonoreDB
26 February 2011 09:33:57PM
1 point
[-]

You're missing the fact that how much Joe values the surgery depends on whether or not he expects to be told whether it worked afterward.

Good point.

If on the other hand his utility-of-belief function were U(p)=sqrt(p), then being told the information would be worth ((1/3)sqrt(1) + (2/3)sqrt(0)) - sqrt(1/3) = -0.244, so he plugs his ears.

I agree here.

But I still suspect that if your U(p) is anything other than linear on p, you can get Dutch-booked. I'll try to come back with a proof, or at least an argument.

Comment author:HonoreDB
28 February 2011 10:43:12PM
*
2 points
[-]

Okay, here we go. I've possibly reinvented the wheel here, but maybe I've come up with a simple, original result. That'd be cool. Or I'm interestingly wrong.

We wish to show that superlinear utility-of-belief functions, or equivalently ones that would cause an agent to prefer ignorance, lead to inconsistency.

Suppose Joe equally wants to believe each of two propositions, P and Q, to be true, with U(x) > x*U(1) for all probabilities x, and U(x) strictly increasing with x. Without loss of generality, we set U(0) to 0 and U(1) to 1. Both propositions concern events that will invisibly occur at some known future time.

Joe anticipates that he will eventually be given the following choice, which will completely determine P and Q:

Option 1: P xor Q. Joe won't know which one is true, so he believes each of them is true with probability 1/2. So he has U(1/2)+U(1/2)=2*U(1/2) utility. By assumption this is greater than 1. So let 2*U(1/2) - 1 = k.

Option 2: One proposition will become definitely true. The other will become true with probability p, where p is chosen to be greater than 0 but less than U-inverse(k). Joe will know which proposition is which. Joe's utility would be less than U(1) + U(U-inverse(k)), or less than 1 + 2*U(1/2) - 1, or less than 2*U(1/2).

Joe prefers Option 1. Therefore he anticipates that he will choose Option 1. Therefore, his current utility is 2*U(1/2). But what if he anticipated that he would choose Option 2? Then his current utility would be 2*U(1/2+p/2). So he wishes his k were smaller than U-inverse(k), meaning he wishes his U(x) were closer to x*U(1). If he were to modify his utility function such that U'(x) = x*U(1) for all x, the new Joe would not regret this decision since it strictly increases his expected utility under the new function.

Thus we can say that all superlinear utility functions are inherently unstable, in that an agent with U(x) > x*U(1) for all probabilities x, and U(x) strictly increasing with x, may increase its expected U by modifying to U'(x) = x*U(1) for all x.

The strongest possible constraint we can give for inherent stability of a utility-of-belief function is that, with utility-of-belief function U, an agent can never improve its U-utility by switching to any other utility function, except under cases wherein it anticipates being modeled by an outside entity. If we removed this exception, no non-degenerate utility-of-belief function could be called stable because we could always posit an outside entity that punishes agents modeled to have specific utility functions. The linear utility of belief function satisfies this condition, since it behaves identically whether it is maximizing the probability of P or its U(p(P)), so it always anticipates itself maximizing its own utility function. We have just shown that no superlinear function satisfies this constraint.

But by conservation of expected evidence, no agent with a linear or sublinear utility-of-belief function can increase its expected utility-of-belief by hiding evidence from itself.

Therefore, a rational agent with a stable utility function cannot make itself happier by hiding evidence from itself, unless it is being modeled by an outside entity.

## Comments (245)

OldThey can. They just do so very rarely, and since accepting some inaccurate beliefs makes it harder to determine which beliefs are and aren't beneficial, in practice we get the highest utility from favoring accuracy. It's very hard to keep the negative effects of a false belief contained; they tend to have subtle downsides. In the example you gave, Joe's belief that he's already smart and beautiful might be stopping him from pursuing self-improvements. But there

definitely arecases where accurate beliefs are definitely detrimental; Nick Bostrom's Information Hazards has a partial taxonomy of them.I don't think it's possible for a reflectively consistent decision-maker to gain utility from self-deception, at least if you're using an updateless decision theory. Hiding an unpleasant fact F from yourself is equivalent to deciding never to know whether F is true or false, which means fixing your belief in F at your prior probability for it. But a consistent decision-maker who loses 10 utilons from believing F with probability ~1 must lose p*10 utilons for believing F with probability p.

*2 points [-]No, this is not true. Many of the reasons why true beliefs can be bad for you are because information about your beliefs can leak out to other agents in ways other than through your actions, and there is is no particular reason for this effect to be linear. For example, blocking communications from a potential blackmailer is good because knowing with probability 1.0 that you're being blackmailed is more than 5 times worse than knowing with probability 0.2 that you will be blackmailed in the future if you don't.

Oh, sure. By "gain utility" I meant "gain utility directly," as in the average Joe story.

I don't think it's linear in the average Joe story, either; if there's one threshold level of belief which changes his behavior, then utility is constant for levels of belief on either side of that threshold and discontinuous in between.

A rational agent can have its behavior depend on a threshold crossing of belief, but if there's some belief that grants it utility

in itself(e.g. Joe likes to believe he is attractive), the utility it gains from that belief has to be linear with the level of belief. Otherwise, Joe can get dutch-booked by a Monte Carlo plastic surgeon.This doesn't sound right. Could you describe the Dutch-booking procedure explicitly? Assume that believing P with probability p gives me utility U(p)=p^2+C.

*0 points [-]An additive constant seems meaningless here: if Joe gets C utilons no matter what p is, then those utilons are unrelated to p or to P--Joe's behavior should be identical if U(p)=p^2, so for simplicity I'll ignore the C.

Now, suppose Joe currently believes he is not attractive. A surgery has a .5 chance of making him attractive and a .5 chance of doing nothing. This surgery is worth U(.5)-U(0)=.25 utilons to Joe; he'll pay up to that amount for it.

Suppose instead the surgeon promises to try again, once, if the first surgery fails. Then Joe's overall chance of becoming attractive is .75, so he'll pay U(.75)-U(0)=.75^2=0.5625 for the deal.

Suppose Joe has taken the first deal, and the surgeon offers to upgrade it to the second. Joe is willing to pay up to the difference in prices for the upgrade, so he'll pay .5625-.25=.3125 for the upgrade.

Joe buys the upgrade. The surgeon performs the first surgery. Joe wakes up and learns that the surgery failed. Joe is entitled to a second surgery, thanks to that .3125-utility purchase of the upgrade. But the second surgery is now worth only .25 utility to him! The surgeon offers to buy that second surgery back from him at a cost of .26 utility. Joe accepts. Joe has spent a net of .0525 utility on an upgrade that gave him no benefit.

As a sanity check, let's look at how it would go if Joe's U(p)=p. The single surgery is worth .5. The double surgery is worth .75. Joe will pay up to .25 utility for the upgrade. After the first surgery fails, the upgrade is worth .5 utility. Joe does not regret his purchase.

*2 points [-]You're missing the fact that how much Joe values the surgery depends on whether or not he expects to be told whether it worked afterward. If Joe expects to have the surgery but to never find out whether or not it worked, then its value is U(0.5)-U(0)=0.25. On the other hand, if he expects to be told whether it worked or not, then he ends up with a belief-score or either 0 or 1, not 0.5, so its value is (0.5*U(1.0) + 0.5*U(0)) - U(0) = 0.5.

Suppose Joe is uncertain whether he's attractive or not - he assigns it a probability of 1/3. Someone offers to tell him the true answer. If Joe's utility-of-belief function is U(p)=p^2, then being told the answer is worth ((1/3)*U(1) + (2/3)*U(0)) - U(1/3) = ((1/3)*1 + (2/3)*0) - (1/9) = 2/9, so he takes the offer. If on the other hand his utility-of-belief function were U(p)=sqrt(p), then being told the information would be worth ((1/3)*sqrt(1) + (2/3)*sqrt(0)) - sqrt(1/3) = -0.244, so he plugs his ears.

Good point.

I agree here.

But I still suspect that if your U(p) is anything other than linear on p, you can get Dutch-booked. I'll try to come back with a proof, or at least an argument.

*2 points [-]Okay, here we go. I've possibly reinvented the wheel here, but maybe I've come up with a simple, original result. That'd be cool. Or I'm interestingly wrong.

We wish to show that superlinear utility-of-belief functions, or equivalently ones that would cause an agent to prefer ignorance, lead to inconsistency.

Suppose Joe equally wants to believe each of two propositions, P and Q, to be true, with U(x) > x*U(1) for all probabilities x, and U(x) strictly increasing with x. Without loss of generality, we set U(0) to 0 and U(1) to 1. Both propositions concern events that will invisibly occur at some known future time.

Joe anticipates that he will eventually be given the following choice, which will completely determine P and Q:

Option 1: P xor Q. Joe won't know which one is true, so he believes each of them is true with probability 1/2. So he has U(1/2)+U(1/2)=2*U(1/2) utility. By assumption this is greater than 1. So let 2*U(1/2) - 1 = k.

Option 2: One proposition will become definitely true. The other will become true with probability p, where p is chosen to be greater than 0 but less than U-inverse(k). Joe will know which proposition is which. Joe's utility would be less than U(1) + U(U-inverse(k)), or less than 1 + 2*U(1/2) - 1, or less than 2*U(1/2).

Joe prefers Option 1. Therefore he anticipates that he will choose Option 1. Therefore, his current utility is 2*U(1/2). But what if he anticipated that he would choose Option 2? Then his current utility would be 2*U(1/2+p/2). So he wishes his k were smaller than U-inverse(k), meaning he wishes his U(x) were closer to x*U(1). If he were to modify his utility function such that U'(x) = x*U(1) for all x, the new Joe would not regret this decision since it strictly increases his expected utility under the new function.

Thus we can say that all superlinear utility functions are inherently unstable, in that an agent with U(x) > x*U(1) for all probabilities x, and U(x) strictly increasing with x, may increase its expected U by modifying to U'(x) = x*U(1) for all x.

The strongest possible constraint we can give for inherent stability of a utility-of-belief function is that, with utility-of-belief function U, an agent can never improve its U-utility by switching to any other utility function, except under cases wherein it anticipates being modeled by an outside entity. If we removed this exception, no non-degenerate utility-of-belief function could be called stable because we could always posit an outside entity that punishes agents modeled to have specific utility functions. The linear utility of belief function satisfies this condition, since it behaves identically whether it is maximizing the probability of P or its U(p(P)), so it always anticipates itself maximizing its own utility function. We have just shown that no superlinear function satisfies this constraint.

But by conservation of expected evidence, no agent with a linear or sublinear utility-of-belief function can increase its expected utility-of-belief by hiding evidence from itself.

Therefore, a rational agent with a stable utility function cannot make itself happier by hiding evidence from itself, unless it is being modeled by an outside entity.