= 27d567bc2d48d9f40e9ccc20ea370b15)
Wei_Dai comments on Naturalism versus unbounded (or unmaximisable) utility options - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (72)
You could generate a random number using a distribution that has infinite expected value, then tell Omega that number. Your expected utility of following this procedure is infinite.
But if there is a non-zero chance of an Omega existing that can grant you an arbitrary amount of utility, then there must also a non-zero chance of some Omega deciding on its own at some future time to grant you a random amount of utility using the above distribution, so you've already got infinite expected utility, no matter what you do.
It doesn't seem to me the third problem ("You're immortal. Tell Omega any real number r > 0, and he'll give you 1-r utility.") corresponds to any real world problems, so generalizing from the first two, the problem is just the well known problem of unbounded utility function leading to infinite or divergent expected utility. I don't understand why a lot of people seem to think very highly of this post. (What's the relevance of using ideas related to Busy Beaver to generate large numbers, if with a simple randomized strategy, or even by doing nothing, you can get infinite expected utility?)
Can a bounded agent actually do this? I'm not entirely sure.
Even so, given any distribution f, you can generate a better (dominant) distribution by taking f and adding 1 to the result. So now, as a bounded agent, you need to choose among possible distributions - it's the same problem again. What's best distribution you can specify and implement, without falling into a loop or otherwise saying yes forever?
??? Your conclusion does not follow, and is irrelevant - we care about the impact of our actions, not about hypothetical gifts that may or may not happen, and are disconnected from anything we do.
First write 1 on a piece of paper. Then start flipping coins. For every head, write a 0 after the 1. If you run out of space on the paper, ask Omega for more. When you get a tail, stop and hand the pieces of paper to Omega. This has expected value of 1/2 * 1 + 1/4 * 10 + 1/8 * 100 + ... which is infinite.
How does that relate to the claim in http://en.wikipedia.org/wiki/Turing_machine#Concurrency that "there is a bound on the size of integer that can be computed by an always-halting nondeterministic Turing machine starting on a blank tape"?
I think my procedure does not satisfy the definition of "always-halting" used in that theorem (since it doesn't halt if you keep getting heads) even though it does halt with probability 1.
That's probably the answer, as your solution seems solid to me.
That still doesn't change my main point: if we posit that certain infinite expectations are better than others (St Petersburg + $1 being better that St Petersburg), you still benefit from choosing your distribution as best you can.
Can you give a mathematical definition of how to compare two infinite/divergent expectations and conclude which one is better? If you can't, then it might be that such a notion is incoherent, and it wouldn't make sense to posit it as an assumption. (My understanding is that people have previously assumed that it's impossible to compare such expectations. See http://singularity.org/files/Convergence-EU.pdf for example.)
Not all infinite expectations can be compared (I believe) but there's lots of reasonable ways that one can say that one is better than another. I've been working on this at the FHI, but let it slide as other things became more important.
One easy comparison device: if X and Y are random variables, you can often calculate the mean of X-Y using the Cauchy principal value (http://en.wikipedia.org/wiki/Cauchy_principal_value). If this is positive, then Y is better than X.
This gives a partial ordering on the space of distributions, so one can always climb higher within this partial ordering.
Assuming you want to eventually incorporate the idea of comparing infinite/divergent expectations into decision theory, how do you propose to choose between choices that can't be compared with each other?
Random variables form a vector space, since X+Y and rX are both defined. Let V be this whole vector space, and let's define a subspace W of comparable random variables. ie if X and Y are in W, then either X is better than Y, worse, or they're equivalent. This can include many random variables with infinite or undefined means (got a bunch of ways of comparing them).
Then we simply need to select a complementary subspace W^perp in V, and claim that all random variables on it are equally worthwhile. This can be either arbitrary, or we can use other principles (there are ways of showing that even if we can't say that Z is better than X, we can still find a Y that is worse than X but incomparable to Y).
The point might be that if all infinite expected utility outcomes are considered equally valuable, it doesn't matter which strategy you follow, so long as you reach infinite expected utility, and if that includes the strategy of doing nothing in particular, all games become irrelevant.
If you don't like comparing infinite expected outcomes (ie if you don't think that (utility) St Petersburg + $1 is better than simply St Petersburg), then just focus on the third problem, which Wei has oddly rejected.
I've often stated my worry that Omega can be used to express problems that have no real-world counterpart, thus distracting our attention away from problems that actually need to be solved. As I stated at the top of this thread, it seems to me that your third problem is such a problem.
Got a different situation where you need to choose sensibly between options with infinite expectation: http://lesswrong.com/r/discussion/lw/gng/higher_than_the_most_high/
Is this a more natural setup?
Actually, the third problem is probably the most relevant of them all - it's akin to a bounded paperclipper uncertain as to whether they've succeeded. Kind of like: "You get utility 1 for creating 1 paperclip and then turning yourself off (and 0 in all other situations)."
I still don't see how it's relevant, since I don't see a reason why we would want to create an AI with a utility function like that. The problem goes away if we remove the "and then turning yourself off" part, right? Why would we give the AI a utility function that assigns 0 utility to an outcome where we get everything we want but it never turns itself off?
The designer of that AI might have (naively?) thought this was a clever way of solving the friendliness problem. Do the thing I want, and then make sure to never do anything again. Surely that won't lead to the whole universe being tiled with paperclips, etc.
This can arise indirectly, or through design, or for a host of reasons. That was the first thought that popped into my mind; I'm sure other relevant examples can be had. We might not assign such a utility - then again, we (or someone) might, which makes it relevant.
Does this not mean that such a task is impossible? http://en.wikipedia.org/wiki/Non-deterministic_Turing_machine#Equivalence_with_DTMs