= 27d567bc2d48d9f40e9ccc20ea370b15)
Matt_Simpson comments on Pinpointing Utility - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (154)
One thing you didn't address that was uncertainty about preferences. Specifically, will I die of radiation poisoning if I use VNM utility to make decisions when I'm uncertain about what my preferences even are? I.e., maximize expected utility, where the expectation is taken over my uncertainty about preferences in addition to any other uncertainty.
I thought you took a position on this and was about to comment on it but I couldn't find what you said about it in the post! Apparently my brain deduced a conclusion on this issue from your post, then decided to blame/give credit to you.
Yeah I totally sidestepped that issue because I don't know how to solve it. I don't think anyone knows, actually. Preference uncertainty is an open problem, AFAIK.
Yes. You can't compare or aggregate utilities from different utility functions. So at present, you basically have to pick one and hope for the best.
Eventually someone will have to build a new thing for preference uncertainty. It will almost surely degenerate to VNM when you know your utility function.
There are other problems that also sink naive decision theory, like acausal stuff, which is what UDT and TDT try to solve, and anthropics, which screw up probabilities. There's a lot more work on those than on preference uncertainty, AFAIK.
This is exactly what my brain claimed you said :) Now I can make my comment.
Game theorists do this all the time - at least economists. They'll create a game, then say something like "now let's introduce noise into the payoffs" but the noise ends up being in the utility function. Then they go and find an equilibrium or something using expected utility.
Now every practical example I can think of off the top of my hand, you can reinterpret the uncertainty as uncertainty about actual outcomes with utilities associated with those outcomes and the math goes through. Usually the situation is something like letting U($)=$ for simplicity because risk aversion is orthogonal to what they're interested in, so you can easily think about the uncertainty as being over $ rather than U($). This simplicity allows them to play fast and loose with VNM utility and get away with it, but I wouldn't be surprised if someone made a model where they really do mean for the uncertainty to be over one's own preferences and went ahead and used VNM utility. In any case, no one ever emphasized this point in any of the econ or game theory courses I've taken, grad or otherwise.
In case you're still interested
Thanks!
If you can do that, it seems to work; Noise in the payoffs is not preference uncertainty, just plain old uncertainty. So I guess my question is what does it look like when you can't do that, and what do we do instead?
If you can do that, it seems to work; Noise in the payoffs is not preference uncertainty, just plain old uncertainty. So I guess my question is what does it look like when you can't do that, and what do we do instead?
You can at least simplify the problem somewhat by applying VNM utility using each of the candidate utility functions, and throwing out all solutions that do not appear in any of them. If you think you like either carrots or apples, you're not going to go to the store and buy asparagus.
The other thorny issue is that uncertainty in the utility function makes learning about your utility function valuable. If you think you like either carrots or apples, then taking two grocery trips is the best answer - on the first trip you buy a carrot and an apple and figure out which one you like, and on the second trip you stock up.
The other thing is that I don't think it's possible to model uncertainty inside your utility function - you can only have uncertainty about how you evaluate certain events. If you don't know whether or not you like carrots, that's a fact about eating carrots and not one about how to decide whether or not to eat a carrot. I think that every uncertainty about a utility function is just a hidden uncertainty about how the being the experiences utility works.
Let me be specific about the math. Suppose you have a lottery L with a 1/3rd chance of result A and a 2/3rd chance of result B. Suppose furthermore that you are uncertain about whether you enjoy things as in U1 or U2, with equal probability of each. L is equivalent to a lottery with 1/6 chance (A, U1), 1/3 chance (B, U1), etc. Now you can make the first utility function of this exercise that takes into account all your uncertainty about preferences.
Note that U1 and U2 aren't numbers - it's how much you enjoy something if your preferences are as in U1.
What this lets us do is convert "there's a chance I get turned into a whale and I'm not sure if I will like it" into "there's a chance that I get turned into a whale and like it, and another chance that I get turned into a whale and don't like it".
Ooops. Radiation poisoning. Utility is about planning, not experiencing or enjoying.
I went through the math a couple days ago with another smart philosopher-type. We are pretty sure that this (adding preference uncertainty as an additional dimension of your ontology) is a fully general solution to preference uncertainty. Unfortunately, it requires a bit of moral philosophy to pin down the relative weights of the utility functions. That is, the utility functions and their respective probabilities is not enough to uniquely identify the combined utility function. Which is actually totally ok, because you can get that information from the same source where you got the partial utility functions.
I'll go through the proof and implications/discussion in an upcoming post. Hopefully. I don't exactly have a track record of following through on things...
Right, to get that answer you need to look inside your utility function... which you're uncertain about. Stated differently, your utility function tells you how to deal with uncertainty about your utility function, but that's another thing you're uncertain about. But luckily your utility function tells you how do deal with uncertainty about uncertainty about your utility function... I think you can see where this is going.
Naively, my intuition is that simply adding uncertainty about preferences as part of your ontology isn't enough because of this regress - you still don't even know in principle how to choose between actions without more precise knowledge of your utility function. However, this regress sounds suspiciously like the sort of thing that once formalized precisely isn't really a problem at all - just "take the limit" as it were.
That's not the issue we ran into.
Your (partial) utility functions do not contain enough information to resolve uncertainty between them. As far as I can tell, utility functions can't contain meta-preferences.
You can't just pull a correct utility function out of thin air, though. You got the utility function from somewhere; it is the output of a moral-philosophy process. You resolve the uncertainty with the same information-source from which you constructed the partial utility functions from in the first place.
No need to take the limit or do any extrapolation (except that stuff like that does seem to show up inside the moral-philosophy process.)
I think we're using "utility function" differently here. I take it to mean the function containing all information about your preferences, preferences about preferences, and higher level meta-preferences. I think you're using the term to refer to the function containing just object-level preference information. Is that correct?
Now that I make this distinction, I'm not sure VNM utility applies to meta-preferences.
It doesn't, AFAIK, which is why I said your utility function does not contain meta-preference and the whole moral dynamic. "utility function" is only a thing in VNM. Using it as a shorthand for "my whole reflective decision system" is incorrect use of the term, IMO.
I am not entirely sure that your utility function can't contain meta-preference, though. I could be convinced by some well-placed mathematics.
My current understanding is that you put the preference uncertainty into your ontology, extend your utility function to deal with those extra dimensions, and lift the actual moral updating to epistemological work over those extra ontology-variables. This still requires some level of preliminary moral philosophy to shoehorn your current incoherent godshatter-soup into that formal framework.
I'll hopefully formalize this some day soon to something coherent enough to be criticized.
I look forward to it!
Nice catch on the radiation poisoning. Revised sentence:
Also
This is 100% expected, since utility functions that vary merely by a scaling factor and changing the zero point are equivalent.
I think we're talking about the same thing when you say "adding preference uncertainty as an additional dimension of your ontology". It's kind of hard to tell at this level of abstraction.