= 27d567bc2d48d9f40e9ccc20ea370b15)
endoself comments on St. Petersburg Mugging Implies You Have Bounded Utility - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (163)
I am currently researching logical uncertainty. I believe that the increased chance of FAI due to this research makes it the best way to act according to my utility function, taking into account the limits to my personal rationality (part of this is personal; I am particularly interested in logical uncertainty right now, so I am more likely to make progress in it than on other problems). This is because, among other things, an FAI will be far better at understanding the difficulties associated with unbounded utility functions than I am.
You have not demonstrated it to be impossible, you have just shown that the most obvious approach to it does not work. Given how questionable some of the axioms we use are, this is not particularly surprising.
An actual description of my preferences. I am unsure whether my utility function is actually unbounded but I find it probable that, for example, my utility function is linear in people. I don't want to rule this out just because that current framework is insufficient for it.
Some paths are far more likely than others. Actively researching FAI in a way that is unlikely to significantly increase the probability of UFAI provides far more expected utility than unlikely ways to help the development of FAI.
Predicting your preferences requires specifying both the utility function and the framework, so offering a utility function without the framework as an explanation for your preferences does not actually explain them. I actually don't know if my question was hypothetical or not. Do we have a decision procedure that gives reasonable results for an unbounded utility function?
The phrase "rule this out" seems interesting here. At any given time, you'll have a set of explanations for your behavior. That doesn't rule out coming up with better explanations later. Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
Perhaps I see what's going on here -- people who want unbounded utility are feeling loss when they imagine giving that up that unbounded goodness in order to avoid bugs like the one described in the OP. I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math. Specifically, dealing effectively with the default future, in which one or more corporations make AI's that optimize for something having no connection to the preferences of any individual human.
Not one compatible with a Solomonoff prior. I agree that a utility function alone is not a full description of preferences.
The best explanation that I have for my preferences does not, AFAICT, work with any known decision theory. However, I know enough of what such a decision theory would look like if it were possible to say that it would not have bounded utility.
I disagree that I am doing such. Whether or not the math is relevant to the issue is a question of values, not fact. Your estimates of your values do not find the math relevant; my estimates of my values do.