= 27d567bc2d48d9f40e9ccc20ea370b15)
Vaniver comments on We Don't Have a Utility Function - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (123)
Stanovich's paper on why humans are apparently worse at following the VNM axioms than some animals has some interesting things to say, although I don't like the way it says them. I quit halfway through the paper out of frustration, but what I got out of the paper (which may not be what the paper itself was trying to say) is more or less the following: humans model the world at different levels of complexity at different times, and at each of those levels different considerations come into play for making decisions. An agent behaving in this way can appear to be behaving VNM-irrationally when really it is just trying to efficiently use cognitive resources by not modeling the world at the maximum level of complexity all the time. Non-human animals may model the world at more similar levels of complexity over time, so they behave more VNM-rationally even if they have less overall optimization power than humans.
A related consideration, which is more about the methodology of studies claiming to measure human irrationality, is that the problem you think a test subject is solving is not necessarily the problem they're actually solving. I guess a well-known example is when you ask people to play the prisoner's dilemma but in their heads they're really playing the iterated prisoner's dilemma.
And another point: an agent can have a utility function and still behave VNM-irrationally if computing the VNM-rational thing to do given its utility function takes too much time, so the agent computes some approximation of it. It's a given in practical applications of Bayesian statistics that Bayesian inference is usually intractable, so it's necessary to compute some approximation to it, e.g. using Monte Carlo methods. The human brain may be doing something similar (a possibility explored in Lieder-Griffiths-Goodman, for example).
(Which reminds me: we don't talk anywhere near enough about computational complexity on LW for my tastes. What's up with that? An agent can't do anything right if it can't compute what "right" means before the Sun explodes.)
I agree with this concern (and my professional life is primarily focused on heuristic optimization methods, where computational complexity is huge).
I suspect it doesn't get talked about much here because of the emphasis on intelligence explosion, missing AI insights, provably friendly, normative rationality, and there not being much to say. (The following are not positions I necessarily endorse.) An arbitrarily powerful intelligence might not care much about computational complexity (though it's obviously important if you still care about marginal benefit and marginal cost at that level of power). Until we understand what's necessary for AGI, the engineering details separating polynomial, exponential, and totally intractable algorithms might not be very important. It's really hard to prove how well heuristics do at optimization, let alone robustness. The Heuristics and Biases literature focuses on areas where it's easy to show humans aren't using the right math, rather than how best to think given the hardware you have, and some of that may be deeply embedded in the LW culture.
I think that there's a strong interest in prescriptive rationality, though, and if you have something to say on that topic or computational complexity, I'm interested in hearing it.