= 27d567bc2d48d9f40e9ccc20ea370b15)
David_Gerard comments on We Don't Have a Utility Function - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (123)
Stanovich's paper on why humans are apparently worse at following the VNM axioms than some animals has some interesting things to say, although I don't like the way it says them. I quit halfway through the paper out of frustration, but what I got out of the paper (which may not be what the paper itself was trying to say) is more or less the following: humans model the world at different levels of complexity at different times, and at each of those levels different considerations come into play for making decisions. An agent behaving in this way can appear to be behaving VNM-irrationally when really it is just trying to efficiently use cognitive resources by not modeling the world at the maximum level of complexity all the time. Non-human animals may model the world at more similar levels of complexity over time, so they behave more VNM-rationally even if they have less overall optimization power than humans.
A related consideration, which is more about the methodology of studies claiming to measure human irrationality, is that the problem you think a test subject is solving is not necessarily the problem they're actually solving. I guess a well-known example is when you ask people to play the prisoner's dilemma but in their heads they're really playing the iterated prisoner's dilemma.
And another point: an agent can have a utility function and still behave VNM-irrationally if computing the VNM-rational thing to do given its utility function takes too much time, so the agent computes some approximation of it. It's a given in practical applications of Bayesian statistics that Bayesian inference is usually intractable, so it's necessary to compute some approximation to it, e.g. using Monte Carlo methods. The human brain may be doing something similar (a possibility explored in Lieder-Griffiths-Goodman, for example).
(Which reminds me: we don't talk anywhere near enough about computational complexity on LW for my tastes. What's up with that? An agent can't do anything right if it can't compute what "right" means before the Sun explodes.)
I spent a large chunk of Sunday and Monday finally reading Death Note and came to appreciate how some people on LW can think that agents meticulously working out each other's "I know that you know that I know" and then acting so as to interact with their simulations of each other, including their simulations of simulating each other, can seem a reasonable thing to aspire to. Even if actual politicians and so forth seem to do it by intuition, i.e., much more in hardware.
Have you ever played that thumb game where you stand around in a circle with some people and at each turn show 0, 1 or 2 thumbs? And each person takes turns calling out a guess for the total number of thumbs that will be shown? Playing that game gives a really strong sense of "Aha! I modeled you correctly because I knew that you knew that I knew ..." but I never actually know if it's real modeling or hindsight bias because of the way the game is played in real time. Maybe there's a way to modify the rules to test that?
I once spent a very entertaining day with a friend wandering around art exhibits once, with both of us doing a lot of "OK, you really like <i>that</i> and <i>that</i> and <i>that</i> and you hate <i>that</i> and <i>that</i>" prediction and subsequent correction.
One thing that quickly became clear was that I could make decent guesses about her judgments long before I could articulate the general rules I was applying to do so, which gave me a really strong sense of having modeled her really well.
One thing that became clear much more slowly was that the general rules I was applying, once I became able to articulate them, were not nearly as complex as they seemed to be when I was simply engaging with them as these ineffable chunks of knowledge.
I concluded from this that that strong ineffable sense of complex modeling is no more evidence of complex modeling than the similar strong ineffable sense of "being on someone's wavelength" is evidence of telepathy. It's just the way my brain feels when it's applying rules it can't articulate to predict the behavior of complex systems.
This kind of explicit modelling is a recurring fictional trope.
For example, Herbert uses it a lot in Dosadi Experiment to show off how totes cognitively advanced the Dosadi are.
Yes, but aspiring to it as an achievable thing very much strikes me as swallowing fictional evidence whole. (And, around LW, manga and anime.)
No argument; just citing prior fictional art. :-)