Call me a chicken, but yes: I would not risk going out empty handed even in 1 out of 100000 if I could have left with $100M.
This kind of super-cautious mindset can't be modeled with any real valued money X (current state of the world) -> utility type of mapping.
Like Vladimir Nesov pointed out, that is false -- not the preference being expressed, of course, but the statement that the preference can't be modeled with the mapping.
Now first let me make it clear that I disapprove of the atmosphere you find in some academic science departments where making a false statement is taken to be a mortifying sin. That kind of attitude is a big barrier to teaching and to learning. Since teaching and learning is a big part of what we want to do here, we should not think poorly of a participant for making a false statement.
But I am a little worried that in 88 hours since the false statement was made, no one downvoted the false statement (or if they did, the vote was canceled out by an upvote). And I am a little worried that in the 81 hours since his reply, no one upvoted Nesov's reply in which he explains why the statement is false. (I have just cast my votes on these 2 comments.)
It is good to have an atmosphere of respect for people even if they make a mistake, but it is bad IMHO when most readers ignore a false statement like the one we have here when there is no doubt about its falseness (it is not open to interpretation) and it involves knowledge central to the mission of the community (e.g., like the one we have here about the most elementary decision theory). Note that elementary decision theory is central to the rationality mission of Less Wrong and to the improve-the-global-situation-with-AI mission of Less Wrong.
Moreover, if you not only read a comment, but also decide to reply to it, well, then IMHO, you should take particular care to make sure you understand the comment, especially when the comment is as short and unnuanced as the one under discussion. But before Nesov's reply, two people replied to the comment under discussion without showing any sign that they recognise that the one statement of fact made in the comment is false. One reply (upvoted 3 times) reads, 'The technical term is "risk-averse", not "chicken"'. The other introduces the Allais paradox, which is irrelevant to why the statement is false.
I do not mean to single out this comment and these 2 replies or the people who wrote them: the only reason I am drawing attention to them is to illustrate something that happens regularly. And I definitely realize that it probably happens a lot less here on Less Wrong than it does in any other conversation on the internet that ranges over as many subject relevant to the human condition as the conversation on Less Wrong does. And a significant reason for that is the hard work Eliezer and others put into the development of the software behind the site.
But I suspect that one of the best opportunities for creating a conversation that is even better than the conversation we are all in right now is to make the response by the community to false statements (the kind not open to interpretation) more salient and more consistent. Wikipedia's response to false statements gives me the impression of rising to the level of saliency and consistency I am talking about, but of course the software behind Wikipedia does not support conversation as well as the software behind Less Wrong does. (And more importantly but more subtly, Wikipedia is badly governed: much of the goodwill and reputation enjoyed by Wikipedia will probably be captured by the ideological and personal agendas of Wikipedia's insiders.)
My original statement was mathematically true. Maybe Vladimir was sloppy reading it (his utilty function satisfied only half of the requirements), but I would not downvote him for that.
Related: Trust in Math
I was reading John Allen Paulos' A Mathematician Plays the Stock Market, in which Paulos relates a version of the well-known "missing dollar" riddle. I had heard it once before, but only vaguely remembered it. If you don't remember it, here it is:
I remembered that the solution involves trickery, but it still took me a minute or two to figure out where it is. At first, I started mentally keeping track of the dollars in the riddle, trying to see where one got dropped so their sum would be 30.
Then I figured it out. The story should end:
I told my fiance the riddle, and asked her where the missing dollar went. She went through the same process as I did, looking for a place in the story where $1 could go missing.
It's remarkable to me how blatantly deceptive the riddle is. The riddler states or implies at the end of the story that the dollars paid by the guests and the dollars kept by the bellhop should be summed, and that that sum should be $30. In fact, there's no reason to sum the dollars paid by the guests and the dollars kept by the bellhop, and no reason for any accounting we do to end up with $30.
The contrasts somewhat with the various proofs that 1 = 2, in which the misstep is hidden somewhere within a chain of reasoning, not boldly announced at the end of the narrative.
Both Paulos and Wikipedia give examples with different numbers that make the deception in the missing dollar riddle more obvious (and less effective). In the case of the missing dollar riddle, the fact that $25, $27, and $30 are close to each other makes following the incorrect path very seductive.
This riddle made me remember reading about how beginning magicians are very nervous in their first public performances, since some of their tricks involve misdirecting the audience by openly lying (e.g., casually pick up a stack of cards shuffled by a volunteer, say "Hmm, good shuffle" while adding a known card to the top of the stack, hand the deck back to the volunteer, and then boldly announce "notice that I have not touched or manipulated the cards!"1). However, they learn to be more comfortable once they find out how easily the audience will pretty much accept whatever false statements they make.
Thinking about these things makes me wonder about how to think rationally given the tendency for human minds to accept some deceptive statements at face value. Can anyone think of good ways to notice when outright deception is being used? How could a rationalist practice her skills at a magic show?
How about other examples of flagrant misdirection? I suspect that political debates might be able to make use of such techniques (I think that there might be some examples in the recent debates over health care reform accounting and the costs of obesity to the health care system, but I haven't been able to find any yet.)
Footnote 1: I remember reading this example very recently, maybe at this site. Please let me know whom to credit for it.