A lot of rationalist thinking about ethics and economy assumes we have very well defined utility functions - knowing exactly our preferences between states and events, not only being able to compare them (I prefer X to Y), but assigning precise numbers to every combinations of them (p% chance of X equals q% chance of Y). Because everyone wants more money, you should theoretically even be able to assign exact numerical values to positive outcomes in your life.
I did a small experiment of making a list of things I wanted, and giving them point value. I must say this experiment ended up in a failure - thinking "If I had X, would I take Y instead", and "If I had Y, would I take X instead" very often resulted in a pair of "No"s. Even thinking about multiple Xs/Ys for one Y/X usually led me to deciding they're really incomparable. Outcomes related to similar subject were relatively comparable, those in different areas in life were usually not.
I finally decided on some vague numbers and evaluated the results two months later. My success on some fields was really big, on other fields not at all, and the only thing that was clear was that numbers I assigned were completely wrong.
This leads me to two possible conclusions:
- I don't know how to draw utility functions, but they are a good model of my preferences, and I could learn how to do it.
- Utility functions are really bad match for human preferences, and one of the major premises we accept is wrong.
Anybody else tried assigning numeric values to different outcomes outside very narrow subject matter? Have you succeeded and want to share some pointers? Or failed and want to share some thought on that?
I understand that details of many utility functions will be highly personal, but if you can share your successful ones, that would be great.
Then, as I said, he cares about something closer to the reality.
The major point I've been trying to make in this thread is that because human preferences are not just in the map but of the map, is that it allows people to persist in delusions about their motivations. And not asking the question is a perfect example of the sort of decision error this can produce!
However, asking the question doesn't magically make the preference about the territory either; in order to prefer the future include his sister's best interests, he must first have an experience of the sister and a reason to wish well of her. But it's still better than not asking, which is basically wireheading.
The irony I find in this discussion is that people seem to think I'm in favor of wireheading because I point out that we're all doing it, all the time. When in fact, the usefulness of being aware that it's all wireheading, is that it makes you better at noticing when you're doing it less-usefully.
The fact that he hadn't asked his sister, or about his sister's actual well-being instantly jumped off the screen at me, because it was (to me) obvious wireheading.
So, you could say that I'm biased by my belief to notice wireheading more, but that's an advantage for a rationalist, not a disadvantage.
Is human knowledge also not just in the map, but exclusively of the map? If not, what's the difference?