In the Wiki article on complexity of value, Eliezer wrote:
The thesis that human values have high Kolmogorov complexity - our preferences, the things we care about, don't compress down to one simple rule, or a few simple rules.
[...]
Thou Art Godshatter describes the evolutionary psychology behind the complexity of human values - how they got to be complex, and why, given that origin, there is no reason in hindsight to expect them to be simple.
But in light of Yvain's recent series of posts (i.e., if we consider our "actual" values to be the values we would endorse in reflective equilibrium, instead of our current apparent values), I don't see any particular reason, whether from evolutionary psychology or elsewhere, that they must be complex either. Most of our apparent values (which admittedly are complex) could easily be mere behavior, which we would discard after sufficient reflection.
For those who might wish to defend the complexity-of-value thesis, what reasons do you have for thinking that human value is complex? Is it from an intuition that we should translate as many of our behaviors into preferences as possible? If other people do not have a similar intuition, or perhaps even have a strong intuition that values should be simple (and therefore would be more willing to discard things that are on the fuzzy border between behaviors and values), could they think that their values are simple, without being wrong?
I don't know, but at least it seems plausible that I might. It's clear that I have boredom as an emotion and as a behavior, but I don't see a clear reason why I would want to make it into a preference. Certainly there are times when I wish I wouldn't get bored as easily as I actually do, so I don't want to translate boredom into a preference "as is".
If I think about why I might not want a future where everyone has no boredom and could enjoy the same thing over and over again, what's driving that seems to be an intuitive aversion towards triviality (i.e., things that look too easy or lack challenge). And if I think more about why I might not want a future where things look easy instead of difficult, I can't really think of anything except that it probably has to do with signaling that I'm someone who likes challenge, which does not seem like something I really want to base my "actual preferences" on.
These are similar for me. If I think about them long enough, it just becomes unclear why I might want to keep them.
Also, perhaps "keep" and "discard" aren't the right word here. What you actually need to do (in this view of what values are), is affirmatively create preferences (e.g., a utility function) from your intuitions. So for any potential value under consideration, you need reasons not for why you would want to "discard", but for why you would want to "create".
This should've been obvious from the start, but your comment has forced me to realize it only now: if we understand reflective equilibrium as the end result of unrestricted iterated self-modification, then it's very sensitive to starting conditions. You and I could end up having very different value systems because I'd begin my self-modification by strengthening my safeguards against simplification of values, while you'd begin by weakening yours. And a stupid person doing unrestricted iterated self-modification will just end up someplace stupid. So this interpretation of "reflective equilibrium" is almost useless, right?