= 27d567bc2d48d9f40e9ccc20ea370b15)
conchis comments on Post Your Utility Function - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (273)
Human utility functions are relative, contextual, and include semi-independent positive-negative axes. You can't model all that crap with one number.
The study of affective synchrony shows that humans have simultaneously-active positive and negative affect systems. At extreme levels in either system, the other is shut down, but the rest of the time, they can support or oppose each other. (And in positions of opposition, we experience conflict and indecision.)
Meanwhile, the activation of these systems is influenced by current state/context/priming, as well as the envisioned future. So unless your attempt at modeling a utility function includes terms for all these things, you're sunk.
(Personally, this is where I think the idea of CEV has its biggest challenge: I know of no theoretical reason why humans must have convergent or consistent utility functions as individuals, let alone as a species.)
Re: Human utility functions are relative, contextual, and include semi-independent positive-negative axes. You can't model all that crap with one number.
That is not a coherent criticism of utilitarianism. Do you understand what it is that you are criticising?
Yes, I do... and it's not utilitarianism. ;-)
What I'm criticizing is the built-in System 2 motivation-comprehending model whose function is predicting the actions of others, but which usually fails when applied to self, because it doesn't model all of the relevant System 1 features.
If you try to build a human-values-friendly AI, or decide what would be of benefit to a person (or people), and you base it on System 2's model, you will get mistakes, because System 2's map of System 1 is flawed, in the same way that Newtonian physics is flawed for predicting near-light-speed mechanics: it leaves out important terms.