endoself comments on The Human's Hidden Utility Function (Maybe) - Less Wrong

44 Post author: lukeprog 23 January 2012 07:39PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (87)

You are viewing a single comment's thread. Show more comments above.

Comment author: gaffa 23 January 2012 10:41:57PM *  1 point [-]

As a first reaction (and without being read up on the details), I'm very skeptical. Assuming these three systems are actually in place, I don't see any convincing reason why any one of them should be trusted in isolation. Natural selection has only ever been able to work on their compound output, oblivious to the role played by each one individually and how they interact.

Maybe the "smart" system has been trained to assign some particular outcome a value of 5 utilons, whereas we would all agree that it's surely and under all circumstances worth more than 20, because as it happens throughout evolution one of the other "dumb" systems has always kicked in and provided the equivalent of at least 15 utilons. If you then extract the first system bare and naked, it might deliver some awful outputs.

Comment author: endoself 24 January 2012 04:09:52AM 0 points [-]

I'm not familiar with the theory beyond what Luke has posted, but I think only one system is active at a time, so there is no summation occurring. However, we don't yet know what determines which system makes a particular decision or how these systems are implemented, so there definitely could be problems isolating them.