TheAncientGeek comments on Value is Fragile - Less Wrong

41 Post author: Eliezer_Yudkowsky 29 January 2009 08:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (88)

Sort By: Old

You are viewing a single comment's thread.

Comment author: TheAncientGeek 16 December 2014 06:27:19PM *  -2 points [-]

Value isn't fragile because value isn't a process. Only processes can be fragile or robust.

Winning the lottery is fragile, is a fragile process, because it had to be done all in one go. Contrast that with the process of writing down a 12 digit phone number: if you to try to memoriese the whole number, and then write it down, you are likely to make a mistake, due to Millers law. Writing digits down one at time, as you hear them, is more robust. Being able to ask for corrections, or having errors pointed out to you, is more robust still.

Processes that are incremental and involve error correction are robust, and can handle large volumes of data. The data aren't the problem. Trying to preload an AI with the total of human rationality is the problem, because it is the most fragile way of installing human value. Safety researchers need to aim for error, connection, ie corrigibility.