Vladimir_Nesov comments on The Sword of Good - Less Wrong

85 Post author: Eliezer_Yudkowsky 03 September 2009 12:53AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (292)

You are viewing a single comment's thread. Show more comments above.

Comment author: Vladimir_Nesov 03 September 2009 04:08:42PM 1 point [-]

Presumably, actual mutants are unlikely, with most "evil" people actually just holding mistaken (about their actual preference) moral beliefs. If the sword is an external moral authority, it's harder to see why one would consult it.

On the other hand, sword checks soundness of the plan against some preference, which is an important step that is absent if one doesn't consult the sword, which can justify accepting a somewhat mismatched preference if that allows to use the test.

This passes the choice of mismatching preferences to a different situation. If the sword tests person's preference, then protagonist's choice is between lack of progress or unlikely good outcome and (if Vhazhar's plan is sound) verified installation of Vhazhar's preference, with the latter presumably close to others' preference, thus being a moderately good option. If the sword tests some kind of standard preference, this standard preference is presumably also close to Vhazhar's preference, thus Vhazhar faces a choice between trying to install his own preference through unverified process, which can go through all kinds of failure modes, and using the sword to test the reliability of his plan.

The fact that Vhazhar is willing to use the sword to test the soundness of his plan, when the failed test means his death, shows that he prefers leaving the rest of the world be to incorrectly changing it. This is a strong signal that should've been part of the information given to protagonist for making the decision.