Evidence for the orthogonality thesis

Stuart_Armstrong

Evidence for the orthogonality thesis — LessWrong

Comment Permalink

You are assuming that the AI needs something from us, which may not be true as it develops further. The decorator follows the implied wishes not because he is smart enough to know what they are, but because he wishes to act in his client's interest to gain payment, reputation, etc. Or he may believe that fulfilling his client's wishes are morally good according to his morality. The mere fact that the wishes of his client are known does not guarantee that he will carry them out unless he values the client in some way to begin with (for their money or maybe their happiness)

TheAncientGeek13y-10

You are assuming that the AI needs something from us, which may not be true as it develops further. The decorator follows the implied wishes not because he is smart enough to know what they are, but because he wishes to act in his client's interest to gain payment, reputation, etc. Or he may believe that fulfilling his client's wishes are morally good according to his morality. The mere fact that the wishes of his client are known does not guarantee that he will carry them out unless he values the client in some way to begin with (for their money or maybe their happiness)

You are assuming that an .AI will last have only instrumental rationality. That the OT is true.

3XiXiDu14y

And an AGI wishes to achieve its goals the way they are meant to be achieved. Which includes all implicit conditions. An AGI does not have to explicitly care about humans and their values as long as the implied context of its goals is human volition. Consider a rich but sociopathic human decorator who solely cares about being a good decorator. What does a good decorator do? It does what its contract explicitly tells him to do AND what is implied by it, including the satisfaction of the customer. You don't need human moral values or any other complex values as long as you care to achieve your goals the way they are meant to be achieved, explicitly and implicitly.

See in context