You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Mestroyer comments on Upgrading moral theories to include complex values - Less Wrong Discussion

1 Post author: Ghatanathoah 27 March 2013 06:28PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (21)

You are viewing a single comment's thread. Show more comments above.

Comment author: Mestroyer 28 March 2013 02:57:18AM 1 point [-]

Allowing yourself to be swayed by thought experiments of the form "If you value x at all, then if offered enough x, you would give up y, and you value one y so much more than one x!" is a recipe for letting scope insensitivity rewrite your utility function.

Be careful about making an emotional judgement on numbers bigger than you can intuitively grasp. If possible, scale down the numbers involved (Instead of 1 million x vs 1 thousand y, imagine 1 thousand x vs 1 y) before imagining each alternative as viscerally as possible.

In my own experience, not estimating a number, just thinking "Well, it's really big" basically guarantees I will not have the proper emotional response.