ArisKatsaris comments on Torture vs Dust Specks Yet Again - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (54)
We call that "Omega". Let's use the same term as always.
The obvious response to this is that "should" and "would" are different verbs. There are probably lots of things that I "should" do that I wouldn't do. Knowing the morally proper thing is different from doing the morally proper thing.
Most of your expressed disagreement seems to stem from this: You don't imagine anyone would do this, so you claim you don't think that someone should. Once you've fully understood and internalized that these are different verbs, the confusion disappears.
Yeah, the factor is that we don't actually want the AI to be moral, we want it to follow humanity's Coherent Extrapolated Volition instead.