My primary moral is to resist the temptation to generalize over all of mind design space
If we focus on the bounded subspace of mind design space which contains all those minds whose makeup can be specified in a trillion bits or less, then every universal generalization that you make has two to the trillionth power chances to be falsified.
That's what I think every time someone brings up the idea of tortured sims. What are the odds of it happening?
My primary moral is to resist the temptation to generalize over all of mind design space
That's what I think every time someone brings up the idea of tortured sims. What are the odds of it happening?
The odds of that happening are almost entirely unrelated to the proportion of mind-space such torture machines make up. That kind of 'torture AI' is something that would be created - either chosen out of mind space deliberately or chosen by making a comparatively tiny error when aiming for a Friendly AI. It isn't the sort of thing that is just randomly selected.
Today's post, The Design Space of Minds-In-General was originally published on 25 June 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was The Psychological Unity of Humankind, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.