What irritates me about this post is that Yudkowsky just seems to assume without questioning (at least not in that article and related ones) that we ought be concerned about human morality. In "Fake Utility Functions", he argues that hedonistic utilitarianism fails to due justice to all the complex human values . But that's not the goal utiltiarians wanted to achieve, that's not their view of ethics. Ethics should be independent of the evolutionary psychology of Homo sapiens. Self-aware beings could have ended up with different values. What are the meta criteria by which we should decide what values to have in the first place? Hedonistic utilitarians answer that what matters, ultimately, can only be conscious experience. Yudkowsky seemed to assume that hedonistic utiltiarians thought that humans must want to be hedonistic utiltiarians deep down. But they don't need that to be the case at all. Human ethical intuitions could well be more misguided than Yudkowsky acknowledges anyway (i.e. that many people have strong intuitions against some of the consequences of consequentialism). Yudkowsky's dismissal of the One Great Moral Principle thus seems hastened. Toby Ord made a similar point in the comments to "Fake Utility Functions". (I don't want to advocate classical utilitarianism here because I think there are reasons that speak against happiness being the relevant criterion, I just wanted to point out that more thought should be given to this foundational issue of ethics.)
Fair enough... so, OK, "I made it to self-organize" isn't right either.
That said, I'll point out that that was your own choice of words ("You've implemented a neural network [..] and made it to self-organize").
I mention this, not to criticize your choice of words, but to point out that you have experience with the dynamic that causes people to choose a brief not-quite-right phrase that more-or-less means what we want to express, rather than a paragraph of text that is more precise.
Which is exactly what's going on when people talk about programming a computer to perform cognitive tasks.
I could have challenged your word choice when you made it (just like you did now, when I echoed it back), but I more or less understood what you meant, and I chose to engage with your approximate meaning instead. Sometimes that's a helpful move in conversations.
Yes, there is some ambiguity in use of words, I myself noticed it yesterday. I can only say that you understood it correctly and made the right move! OK, I'll try to be more accurate in using words (sometimes it is not simple, requires time and effort).
Today's post, Heading Toward Morality was originally published on 20 June 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was LA-602 vs RHIC Review, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.