You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

mwengler comments on Do Earths with slower economic growth have a better chance at FAI? - Less Wrong Discussion

30 Post author: Eliezer_Yudkowsky 12 June 2013 07:54PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (174)

You are viewing a single comment's thread. Show more comments above.

Comment author: mwengler 14 June 2013 03:11:49PM 0 points [-]

Extremely high IQ arising from engineering... is that not AI?

This is not a joke. UAI is essentially the fear that "we" will be replaced by another form of intelligence, outcompeted for resources by what is essentially another life form.

But how do "we" not face the same threat from an engineered lifeform just because some of the ingredients are us? If such a new engineered lifeform replaces natural humanity, is that not a UAI? If we can build in some curator instinct or 3 laws or whatever into this engineered superhuman, is that not FAI?

The interesting thing to me here is what we mean by "we." I think it is more common for a lesswrong poster to identify as "we" with an engineered superhuman in meat substrate than with an engineered non-human intelligence in non-meat substrate.

Considering this, maybe an FAI is just an AI that learns enough about what we think of as human so that it can hack it. It could construct itself so that it felt to us like our descendent, our child. Then "we" do not resent the AI for taking all "our" resources because the AI has successfully lead us to be happy to see our child succeed beyond what we managed.

Perhaps one might say but of course this would be on our list of things we would define as unfriendly. Then we build AI's that "curate" humans as we are now and we are precluding from enhancing ourselves or evolving past some limit we have preprogrammed in to our FAI?