Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

TimS comments on Open Thread, June 1-15, 2012 - Less Wrong Discussion

3 Post author: OpenThreadGuy 01 June 2012 04:01AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (252)

Sort By: Best

You are viewing a single comment's thread. Show more comments above.

Comment author: TimS 08 June 2012 02:32:05PM 0 points [-]

My sense is that explicit technological modifications of humans are already heavily concerned with the question "Will we still be human after this modification" which at least gestures at the problems you identify. It is exact the lack of this type of concern in the AI field that motivates SIAI's Friendliness activism. But the sorts of technological advances you are pointing towards seem more likely to arise in part from medical researcher methodologies, which seem more concerned with potential negative psychological and sociological effects than some other forms of technological research.

In short, if every AI researcher was already worried about safety to the extent that medical researchers seem to be worried, then there would be no need for SIAI to exist - all AI researchers worrying about the Friendliness problem is what winning looks like for SIAI. Since medical researchers are already worried about these types of problems, an SIAI-equivalent is not necessary. Consider all the different medical ethics councils - which are much more powerful than their institutional equivalents in AI research.