Such an eventuality would seem to require that (a) human beings are not computable or (b) human beings are not Friendly.
In the latter case, if nothing else, there is [individual]-Friendliness to consider.
I think human history has demonstrated that (b) is certainly true... sometimes I am surprised we are still here.
An Open Thread: a place for things foolishly April, and other assorted discussions.
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.
Update: Tom McCabe has created a sub-Reddit to use for assorted discussions instead of relying on open threads. Go there for the sub-Reddit and discussion about it, and go here to vote on the idea.